I think so too. Young-Hyun
On Tue, Jun 3, 2014 at 3:58 PM, Curtis <[email protected]> wrote: > On Tue, Jun 3, 2014 at 1:52 PM, Aaron Peeler <[email protected]> > wrote: > > Would using the private IP instead of hostnames resolve it? It would > > take a bit of work, but could be done. > > Yup that would work...as far as I can tell. :) > > Thanks, > Curtis. > > > > > Aaron > > > > > > > > On Tue, Jun 3, 2014 at 3:22 PM, Curtis <[email protected]> wrote: > >> On Tue, Jun 3, 2014 at 11:53 AM, YOUNG OH <[email protected]> > wrote: > >>> Hi Curtis, > >>> > >>> Thank you for updating the status and correction about the OpenStack > Perl > >>> SDKs. > >>> > >>> If DNS lookup is not reliable to find the private ip of an instance > for a > >>> multi-host openstack environment, then do you have any suggestion to > deal > >>> with this issue? > >>> > >>> At this point, I think at least we have an agreement that we do NOT > want to > >>> use /etc/hosts anymore, right? > >>> If so, we can first try to use the database to get/update the private > ip > >>> address with DNS lookup for a single-host openstack environment. After > >>> that, we can try to find a solution to manage the DNS lookup or other > >>> methods of finding the private ip of an instance in multi-host > openstack > >>> environment. If you have any other suggestions, please let me know. > >>> > >> > >> I'm not too sure what to do. I'm hoping the Apache VCL core devs chime > in. > >> > >> As far as I can tell, Apache VCL needs DNS resolution to work, so > >> simply adding the IP into the database isn't going to be enough. > >> > >> For the current Apache VCL + OpenStack multihost cloud installation > >> I'm working on I'm just going to add a python script that queries > >> OpenStack and generates /etc/hosts every 20 or 30 seconds or something > >> like that. > >> > >> Thanks, > >> Curtis. > >> > >>> Thanks, > >>> Young-Hyun > >>> > >>> > >>> > >>> On Mon, Jun 2, 2014 at 3:56 PM, Curtis <[email protected]> wrote: > >>> > >>>> On Mon, Jun 2, 2014 at 12:45 PM, YOUNG OH <[email protected]> > wrote: > >>>> > Hi All, > >>>> > > >>>> > I would like to bring your attention again to discuss what OpenStack > >>>> module > >>>> > should include. I would like to discus two issues first. > >>>> > > >>>> > First, we know that using /etc/hosts to get the private ip of an > instance > >>>> > can be problematic. Specifically, Cameron (at Cybera) and I had > found > >>>> that > >>>> > there might duplicated or blank entries and rarely but OpenStack > APIs > >>>> > cannot get the private ip (not populated) of an instance properly. > For > >>>> the > >>>> > later case, Cameron added DNS Lookup function to get the correct > private > >>>> IP > >>>> > (Cameron, correct me if I am wrong). This problem leads us to avoid > using > >>>> > /etc/hosts file to get the private ip but I think we need to first > >>>> discuss > >>>> > what the best way is to handle the private ip: > >>>> > > >>>> > (a). Use the privateIPaddress field in computer table. > >>>> > Andy suggests that we can change the get_computer_private_ip to get > >>>> private > >>>> > ip from the database instead of using /etc/hosts (Andy, correct me > if I > >>>> am > >>>> > wrong). I agree with his idea because this can avoid duplicated or > blank > >>>> > entry problem of using /etc/hosts. > >>>> > > >>>> > (b). OpenStack (DNS lookup for finding unpopulated private ip) > >>>> > In our openstack module, DNS lookup or other methods (if any) can > give > >>>> the > >>>> > correct the private ip of the instance in order to deal with the > problem > >>>> > for the private ip not being populated in the hosts file. Cameron > >>>> modified > >>>> > the get_computer_private_ip in DataStructure.pm to do a DNS lookup > >>>> instead > >>>> > of using the hosts file. But I would like to suggest that if > possible, it > >>>> > can be better to add the DNS lookup function to openstack module > rather > >>>> > than changing the DataStructure.pm. That is, we can update the > >>>> > privateIPaddress field in computer table using DNS lookup instead of > >>>> > injecting the private ip to /etc/hosts in openstack module. > >>>> > >>>> We've just found out recently that the helpful DNS resolution that > >>>> openstack provides does not work well in a mutlihost openstack > >>>> environment, so can't be relied on. In multihost mode each compute > >>>> host only knows about the vms running on it locally. > >>>> > >>>> > > >>>> > I think, if possible, one of the best methods for dealing with the > >>>> private > >>>> > ip is that we can use the database to get the private ip of an > instance > >>>> and > >>>> > using DNS lookup in openstack module to update the information. Any > >>>> > suggestions and corrections about my opinion would be very helpful > to > >>>> solve > >>>> > this problem. > >>>> > > >>>> > Second, currently we have two versions (not exactly but for > convenience); > >>>> > one uses OpenStack command line interfaces (CLIs) and one uses > OpenStack > >>>> > Perl SDKs. As I understand, the Perl SDKs are wrapper APIs for the > >>>> > OpenStcak CLIs and required to install through CPAN. I think both > >>>> versions > >>>> > have pros and cons. So, I would like to ask your opinions whether > it is > >>>> > better to include two different openstack versions to the next VCL > >>>> release, > >>>> > to find a way to merge two versions into one openstack module, OR to > >>>> create > >>>> > a higher level of openstack module so that it can choose one version > >>>> based > >>>> > on user's environment. Any suggestions would be also very helpful to > >>>> decide > >>>> > how we can manage openstack module. Thank you. > >>>> > >>>> FYI -- The perl openstack sdk is not a wrapper for the command line > >>>> utils. It uses the openstack api, not the command line tools. > >>>> > >>>> Thanks, > >>>> Curtis. > >>>> > >>>> > > >>>> > Best regards, > >>>> > Young-Hyun > >>>> > > >>>> > > >>>> > > >>>> > On Mon, Jun 2, 2014 at 2:22 PM, YOUNG OH <[email protected]> > >>>> wrote: > >>>> > > >>>> >> Hi All, > >>>> >> > >>>> >> I have tested my openstack module ( > >>>> >> https://issues.apache.org/jira/browse/VCL-590) on the OpenStack > >>>> Icehouse > >>>> >> (The ninth release of OpenStack, April 17, 2014) and I could > >>>> provisioning a > >>>> >> Linux image through it. > >>>> >> > >>>> >> The environment details are as follows: > >>>> >> > >>>> >> 1. Host Blade > >>>> >> :- CPU: Intel(R) Xeon(R) CPU E5645 @ 2.40GHz > >>>> >> :- Memory: 49G > >>>> >> :- Disk: 134G > >>>> >> :- OS: RHEL 6.4 x64 with KVM hypervisor > >>>> >> 2. OpenStack Icehouse Components > >>>> >> :- Compute > >>>> >> :- Network (Nova-Network, not OpenStack Neutron) > >>>> >> :- Glance > >>>> >> 3. Provisioning Linux image > >>>> >> :- CentOS 6.3 x64 > >>>> >> 4. VCL 2.3 > >>>> >> > >>>> >> In this test, I've just tested the provisioning an image but I > would > >>>> like > >>>> >> to post the detail information to jira once I finish other test. In > >>>> >> addition, I will try to add a function of selecting OpenStack > flavors > >>>> based > >>>> >> on the size of an image. The current openstack module sets flavors > for > >>>> each > >>>> >> image because the previous OpenStack releases (Essex as I know) > didn't > >>>> >> support the size information of images but now it supports. I will > also > >>>> >> post to jira issue when I add the function to my module. Thank you. > >>>> >> > >>>> >> Best regards, > >>>> >> Young-Hyun > >>>> >> > >>>> >> > >>>> >> > >>>> >> On Mon, Mar 31, 2014 at 12:55 PM, Cameron Mann < > [email protected]> > >>>> >> wrote: > >>>> >> > >>>> >>> > > >>>> >>> > I haven't looked at Try::Tiny or any newer exception handling > >>>> features > >>>> >>> of > >>>> >>> > Perl, but I'd prefer to keep the number of external modules to a > >>>> >>> minimum if > >>>> >>> > possible. If there are new features in Perl which would be > >>>> beneficial, > >>>> >>> I'd > >>>> >>> > rather make the newer version of Perl a requirement and use the > >>>> built in > >>>> >>> > features. Otherwise, we're locked into using an external > module, > >>>> >>> usually > >>>> >>> > from CPAN, which has no quality assurance. I regret adding > some of > >>>> the > >>>> >>> > current modules such as InsideOut, rather then writing the code > which > >>>> >>> could > >>>> >>> > accomplish the same objective. > >>>> >>> > Did you need to catch exceptions because one of the OpenStack > modules > >>>> >>> > you're using may throw a die exception? When this happens, the > >>>> entire > >>>> >>> vcld > >>>> >>> > process for the reservation exits abruptly which causes problems > >>>> such as > >>>> >>> > the request state remaining in pending indefinitely. This > happens > >>>> even > >>>> >>> if > >>>> >>> > you wrap the call in an eval block. I have worked around this > >>>> problem > >>>> >>> > before by locally overriding the die handler. Take a look at > >>>> >>> > vSphere_SDK.pm and find lines such as: > >>>> >>> > # Override the die handler > >>>> >>> > local $SIG{__DIE__} = sub{}; > >>>> >>> > The vSphere SDK is one example which may call die (via Carp). > If > >>>> this > >>>> >>> > happens with die overridden, the eval block returns normally. > If an > >>>> >>> > exception occurred, $EVAL_ERROR should be set. > >>>> >>> > >>>> >>> > >>>> >>> The OpenStack SDK throws exceptions when requests that > >>>> >>> delete/modify/create > >>>> >>> resources fail, no die exceptions though. There wasn't any > specific > >>>> >>> feature > >>>> >>> that made me choose Try::Tiny, it was more my unfamiliarity with > >>>> exception > >>>> >>> handling in Perl which seemed to have a number of gotchas prior to > >>>> 5.14 so > >>>> >>> I figured it'd be better to let someone else's code worry about > that. > >>>> >>> > >>>> >>> > >>>> >>> > >>>> > http://blog.twoshortplanks.com/2011/06/06/unexceptional-exceptions-in-perl-5-14/ > >>>> >>> > >>>> >>> I see no problems. Any barriers or difficulty you encounter with > the > >>>> core > >>>> >>> > code should be addressed by improving the core code. > >>>> >>> > Reworking the dependency on /etc/hosts is something I'd like to > >>>> improve > >>>> >>> > regardless. People often encounter problems with this if they > >>>> forget to > >>>> >>> > add entries to /etc/hosts for all their computers, including the > >>>> >>> management > >>>> >>> > node(s). I'd like to not read /etc/hosts on the management > node at > >>>> all > >>>> >>> and > >>>> >>> > have the value in the computer table be authoritative. This > should > >>>> >>> > facilitate cases where the provisioning module needs to do > something > >>>> >>> > different regarding private IPs such as if they are dynamically > >>>> >>> allocated. > >>>> >>> > The provisioning module would retrieve the IP when the image > gets > >>>> >>> loaded > >>>> >>> > by querying DNS or some other method and then update the > database. > >>>> The > >>>> >>> OS > >>>> >>> > module would then use this value to configure the computer. > >>>> >>> > >>>> >>> > >>>> >>> Sounds good. I agree that avoiding the hosts file would be best, > >>>> >>> especially > >>>> >>> in a case like this where it needs to be continuously modified. > Here's > >>>> our > >>>> >>> changed get_computer_private_ip_address, lines 45-55 are the new > code: > >>>> >>> > >>>> >>> https://gist.github.com/cmann/9896513 > >>>> >>> > >>>> >>> Finally we're still working on putting together more comprehensive > >>>> >>> installation documentation for anyone who wants to try it out. > We're > >>>> use > >>>> >>> automation for a number of the tasks and I haven't had a chance > to run > >>>> >>> through it from scratch yet. For anyone feeling adventurous these > are > >>>> the > >>>> >>> most basic instructions: > >>>> >>> > >>>> >>> 1. Apply the NAT patch > >>>> >>> (https://issues.apache.org/jira/browse/VCL-174tested on 2.3 and > 2.3.1, > >>>> >>> make sure to read the Google doc) > >>>> >>> 2. Follow the instructions in the readme in the Github repo for > the > >>>> >>> OpenStack module. > >>>> >>> > >>>> >>> Cameron > >>>> >>> > >>>> >>> > >>>> >>> On Fri, Mar 28, 2014 at 7:54 AM, Andy Kurth <[email protected]> > >>>> wrote: > >>>> >>> > >>>> >>> > > > >>>> >>> > > > >>>> >>> > > 2. We use the Try::Tiny module to handle exceptions since the > >>>> version > >>>> >>> of > >>>> >>> > > Perl VCL 2.3 uses doesn't support the saner try/catch > mechanisms > >>>> >>> > available > >>>> >>> > > in newer releases, however this would add an additional > dependency > >>>> to > >>>> >>> > VCL. > >>>> >>> > > Depending on the version of Perl VCL 2.4 is planning to use > what > >>>> are > >>>> >>> > > people's thoughts on using Try::Tiny versus switching to > Perl's > >>>> >>> internal > >>>> >>> > > mechanisms? > >>>> >>> > > > >>>> >>> > > >>>> >>> > I haven't looked at Try::Tiny or any newer exception handling > >>>> features > >>>> >>> of > >>>> >>> > Perl, but I'd prefer to keep the number of external modules to a > >>>> >>> minimum if > >>>> >>> > possible. If there are new features in Perl which would be > >>>> beneficial, > >>>> >>> I'd > >>>> >>> > rather make the newer version of Perl a requirement and use the > >>>> built in > >>>> >>> > features. Otherwise, we're locked into using an external > module, > >>>> >>> usually > >>>> >>> > from CPAN, which has no quality assurance. I regret adding > some of > >>>> the > >>>> >>> > current modules such as InsideOut, rather then writing the code > which > >>>> >>> could > >>>> >>> > accomplish the same objective. > >>>> >>> > > >>>> >>> > Did you need to catch exceptions because one of the OpenStack > modules > >>>> >>> > you're using may throw a die exception? When this happens, the > >>>> entire > >>>> >>> vcld > >>>> >>> > process for the reservation exits abruptly which causes problems > >>>> such as > >>>> >>> > the request state remaining in pending indefinitely. This > happens > >>>> even > >>>> >>> if > >>>> >>> > you wrap the call in an eval block. I have worked around this > >>>> problem > >>>> >>> > before by locally overriding the die handler. Take a look at > >>>> >>> > vSphere_SDK.pm and find lines such as: > >>>> >>> > > >>>> >>> > # Override the die handler > >>>> >>> > local $SIG{__DIE__} = sub{}; > >>>> >>> > > >>>> >>> > The vSphere SDK is one example which may call die (via Carp). > If > >>>> this > >>>> >>> > happens with die overridden, the eval block returns normally. > If an > >>>> >>> > exception occurred, $EVAL_ERROR should be set. > >>>> >>> > > >>>> >>> > > >>>> >>> > > > >>>> >>> > > 3. I've saved the best for last, but first some background. > The > >>>> >>> biggest > >>>> >>> > > change we've made is that we no longer rely on the /etc/hosts > file > >>>> at > >>>> >>> all > >>>> >>> > > (though we haven't removed the code that keeps it up to > date). We > >>>> had > >>>> >>> a > >>>> >>> > > number of issues with duplicate entries ending up in the > hosts file > >>>> >>> and > >>>> >>> > > weren't able to pin down the exact cause. This led us to > adding a > >>>> >>> table > >>>> >>> > to > >>>> >>> > > the database to track VCL computer to OpenStack instance > mappings. > >>>> The > >>>> >>> > next > >>>> >>> > > issue we ran into was the private IP address not being > populated in > >>>> >>> the > >>>> >>> > > hosts file. This happened very rarely and we're still unsure > of the > >>>> >>> cause > >>>> >>> > > especially since there's an explicit check that the address is > >>>> >>> populated > >>>> >>> > > with a value before we write the hosts file. I'd assume that > the > >>>> Nova > >>>> >>> API > >>>> >>> > > occasionally returns a response where the addresses component > is > >>>> >>> > structured > >>>> >>> > > differently than we expect, though I was never able to > directly > >>>> >>> observe > >>>> >>> > > this behavior. Since OpenStack automatically creates DNS > entries > >>>> for > >>>> >>> new > >>>> >>> > > instances we decided the best course of action was to bypass > the > >>>> hosts > >>>> >>> > file > >>>> >>> > > entirely. In the end we modified > get_computer_private_ip_address in > >>>> >>> > > DataStructure.pm to do a DNS lookup rather than looking in the > >>>> hosts > >>>> >>> > file. > >>>> >>> > > > >>>> >>> > > The question for the community is are changes to core VCL code > >>>> >>> > acceptable? > >>>> >>> > > Or should we find a solution that keeps everything contained > to the > >>>> >>> > > provisioning module? If changes to core code are acceptable > we'd > >>>> also > >>>> >>> > have > >>>> >>> > > to decide on a method of detecting when to do a DNS lookup. > Should > >>>> it > >>>> >>> > > depend on the provisioning module in use? A configuration > option? > >>>> >>> > > >>>> >>> > > >>>> >>> > I see no problems. Any barriers or difficulty you encounter > with the > >>>> >>> core > >>>> >>> > code should be addressed by improving the core code. > >>>> >>> > > >>>> >>> > Reworking the dependency on /etc/hosts is something I'd like to > >>>> improve > >>>> >>> > regardless. People often encounter problems with this if they > >>>> forget to > >>>> >>> > add entries to /etc/hosts for all their computers, including the > >>>> >>> management > >>>> >>> > node(s). I'd like to not read /etc/hosts on the management > node at > >>>> all > >>>> >>> and > >>>> >>> > have the value in the computer table be authoritative. This > should > >>>> >>> > facilitate cases where the provisioning module needs to do > something > >>>> >>> > different regarding private IPs such as if they are dynamically > >>>> >>> allocated. > >>>> >>> > The provisioning module would retrieve the IP when the image > gets > >>>> >>> loaded > >>>> >>> > by querying DNS or some other method and then update the > database. > >>>> The > >>>> >>> OS > >>>> >>> > module would then use this value to configure the computer. > >>>> >>> > > >>>> >>> > One comment regarding adding tables... great! Add all the > tables you > >>>> >>> think > >>>> >>> > are beneficial. I have seen some try to work around the current > >>>> >>> database > >>>> >>> > schema rather than adding to it. There's no reason for this. > If > >>>> there > >>>> >>> are > >>>> >>> > configuration options which apply only to OpenStack, add > something > >>>> like > >>>> >>> an > >>>> >>> > openstackconfig table. If the vmprofile table doesn't align, > add > >>>> >>> something > >>>> >>> > like an openstackprofile table. If there are general options > such as > >>>> >>> how > >>>> >>> > to handle DNS or private IP resolution, we could add a more > general > >>>> >>> table > >>>> >>> > or possibly extend the managementnode or computer table. > >>>> >>> > > >>>> >>> > Thanks, > >>>> >>> > Andy > >>>> >>> > > >>>> >>> > >>>> >> > >>>> >> > >>>> > >>>> > >>>> > >>>> -- > >>>> Twitter: @serverascode > >>>> Blog: serverascode.com > >>>> > >> > >> > >> > >> -- > >> Twitter: @serverascode > >> Blog: serverascode.com > > > > > > > > -- > > Aaron Peeler > > Program Manager > > Virtual Computing Lab > > NC State University > > > > All electronic mail messages in connection with State business which > > are sent to or received by this account are subject to the NC Public > > Records Law and may be disclosed to third parties. > > > > -- > Twitter: @serverascode > Blog: serverascode.com >
