Hi Curtis, I think you can change the default lease time for dnsmasq protocol in /etc/dnsmasq.conf. It seems the default dhcp lease time is 150 (150 seconds I guess).
Thanks, Young-Hyun On Wed, Jun 4, 2014 at 11:52 AM, Curtis <[email protected]> wrote: > Also, as another note/issue, in openstack the dhcp times are usually > set to a minute or two, which I find quite low but that is what it is > by default. For windows, during the provisioning process VCL sets the > time which causes the dhcp ticket to time out and the instance is left > without an IP and reservations fail. Right now I have commented out > the part of the code that sets the time. > > On Tue, Jun 3, 2014 at 2:21 PM, Aaron Peeler <[email protected]> > wrote: > > It was Andy's idea :) > > > > Aaron > > > > On Tue, Jun 3, 2014 at 4:19 PM, YOUNG OH <[email protected]> > wrote: > >> I think so too. > >> > >> Young-Hyun > >> > >> > >> On Tue, Jun 3, 2014 at 3:58 PM, Curtis <[email protected]> wrote: > >> > >>> On Tue, Jun 3, 2014 at 1:52 PM, Aaron Peeler <[email protected]> > >>> wrote: > >>> > Would using the private IP instead of hostnames resolve it? It would > >>> > take a bit of work, but could be done. > >>> > >>> Yup that would work...as far as I can tell. :) > >>> > >>> Thanks, > >>> Curtis. > >>> > >>> > > >>> > Aaron > >>> > > >>> > > >>> > > >>> > On Tue, Jun 3, 2014 at 3:22 PM, Curtis <[email protected]> > wrote: > >>> >> On Tue, Jun 3, 2014 at 11:53 AM, YOUNG OH <[email protected]> > >>> wrote: > >>> >>> Hi Curtis, > >>> >>> > >>> >>> Thank you for updating the status and correction about the > OpenStack > >>> Perl > >>> >>> SDKs. > >>> >>> > >>> >>> If DNS lookup is not reliable to find the private ip of an instance > >>> for a > >>> >>> multi-host openstack environment, then do you have any suggestion > to > >>> deal > >>> >>> with this issue? > >>> >>> > >>> >>> At this point, I think at least we have an agreement that we do NOT > >>> want to > >>> >>> use /etc/hosts anymore, right? > >>> >>> If so, we can first try to use the database to get/update the > private > >>> ip > >>> >>> address with DNS lookup for a single-host openstack environment. > After > >>> >>> that, we can try to find a solution to manage the DNS lookup or > other > >>> >>> methods of finding the private ip of an instance in multi-host > >>> openstack > >>> >>> environment. If you have any other suggestions, please let me know. > >>> >>> > >>> >> > >>> >> I'm not too sure what to do. I'm hoping the Apache VCL core devs > chime > >>> in. > >>> >> > >>> >> As far as I can tell, Apache VCL needs DNS resolution to work, so > >>> >> simply adding the IP into the database isn't going to be enough. > >>> >> > >>> >> For the current Apache VCL + OpenStack multihost cloud installation > >>> >> I'm working on I'm just going to add a python script that queries > >>> >> OpenStack and generates /etc/hosts every 20 or 30 seconds or > something > >>> >> like that. > >>> >> > >>> >> Thanks, > >>> >> Curtis. > >>> >> > >>> >>> Thanks, > >>> >>> Young-Hyun > >>> >>> > >>> >>> > >>> >>> > >>> >>> On Mon, Jun 2, 2014 at 3:56 PM, Curtis <[email protected]> > wrote: > >>> >>> > >>> >>>> On Mon, Jun 2, 2014 at 12:45 PM, YOUNG OH < > [email protected]> > >>> wrote: > >>> >>>> > Hi All, > >>> >>>> > > >>> >>>> > I would like to bring your attention again to discuss what > OpenStack > >>> >>>> module > >>> >>>> > should include. I would like to discus two issues first. > >>> >>>> > > >>> >>>> > First, we know that using /etc/hosts to get the private ip of an > >>> instance > >>> >>>> > can be problematic. Specifically, Cameron (at Cybera) and I had > >>> found > >>> >>>> that > >>> >>>> > there might duplicated or blank entries and rarely but OpenStack > >>> APIs > >>> >>>> > cannot get the private ip (not populated) of an instance > properly. > >>> For > >>> >>>> the > >>> >>>> > later case, Cameron added DNS Lookup function to get the correct > >>> private > >>> >>>> IP > >>> >>>> > (Cameron, correct me if I am wrong). This problem leads us to > avoid > >>> using > >>> >>>> > /etc/hosts file to get the private ip but I think we need to > first > >>> >>>> discuss > >>> >>>> > what the best way is to handle the private ip: > >>> >>>> > > >>> >>>> > (a). Use the privateIPaddress field in computer table. > >>> >>>> > Andy suggests that we can change the get_computer_private_ip to > get > >>> >>>> private > >>> >>>> > ip from the database instead of using /etc/hosts (Andy, correct > me > >>> if I > >>> >>>> am > >>> >>>> > wrong). I agree with his idea because this can avoid duplicated > or > >>> blank > >>> >>>> > entry problem of using /etc/hosts. > >>> >>>> > > >>> >>>> > (b). OpenStack (DNS lookup for finding unpopulated private ip) > >>> >>>> > In our openstack module, DNS lookup or other methods (if any) > can > >>> give > >>> >>>> the > >>> >>>> > correct the private ip of the instance in order to deal with the > >>> problem > >>> >>>> > for the private ip not being populated in the hosts file. > Cameron > >>> >>>> modified > >>> >>>> > the get_computer_private_ip in DataStructure.pm to do a DNS > lookup > >>> >>>> instead > >>> >>>> > of using the hosts file. But I would like to suggest that if > >>> possible, it > >>> >>>> > can be better to add the DNS lookup function to openstack module > >>> rather > >>> >>>> > than changing the DataStructure.pm. That is, we can update the > >>> >>>> > privateIPaddress field in computer table using DNS lookup > instead of > >>> >>>> > injecting the private ip to /etc/hosts in openstack module. > >>> >>>> > >>> >>>> We've just found out recently that the helpful DNS resolution that > >>> >>>> openstack provides does not work well in a mutlihost openstack > >>> >>>> environment, so can't be relied on. In multihost mode each compute > >>> >>>> host only knows about the vms running on it locally. > >>> >>>> > >>> >>>> > > >>> >>>> > I think, if possible, one of the best methods for dealing with > the > >>> >>>> private > >>> >>>> > ip is that we can use the database to get the private ip of an > >>> instance > >>> >>>> and > >>> >>>> > using DNS lookup in openstack module to update the information. > Any > >>> >>>> > suggestions and corrections about my opinion would be very > helpful > >>> to > >>> >>>> solve > >>> >>>> > this problem. > >>> >>>> > > >>> >>>> > Second, currently we have two versions (not exactly but for > >>> convenience); > >>> >>>> > one uses OpenStack command line interfaces (CLIs) and one uses > >>> OpenStack > >>> >>>> > Perl SDKs. As I understand, the Perl SDKs are wrapper APIs for > the > >>> >>>> > OpenStcak CLIs and required to install through CPAN. I think > both > >>> >>>> versions > >>> >>>> > have pros and cons. So, I would like to ask your opinions > whether > >>> it is > >>> >>>> > better to include two different openstack versions to the next > VCL > >>> >>>> release, > >>> >>>> > to find a way to merge two versions into one openstack module, > OR to > >>> >>>> create > >>> >>>> > a higher level of openstack module so that it can choose one > version > >>> >>>> based > >>> >>>> > on user's environment. Any suggestions would be also very > helpful to > >>> >>>> decide > >>> >>>> > how we can manage openstack module. Thank you. > >>> >>>> > >>> >>>> FYI -- The perl openstack sdk is not a wrapper for the command > line > >>> >>>> utils. It uses the openstack api, not the command line tools. > >>> >>>> > >>> >>>> Thanks, > >>> >>>> Curtis. > >>> >>>> > >>> >>>> > > >>> >>>> > Best regards, > >>> >>>> > Young-Hyun > >>> >>>> > > >>> >>>> > > >>> >>>> > > >>> >>>> > On Mon, Jun 2, 2014 at 2:22 PM, YOUNG OH < > [email protected]> > >>> >>>> wrote: > >>> >>>> > > >>> >>>> >> Hi All, > >>> >>>> >> > >>> >>>> >> I have tested my openstack module ( > >>> >>>> >> https://issues.apache.org/jira/browse/VCL-590) on the > OpenStack > >>> >>>> Icehouse > >>> >>>> >> (The ninth release of OpenStack, April 17, 2014) and I could > >>> >>>> provisioning a > >>> >>>> >> Linux image through it. > >>> >>>> >> > >>> >>>> >> The environment details are as follows: > >>> >>>> >> > >>> >>>> >> 1. Host Blade > >>> >>>> >> :- CPU: Intel(R) Xeon(R) CPU E5645 @ 2.40GHz > >>> >>>> >> :- Memory: 49G > >>> >>>> >> :- Disk: 134G > >>> >>>> >> :- OS: RHEL 6.4 x64 with KVM hypervisor > >>> >>>> >> 2. OpenStack Icehouse Components > >>> >>>> >> :- Compute > >>> >>>> >> :- Network (Nova-Network, not OpenStack Neutron) > >>> >>>> >> :- Glance > >>> >>>> >> 3. Provisioning Linux image > >>> >>>> >> :- CentOS 6.3 x64 > >>> >>>> >> 4. VCL 2.3 > >>> >>>> >> > >>> >>>> >> In this test, I've just tested the provisioning an image but I > >>> would > >>> >>>> like > >>> >>>> >> to post the detail information to jira once I finish other > test. In > >>> >>>> >> addition, I will try to add a function of selecting OpenStack > >>> flavors > >>> >>>> based > >>> >>>> >> on the size of an image. The current openstack module sets > flavors > >>> for > >>> >>>> each > >>> >>>> >> image because the previous OpenStack releases (Essex as I know) > >>> didn't > >>> >>>> >> support the size information of images but now it supports. I > will > >>> also > >>> >>>> >> post to jira issue when I add the function to my module. Thank > you. > >>> >>>> >> > >>> >>>> >> Best regards, > >>> >>>> >> Young-Hyun > >>> >>>> >> > >>> >>>> >> > >>> >>>> >> > >>> >>>> >> On Mon, Mar 31, 2014 at 12:55 PM, Cameron Mann < > >>> [email protected]> > >>> >>>> >> wrote: > >>> >>>> >> > >>> >>>> >>> > > >>> >>>> >>> > I haven't looked at Try::Tiny or any newer exception > handling > >>> >>>> features > >>> >>>> >>> of > >>> >>>> >>> > Perl, but I'd prefer to keep the number of external modules > to a > >>> >>>> >>> minimum if > >>> >>>> >>> > possible. If there are new features in Perl which would be > >>> >>>> beneficial, > >>> >>>> >>> I'd > >>> >>>> >>> > rather make the newer version of Perl a requirement and use > the > >>> >>>> built in > >>> >>>> >>> > features. Otherwise, we're locked into using an external > >>> module, > >>> >>>> >>> usually > >>> >>>> >>> > from CPAN, which has no quality assurance. I regret adding > >>> some of > >>> >>>> the > >>> >>>> >>> > current modules such as InsideOut, rather then writing the > code > >>> which > >>> >>>> >>> could > >>> >>>> >>> > accomplish the same objective. > >>> >>>> >>> > Did you need to catch exceptions because one of the > OpenStack > >>> modules > >>> >>>> >>> > you're using may throw a die exception? When this happens, > the > >>> >>>> entire > >>> >>>> >>> vcld > >>> >>>> >>> > process for the reservation exits abruptly which causes > problems > >>> >>>> such as > >>> >>>> >>> > the request state remaining in pending indefinitely. This > >>> happens > >>> >>>> even > >>> >>>> >>> if > >>> >>>> >>> > you wrap the call in an eval block. I have worked around > this > >>> >>>> problem > >>> >>>> >>> > before by locally overriding the die handler. Take a look > at > >>> >>>> >>> > vSphere_SDK.pm and find lines such as: > >>> >>>> >>> > # Override the die handler > >>> >>>> >>> > local $SIG{__DIE__} = sub{}; > >>> >>>> >>> > The vSphere SDK is one example which may call die (via > Carp). > >>> If > >>> >>>> this > >>> >>>> >>> > happens with die overridden, the eval block returns > normally. > >>> If an > >>> >>>> >>> > exception occurred, $EVAL_ERROR should be set. > >>> >>>> >>> > >>> >>>> >>> > >>> >>>> >>> The OpenStack SDK throws exceptions when requests that > >>> >>>> >>> delete/modify/create > >>> >>>> >>> resources fail, no die exceptions though. There wasn't any > >>> specific > >>> >>>> >>> feature > >>> >>>> >>> that made me choose Try::Tiny, it was more my unfamiliarity > with > >>> >>>> exception > >>> >>>> >>> handling in Perl which seemed to have a number of gotchas > prior to > >>> >>>> 5.14 so > >>> >>>> >>> I figured it'd be better to let someone else's code worry > about > >>> that. > >>> >>>> >>> > >>> >>>> >>> > >>> >>>> >>> > >>> >>>> > >>> > http://blog.twoshortplanks.com/2011/06/06/unexceptional-exceptions-in-perl-5-14/ > >>> >>>> >>> > >>> >>>> >>> I see no problems. Any barriers or difficulty you encounter > with > >>> the > >>> >>>> core > >>> >>>> >>> > code should be addressed by improving the core code. > >>> >>>> >>> > Reworking the dependency on /etc/hosts is something I'd > like to > >>> >>>> improve > >>> >>>> >>> > regardless. People often encounter problems with this if > they > >>> >>>> forget to > >>> >>>> >>> > add entries to /etc/hosts for all their computers, > including the > >>> >>>> >>> management > >>> >>>> >>> > node(s). I'd like to not read /etc/hosts on the management > >>> node at > >>> >>>> all > >>> >>>> >>> and > >>> >>>> >>> > have the value in the computer table be authoritative. This > >>> should > >>> >>>> >>> > facilitate cases where the provisioning module needs to do > >>> something > >>> >>>> >>> > different regarding private IPs such as if they are > dynamically > >>> >>>> >>> allocated. > >>> >>>> >>> > The provisioning module would retrieve the IP when the > image > >>> gets > >>> >>>> >>> loaded > >>> >>>> >>> > by querying DNS or some other method and then update the > >>> database. > >>> >>>> The > >>> >>>> >>> OS > >>> >>>> >>> > module would then use this value to configure the computer. > >>> >>>> >>> > >>> >>>> >>> > >>> >>>> >>> Sounds good. I agree that avoiding the hosts file would be > best, > >>> >>>> >>> especially > >>> >>>> >>> in a case like this where it needs to be continuously > modified. > >>> Here's > >>> >>>> our > >>> >>>> >>> changed get_computer_private_ip_address, lines 45-55 are the > new > >>> code: > >>> >>>> >>> > >>> >>>> >>> https://gist.github.com/cmann/9896513 > >>> >>>> >>> > >>> >>>> >>> Finally we're still working on putting together more > comprehensive > >>> >>>> >>> installation documentation for anyone who wants to try it out. > >>> We're > >>> >>>> use > >>> >>>> >>> automation for a number of the tasks and I haven't had a > chance > >>> to run > >>> >>>> >>> through it from scratch yet. For anyone feeling adventurous > these > >>> are > >>> >>>> the > >>> >>>> >>> most basic instructions: > >>> >>>> >>> > >>> >>>> >>> 1. Apply the NAT patch > >>> >>>> >>> (https://issues.apache.org/jira/browse/VCL-174tested on 2.3 > and > >>> 2.3.1, > >>> >>>> >>> make sure to read the Google doc) > >>> >>>> >>> 2. Follow the instructions in the readme in the Github repo > for > >>> the > >>> >>>> >>> OpenStack module. > >>> >>>> >>> > >>> >>>> >>> Cameron > >>> >>>> >>> > >>> >>>> >>> > >>> >>>> >>> On Fri, Mar 28, 2014 at 7:54 AM, Andy Kurth < > [email protected]> > >>> >>>> wrote: > >>> >>>> >>> > >>> >>>> >>> > > > >>> >>>> >>> > > > >>> >>>> >>> > > 2. We use the Try::Tiny module to handle exceptions since > the > >>> >>>> version > >>> >>>> >>> of > >>> >>>> >>> > > Perl VCL 2.3 uses doesn't support the saner try/catch > >>> mechanisms > >>> >>>> >>> > available > >>> >>>> >>> > > in newer releases, however this would add an additional > >>> dependency > >>> >>>> to > >>> >>>> >>> > VCL. > >>> >>>> >>> > > Depending on the version of Perl VCL 2.4 is planning to > use > >>> what > >>> >>>> are > >>> >>>> >>> > > people's thoughts on using Try::Tiny versus switching to > >>> Perl's > >>> >>>> >>> internal > >>> >>>> >>> > > mechanisms? > >>> >>>> >>> > > > >>> >>>> >>> > > >>> >>>> >>> > I haven't looked at Try::Tiny or any newer exception > handling > >>> >>>> features > >>> >>>> >>> of > >>> >>>> >>> > Perl, but I'd prefer to keep the number of external modules > to a > >>> >>>> >>> minimum if > >>> >>>> >>> > possible. If there are new features in Perl which would be > >>> >>>> beneficial, > >>> >>>> >>> I'd > >>> >>>> >>> > rather make the newer version of Perl a requirement and use > the > >>> >>>> built in > >>> >>>> >>> > features. Otherwise, we're locked into using an external > >>> module, > >>> >>>> >>> usually > >>> >>>> >>> > from CPAN, which has no quality assurance. I regret adding > >>> some of > >>> >>>> the > >>> >>>> >>> > current modules such as InsideOut, rather then writing the > code > >>> which > >>> >>>> >>> could > >>> >>>> >>> > accomplish the same objective. > >>> >>>> >>> > > >>> >>>> >>> > Did you need to catch exceptions because one of the > OpenStack > >>> modules > >>> >>>> >>> > you're using may throw a die exception? When this happens, > the > >>> >>>> entire > >>> >>>> >>> vcld > >>> >>>> >>> > process for the reservation exits abruptly which causes > problems > >>> >>>> such as > >>> >>>> >>> > the request state remaining in pending indefinitely. This > >>> happens > >>> >>>> even > >>> >>>> >>> if > >>> >>>> >>> > you wrap the call in an eval block. I have worked around > this > >>> >>>> problem > >>> >>>> >>> > before by locally overriding the die handler. Take a look > at > >>> >>>> >>> > vSphere_SDK.pm and find lines such as: > >>> >>>> >>> > > >>> >>>> >>> > # Override the die handler > >>> >>>> >>> > local $SIG{__DIE__} = sub{}; > >>> >>>> >>> > > >>> >>>> >>> > The vSphere SDK is one example which may call die (via > Carp). > >>> If > >>> >>>> this > >>> >>>> >>> > happens with die overridden, the eval block returns > normally. > >>> If an > >>> >>>> >>> > exception occurred, $EVAL_ERROR should be set. > >>> >>>> >>> > > >>> >>>> >>> > > >>> >>>> >>> > > > >>> >>>> >>> > > 3. I've saved the best for last, but first some > background. > >>> The > >>> >>>> >>> biggest > >>> >>>> >>> > > change we've made is that we no longer rely on the > /etc/hosts > >>> file > >>> >>>> at > >>> >>>> >>> all > >>> >>>> >>> > > (though we haven't removed the code that keeps it up to > >>> date). We > >>> >>>> had > >>> >>>> >>> a > >>> >>>> >>> > > number of issues with duplicate entries ending up in the > >>> hosts file > >>> >>>> >>> and > >>> >>>> >>> > > weren't able to pin down the exact cause. This led us to > >>> adding a > >>> >>>> >>> table > >>> >>>> >>> > to > >>> >>>> >>> > > the database to track VCL computer to OpenStack instance > >>> mappings. > >>> >>>> The > >>> >>>> >>> > next > >>> >>>> >>> > > issue we ran into was the private IP address not being > >>> populated in > >>> >>>> >>> the > >>> >>>> >>> > > hosts file. This happened very rarely and we're still > unsure > >>> of the > >>> >>>> >>> cause > >>> >>>> >>> > > especially since there's an explicit check that the > address is > >>> >>>> >>> populated > >>> >>>> >>> > > with a value before we write the hosts file. I'd assume > that > >>> the > >>> >>>> Nova > >>> >>>> >>> API > >>> >>>> >>> > > occasionally returns a response where the addresses > component > >>> is > >>> >>>> >>> > structured > >>> >>>> >>> > > differently than we expect, though I was never able to > >>> directly > >>> >>>> >>> observe > >>> >>>> >>> > > this behavior. Since OpenStack automatically creates DNS > >>> entries > >>> >>>> for > >>> >>>> >>> new > >>> >>>> >>> > > instances we decided the best course of action was to > bypass > >>> the > >>> >>>> hosts > >>> >>>> >>> > file > >>> >>>> >>> > > entirely. In the end we modified > >>> get_computer_private_ip_address in > >>> >>>> >>> > > DataStructure.pm to do a DNS lookup rather than looking > in the > >>> >>>> hosts > >>> >>>> >>> > file. > >>> >>>> >>> > > > >>> >>>> >>> > > The question for the community is are changes to core VCL > code > >>> >>>> >>> > acceptable? > >>> >>>> >>> > > Or should we find a solution that keeps everything > contained > >>> to the > >>> >>>> >>> > > provisioning module? If changes to core code are > acceptable > >>> we'd > >>> >>>> also > >>> >>>> >>> > have > >>> >>>> >>> > > to decide on a method of detecting when to do a DNS > lookup. > >>> Should > >>> >>>> it > >>> >>>> >>> > > depend on the provisioning module in use? A configuration > >>> option? > >>> >>>> >>> > > >>> >>>> >>> > > >>> >>>> >>> > I see no problems. Any barriers or difficulty you encounter > >>> with the > >>> >>>> >>> core > >>> >>>> >>> > code should be addressed by improving the core code. > >>> >>>> >>> > > >>> >>>> >>> > Reworking the dependency on /etc/hosts is something I'd > like to > >>> >>>> improve > >>> >>>> >>> > regardless. People often encounter problems with this if > they > >>> >>>> forget to > >>> >>>> >>> > add entries to /etc/hosts for all their computers, > including the > >>> >>>> >>> management > >>> >>>> >>> > node(s). I'd like to not read /etc/hosts on the management > >>> node at > >>> >>>> all > >>> >>>> >>> and > >>> >>>> >>> > have the value in the computer table be authoritative. This > >>> should > >>> >>>> >>> > facilitate cases where the provisioning module needs to do > >>> something > >>> >>>> >>> > different regarding private IPs such as if they are > dynamically > >>> >>>> >>> allocated. > >>> >>>> >>> > The provisioning module would retrieve the IP when the > image > >>> gets > >>> >>>> >>> loaded > >>> >>>> >>> > by querying DNS or some other method and then update the > >>> database. > >>> >>>> The > >>> >>>> >>> OS > >>> >>>> >>> > module would then use this value to configure the computer. > >>> >>>> >>> > > >>> >>>> >>> > One comment regarding adding tables... great! Add all the > >>> tables you > >>> >>>> >>> think > >>> >>>> >>> > are beneficial. I have seen some try to work around the > current > >>> >>>> >>> database > >>> >>>> >>> > schema rather than adding to it. There's no reason for > this. > >>> If > >>> >>>> there > >>> >>>> >>> are > >>> >>>> >>> > configuration options which apply only to OpenStack, add > >>> something > >>> >>>> like > >>> >>>> >>> an > >>> >>>> >>> > openstackconfig table. If the vmprofile table doesn't > align, > >>> add > >>> >>>> >>> something > >>> >>>> >>> > like an openstackprofile table. If there are general > options > >>> such as > >>> >>>> >>> how > >>> >>>> >>> > to handle DNS or private IP resolution, we could add a more > >>> general > >>> >>>> >>> table > >>> >>>> >>> > or possibly extend the managementnode or computer table. > >>> >>>> >>> > > >>> >>>> >>> > Thanks, > >>> >>>> >>> > Andy > >>> >>>> >>> > > >>> >>>> >>> > >>> >>>> >> > >>> >>>> >> > >>> >>>> > >>> >>>> > >>> >>>> > >>> >>>> -- > >>> >>>> Twitter: @serverascode > >>> >>>> Blog: serverascode.com > >>> >>>> > >>> >> > >>> >> > >>> >> > >>> >> -- > >>> >> Twitter: @serverascode > >>> >> Blog: serverascode.com > >>> > > >>> > > >>> > > >>> > -- > >>> > Aaron Peeler > >>> > Program Manager > >>> > Virtual Computing Lab > >>> > NC State University > >>> > > >>> > All electronic mail messages in connection with State business which > >>> > are sent to or received by this account are subject to the NC Public > >>> > Records Law and may be disclosed to third parties. > >>> > >>> > >>> > >>> -- > >>> Twitter: @serverascode > >>> Blog: serverascode.com > >>> > > > > > > > > -- > > Aaron Peeler > > Program Manager > > Virtual Computing Lab > > NC State University > > > > All electronic mail messages in connection with State business which > > are sent to or received by this account are subject to the NC Public > > Records Law and may be disclosed to third parties. > > > > -- > Twitter: @serverascode > Blog: serverascode.com >
