In the end, the surgery we do (to method_named), is to replace the prior $op's next() pointer to point at the $gv op we copied from a known subroutine's op-tree (that uses a typical subroutine call instead of a method lookup). Since we relocated that next() pointer, we need to decrement the internal refcnt for the method_named op to avoid leaks/segfaults during garbage collection of the ithread.
None of the many issues Doug faced back in 2000 to do this on a more generic level actually need to be done for this implementation. You just need to know that any code that tries to walk this tree (eg B::Deparse) post-tweaks may choke on the zombie method_named op lying around in one of the sibling() linked lists. That probably includes the ithread-cloniing mechanism itself, so only use :sealed post ithread construction, not prior to it. On Wed, Aug 31, 2022 at 10:27 AM Joe Schaefer <j...@sunstarsys.com> wrote: > The :sealed attribute is a statement about a class's @ISA: you are saying > you are using its compile-time setting. > And because we invoke $class->can to do the lookup, module authors who > override UNIVERSAL::can() with a customized > one can support AUTOLOADed methods themselves. > > Under :sealed, you control which lexical object references you want to use > compile-time method lookups (via can()), > by declaring them with a class (type), or avoiding doing that in the > lexical's declaration. It only impacts your subroutine > definitions that declare :sealed and a typed lexicals, not any other > module's code elsewhere. > > You absolutely *can* use sealed.pm outside mod_perl, but you need to be > wary about using typed lexicals on your method > argument stack, because end-users may pass properly derived objects to > your method, and will expect your module to use > the derived-class's overrides of any method calls in your codebase. > Basically the law of the land is that consumers of an API > expect all method calls to operate the way *virtual* method calls work in > Java/C++. However, what you do internally with API's > outside of your published API's argument stack is fair game for :sealed. > My advice that it's only practical to seal XS method calls remains. > > On Wed, Aug 31, 2022 at 9:52 AM Joe Schaefer <j...@sunstarsys.com> wrote: > >> Submitted a Pull Request for the Generate.xs patch: >> https://github.com/rurban/b-generate/pull/2 >> Added more comments to sealed.pm to explain the rationale behind the # >> replace $methop logic, >> since it differs from what Doug did back in 2000. >> >> On Tue, Aug 30, 2022 at 2:52 PM Joe Schaefer <j...@sunstarsys.com> wrote: >> >>> Take a walk down history lane with me here: >>> https://www.perl.com/pub/2000/06/dougpatch.html/ >>> >>> Like ithreads, the idea was sparked from Gurusamy's genius, coded up by >>> Doug, and largely forgotten by p5p politics. >>> It's not that it couldn't be done, they arrived at the place where it >>> *shouldn't* be done, which was deflating for mod_perl fans. >>> Simon Couzens made a lot of inroads since, with modularized Perl >>> compilers and B::Generate, but it wasn't until >>> Perl7 that I was motivated to try any way forward, on a more limited, >>> controllable scale. >>> >>> What do you think? How should this piece of the mod_perl puzzle fit in >>> to the CPAN universe? >>> >>> On Tue, Aug 30, 2022 at 2:12 PM Joe Schaefer <j...@sunstarsys.com> wrote: >>> >>>> Every method call that's implemented in XS is looked-up at compile-time >>>> in that script, even for class methods. >>>> That's the sweet spot for :sealed. The only other things I do with it >>>> are a few hot methods in Dotiac::DTL::Core, but that's probably not worth >>>> the bother. >>>> >>>> On Tue, Aug 30, 2022 at 1:14 PM Joe Schaefer <j...@sunstarsys.com> >>>> wrote: >>>> >>>>> Just look through my commit history on this sample Registry script to >>>>> see what's involved in getting sealed activated on your scripts. >>>>> >>>>> https://github.com/SunStarSys/cms/blob/master/enquiry.pl >>>>> >>>>> On Tue, Aug 30, 2022 at 1:12 PM Joe Schaefer <j...@sunstarsys.com> >>>>> wrote: >>>>> >>>>>> It'd be pretty harmless to apply the RegistryCooker.pm patch once we >>>>>> find a home for sealed.pm (either in this project or as a >>>>>> stand-alone pragma on CPAN). >>>>>> Nothing will segfault without consciously using types on your lexical >>>>>> object reference declarations; otherwise it's a giant noop. >>>>>> >>>>>> On Tue, Aug 30, 2022 at 12:53 PM Joe Schaefer <j...@sunstarsys.com> >>>>>> wrote: >>>>>> >>>>>>> If you really beat the hell out of it thread-wise, sealed.pm v4.0.0 >>>>>>> will still segfault, but there's not much more I can do with the code at >>>>>>> this point to prevent that. >>>>>>> B::Generate doesn't really support what I'm doing here, which is to >>>>>>> do surgery on an existing op-tree, instead of just playing around with a >>>>>>> user-generated one. >>>>>>> There's no good way to remove the target "method_named" op that >>>>>>> we're replacing with a gv_op. If that can't be supported using B::OP >>>>>>> APIs, >>>>>>> then it should >>>>>>> be handled from perl itself. The problem is that the politics >>>>>>> around the feature were never resolved, because nobody wants to change >>>>>>> the >>>>>>> default "virtual method" >>>>>>> behavior of Perl's OO-runtime-lookups. Now with the new :sealed >>>>>>> SUBROUTINE ATTRIBUTE, it's only enabled for people (like us) who want it >>>>>>> conditionally applied, >>>>>>> just like we do for the :method attribute. >>>>>>> >>>>>>> On Tue, Aug 30, 2022 at 11:26 AM Joe Schaefer <j...@sunstarsys.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Someday this patch might be interesting: >>>>>>>> >>>>>>>> diff -u RegistryCooker.pm~ RegistryCooker.pm >>>>>>>> --- RegistryCooker.pm~ 2022-08-30 11:10:19.790171019 -0400 >>>>>>>> +++ RegistryCooker.pm 2022-08-30 11:12:34.319572045 -0400 >>>>>>>> @@ -399,7 +399,8 @@ >>>>>>>> my $eval = join '', >>>>>>>> 'package ', >>>>>>>> $self->{PACKAGE}, ";", >>>>>>>> - "sub handler {", >>>>>>>> + "use base 'sealed';", >>>>>>>> + "sub handler :Sealed {", >>>>>>>> "local \$0 = '$script_name';", >>>>>>>> $nph, >>>>>>>> $shebang, >>>>>>>> >>>>>>>> On Mon, Aug 29, 2022 at 2:21 PM Joe Schaefer <j...@sunstarsys.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Forgive me for the pent up frustration of having our wonderful >>>>>>>>> mod_perl project being completely ignored and abandoned by the Perl >>>>>>>>> Steering Committee's frivolous lingustic interests over the years >>>>>>>>> since the >>>>>>>>> Parrot announcement. >>>>>>>>> SaywerX gave us a reason to be hopeful again. Let's see what they >>>>>>>>> do with it. >>>>>>>>> >>>>>>>>> On Mon, Aug 29, 2022 at 1:34 PM Joe Schaefer <j...@sunstarsys.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> If the Perl steering committee had any brains left it would have >>>>>>>>>> capitalized on the perl 5.34 release and Co announced modperl2 >>>>>>>>>> ithread >>>>>>>>>> compatibility now available with Perl7’s new release. >>>>>>>>>> >>>>>>>>>> Instead they are going to kick the tires on the defaults for >>>>>>>>>> strictures and warnings until nobody cares any more. >>>>>>>>>> >>>>>>>>>> Get Outlook for iOS <https://aka.ms/o0ukef> >>>>>>>>>> ------------------------------ >>>>>>>>>> *From:* Joe Schaefer <j...@sunstarsys.com> >>>>>>>>>> *Sent:* Monday, August 29, 2022 1:17:17 PM >>>>>>>>>> *To:* mod_perl list <modperl@perl.apache.org> >>>>>>>>>> *Subject:* Re: sealed.pm v4.0.0 is out >>>>>>>>>> >>>>>>>>>> The only reason I’ve been vacillating about glibc/malloc thread >>>>>>>>>> safety is because I couldn’t fathom the fact that people still >>>>>>>>>> believed >>>>>>>>>> modperl isn’t compatible with mpm_event at this point in the Perl7 >>>>>>>>>> storyline. The old segfaults of the past that happened in glibc >>>>>>>>>> malloc >>>>>>>>>> were because Perl was corrupting the heap in some other part of the >>>>>>>>>> codebase, and there’s no simple way to track it down without a tool >>>>>>>>>> like >>>>>>>>>> Valgrind, but we weren’t successful with that effort either. >>>>>>>>>> >>>>>>>>>> Get Outlook for iOS <https://aka.ms/o0ukef> >>>>>>>>>> ------------------------------ >>>>>>>>>> *From:* Joe Schaefer <j...@sunstarsys.com> >>>>>>>>>> *Sent:* Monday, August 29, 2022 1:08:00 PM >>>>>>>>>> *To:* mod_perl list <modperl@perl.apache.org> >>>>>>>>>> *Subject:* Re: sealed.pm v4.0.0 is out >>>>>>>>>> >>>>>>>>>> Religiously avoid setting up per request ithread environment >>>>>>>>>> variables. Just use PerlSetEnv in your Webserver config. Everything >>>>>>>>>> we did >>>>>>>>>> in modperl to support CGI scripts is a train wreck. >>>>>>>>>> >>>>>>>>>> Get Outlook for iOS <https://aka.ms/o0ukef> >>>>>>>>>> ------------------------------ >>>>>>>>>> *From:* Joe Schaefer <j...@sunstarsys.com> >>>>>>>>>> *Sent:* Monday, August 29, 2022 1:04:03 PM >>>>>>>>>> *To:* mod_perl list <modperl@perl.apache.org> >>>>>>>>>> *Subject:* Re: sealed.pm v4.0.0 is out >>>>>>>>>> >>>>>>>>>> Look into reducing the scope of your interpreters down from the >>>>>>>>>> request level to the handler level. If all you are doing is running >>>>>>>>>> registry scripts, you will get even better scaling out of just a few >>>>>>>>>> ithreads per worker process. >>>>>>>>>> >>>>>>>>>> Get Outlook for iOS <https://aka.ms/o0ukef> >>>>>>>>>> ------------------------------ >>>>>>>>>> *From:* Joe Schaefer <j...@sunstarsys.com> >>>>>>>>>> *Sent:* Monday, August 29, 2022 12:57:14 PM >>>>>>>>>> *To:* mod_perl list <modperl@perl.apache.org> >>>>>>>>>> *Subject:* Re: sealed.pm v4.0.0 is out >>>>>>>>>> >>>>>>>>>> The only impact to your work with modperl is that you will need >>>>>>>>>> to assess the ithread-safety of your dependent XS-based modules. For >>>>>>>>>> example, use a JSON::XS thread safe alternative- there are several. >>>>>>>>>> >>>>>>>>>> Get Outlook for iOS <https://aka.ms/o0ukef> >>>>>>>>>> ------------------------------ >>>>>>>>>> *From:* Joe Schaefer <j...@sunstarsys.com> >>>>>>>>>> *Sent:* Monday, August 29, 2022 12:49:22 PM >>>>>>>>>> *To:* mod_perl list <modperl@perl.apache.org> >>>>>>>>>> *Subject:* Re: sealed.pm v4.0.0 is out >>>>>>>>>> >>>>>>>>>> There is a mountain of awful advice floating around about >>>>>>>>>> ithreads, including pretty much everything going on in Raku around >>>>>>>>>> adopting >>>>>>>>>> the node.js model instead. It is safe to ignore all that now that >>>>>>>>>> SawyerX >>>>>>>>>> spit polished all of the perl5 internals. >>>>>>>>>> >>>>>>>>>> Get Outlook for iOS <https://aka.ms/o0ukef> >>>>>>>>>> ------------------------------ >>>>>>>>>> *From:* Joe Schaefer <j...@sunstarsys.com> >>>>>>>>>> *Sent:* Monday, August 29, 2022 12:40:43 PM >>>>>>>>>> *To:* mod_perl list <modperl@perl.apache.org> >>>>>>>>>> *Subject:* Re: sealed.pm v4.0.0 is out >>>>>>>>>> >>>>>>>>>> Many of the performance hacks we’ve encouraged over the years, eg >>>>>>>>>> around HTTPD’s lingering close effect, are obsoleted with ithreads. >>>>>>>>>> Unless >>>>>>>>>> you send flush buckets down the output filter stack yourself, the >>>>>>>>>> “response >>>>>>>>>> handler” phase exits long before the “connection handler” starts >>>>>>>>>> making non >>>>>>>>>> blocking socket system calls. So you need an order of magnitude >>>>>>>>>> fewer >>>>>>>>>> ithreads than you do prefork children in a multitier arch. >>>>>>>>>> >>>>>>>>>> Get Outlook for iOS <https://aka.ms/o0ukef> >>>>>>>>>> ------------------------------ >>>>>>>>>> *From:* Joe Schaefer <j...@sunstarsys.com> >>>>>>>>>> *Sent:* Sunday, August 28, 2022 11:09:14 AM >>>>>>>>>> *To:* mod_perl list <modperl@perl.apache.org> >>>>>>>>>> *Subject:* Re: sealed.pm v4.0.0 is out >>>>>>>>>> >>>>>>>>>> Benchmark ran on my 2021 Dell Precision Laptop w/ 8 cores + HT >>>>>>>>>> (so 16vCPU) and Ubuntu 22.04 inside WSL2. Never topped 50% avg CPU, >>>>>>>>>> and >>>>>>>>>> almost all of the CPU was in userland (not system calls). >>>>>>>>>> >>>>>>>>>> On Sat, Aug 27, 2022 at 11:42 AM <j...@sunstarsys.com> wrote: >>>>>>>>>> >>>>>>>>>> See https://sunstarsys.com/essays/perl7-sealed-lexicals. For >>>>>>>>>> the full effect, you will need to build B::Generate with this patched >>>>>>>>>> version instead: >>>>>>>>>> https://github.com/SunStarSys/cms/blob/master/Generate.xs >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Sample mod_perl config + benchmarks: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> <IfModule mpm_event_module> >>>>>>>>>> >>>>>>>>>> StartServers 2 >>>>>>>>>> >>>>>>>>>> MinSpareThreads 100 >>>>>>>>>> >>>>>>>>>> MaxSpareThreads 500 >>>>>>>>>> >>>>>>>>>> ThreadLimit 1000 >>>>>>>>>> >>>>>>>>>> ThreadsPerChild 100 >>>>>>>>>> >>>>>>>>>> MaxRequestWorkers 1000000 >>>>>>>>>> >>>>>>>>>> MaxConnectionsPerChild 0 >>>>>>>>>> >>>>>>>>>> </IfModule> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> <IfModule mod_perl.c> >>>>>>>>>> >>>>>>>>>> PerlSwitches -T -I/home/joesuf4/src/cms/lib >>>>>>>>>> >>>>>>>>>> PerlInterpStart 2 >>>>>>>>>> >>>>>>>>>> PerlInterpMax 4 >>>>>>>>>> >>>>>>>>>> PerlInterpMinSpare 1 >>>>>>>>>> >>>>>>>>>> PerlInterpMaxSpare 4 >>>>>>>>>> >>>>>>>>>> PerlInterpMaxRequests 1000000 >>>>>>>>>> >>>>>>>>>> PerlOptions +GlobalRequest >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> <Directory /home/joesuf4/src/cms> >>>>>>>>>> >>>>>>>>>> Require all granted >>>>>>>>>> >>>>>>>>>> AddHandler perl-script .pl >>>>>>>>>> >>>>>>>>>> PerlResponseHandler ModPerl::Registry >>>>>>>>>> >>>>>>>>>> Options +ExecCGI >>>>>>>>>> >>>>>>>>>> </Directory> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> <Directory /home/joesuf4/src/trunk/content> >>>>>>>>>> >>>>>>>>>> Require all granted >>>>>>>>>> >>>>>>>>>> </Directory> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> <VirtualHost *:80> >>>>>>>>>> >>>>>>>>>> ServerName localhost >>>>>>>>>> >>>>>>>>>> DocumentRoot /home/joesuf4/src/trunk/content >>>>>>>>>> >>>>>>>>>> Alias /perl-script /home/joesuf4/src/cms >>>>>>>>>> >>>>>>>>>> </VirtualHost> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> </IfModule> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> ab -n 10000 -c 1000 http://localhost/perl-script/enquiry.pl >>>>>>>>>> >>>>>>>>>> This is ApacheBench, Version 2.3 <$Revision: 1879490 $> >>>>>>>>>> >>>>>>>>>> Copyright 1996 Adam Twiss, Zeus Technology Ltd, >>>>>>>>>> http://www.zeustech.net/ >>>>>>>>>> >>>>>>>>>> Licensed to The Apache Software Foundation, >>>>>>>>>> http://www.apache.org/ >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Benchmarking localhost (be patient) >>>>>>>>>> >>>>>>>>>> Completed 1000 requests >>>>>>>>>> >>>>>>>>>> Completed 2000 requests >>>>>>>>>> >>>>>>>>>> Completed 3000 requests >>>>>>>>>> >>>>>>>>>> Completed 4000 requests >>>>>>>>>> >>>>>>>>>> Completed 5000 requests >>>>>>>>>> >>>>>>>>>> Completed 6000 requests >>>>>>>>>> >>>>>>>>>> Completed 7000 requests >>>>>>>>>> >>>>>>>>>> Completed 8000 requests >>>>>>>>>> >>>>>>>>>> Completed 9000 requests >>>>>>>>>> >>>>>>>>>> Completed 10000 requests >>>>>>>>>> >>>>>>>>>> Finished 10000 requests >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Server Software: Apache/2.4.52 >>>>>>>>>> >>>>>>>>>> Server Hostname: localhost >>>>>>>>>> >>>>>>>>>> Server Port: 80 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Document Path: /perl-script/enquiry.pl >>>>>>>>>> >>>>>>>>>> Document Length: 1329 bytes >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Concurrency Level: 1000 >>>>>>>>>> >>>>>>>>>> Time taken for tests: 1.218 seconds >>>>>>>>>> >>>>>>>>>> Complete requests: 10000 >>>>>>>>>> >>>>>>>>>> Failed requests: 0 >>>>>>>>>> >>>>>>>>>> Total transferred: 15010000 bytes >>>>>>>>>> >>>>>>>>>> HTML transferred: 13290000 bytes >>>>>>>>>> >>>>>>>>>> Requests per second: 8207.94 [#/sec] (mean) >>>>>>>>>> >>>>>>>>>> Time per request: 121.833 [ms] (mean) >>>>>>>>>> >>>>>>>>>> Time per request: 0.122 [ms] (mean, across all concurrent >>>>>>>>>> requests) >>>>>>>>>> >>>>>>>>>> Transfer rate: 12031.37 [Kbytes/sec] received >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Connection Times (ms) >>>>>>>>>> >>>>>>>>>> min mean[+/-sd] median max >>>>>>>>>> >>>>>>>>>> Connect: 0 2 6.2 0 24 >>>>>>>>>> >>>>>>>>>> Processing: 4 93 49.6 82 458 >>>>>>>>>> >>>>>>>>>> Waiting: 1 80 44.5 71 455 >>>>>>>>>> >>>>>>>>>> Total: 17 95 49.5 84 458 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Percentage of the requests served within a certain time (ms) >>>>>>>>>> >>>>>>>>>> 50% 84 >>>>>>>>>> >>>>>>>>>> 66% 100 >>>>>>>>>> >>>>>>>>>> 75% 112 >>>>>>>>>> >>>>>>>>>> 80% 120 >>>>>>>>>> >>>>>>>>>> 90% 147 >>>>>>>>>> >>>>>>>>>> 95% 173 >>>>>>>>>> >>>>>>>>>> 98% 233 >>>>>>>>>> >>>>>>>>>> 99% 318 >>>>>>>>>> >>>>>>>>>> 100% 458 (longest request) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> % pgrep -f apache2 | xargs -n1 ps -uwww >>>>>>>>>> >>>>>>>>>> USER PID %CPU %MEM VSZ RSS TTY STAT START >>>>>>>>>> TIME COMMAND >>>>>>>>>> >>>>>>>>>> root 442827 0.0 0.1 18180 14244 ? Ss 11:27 >>>>>>>>>> 0:00 /usr/sbin/apache2 -k start >>>>>>>>>> >>>>>>>>>> USER PID %CPU %MEM VSZ RSS TTY STAT START >>>>>>>>>> TIME COMMAND >>>>>>>>>> >>>>>>>>>> www-data 446387 1.7 1.5 7549352 129692 ? Sl 11:28 >>>>>>>>>> 0:12 /usr/sbin/apache2 -k start >>>>>>>>>> >>>>>>>>>> USER PID %CPU %MEM VSZ RSS TTY STAT START >>>>>>>>>> TIME COMMAND >>>>>>>>>> >>>>>>>>>> www-data 451006 15.2 1.5 7483708 128468 ? Sl 11:39 >>>>>>>>>> 0:10 /usr/sbin/apache2 -k start >>>>>>>>>> >>>>>>>>>> USER PID %CPU %MEM VSZ RSS TTY STAT START >>>>>>>>>> TIME COMMAND >>>>>>>>>> >>>>>>>>>> www-data 451317 11.7 1.4 7483772 119836 ? Sl 11:39 >>>>>>>>>> 0:07 /usr/sbin/apache2 -k start >>>>>>>>>> >>>>>>>>>> USER PID %CPU %MEM VSZ RSS TTY STAT START >>>>>>>>>> TIME COMMAND >>>>>>>>>> >>>>>>>>>> www-data 451629 6.4 1.3 7483804 113012 ? Sl 11:39 >>>>>>>>>> 0:03 /usr/sbin/apache2 -k start >>>>>>>>>> >>>>>>>>>> USER PID %CPU %MEM VSZ RSS TTY STAT START >>>>>>>>>> TIME COMMAND >>>>>>>>>> >>>>>>>>>> www-data 451929 1.1 1.4 7483816 116668 ? Sl 11:39 >>>>>>>>>> 0:00 /usr/sbin/apache2 -k start >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Joe Schaefer, Ph.D. >>>>>>>>>> We only build what you need built. >>>>>>>>>> <j...@sunstarsys.com> >>>>>>>>>> 954.253.3732 <//954.253.3732> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Joe Schaefer, Ph.D. >>>>>>>>> We only build what you need built. >>>>>>>>> <j...@sunstarsys.com> >>>>>>>>> 954.253.3732 <//954.253.3732> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Joe Schaefer, Ph.D. >>>>>>>> We only build what you need built. >>>>>>>> <j...@sunstarsys.com> >>>>>>>> 954.253.3732 <//954.253.3732> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Joe Schaefer, Ph.D. >>>>>>> We only build what you need built. >>>>>>> <j...@sunstarsys.com> >>>>>>> 954.253.3732 <//954.253.3732> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> Joe Schaefer, Ph.D. >>>>>> We only build what you need built. >>>>>> <j...@sunstarsys.com> >>>>>> 954.253.3732 <//954.253.3732> >>>>>> >>>>>> >>>>>> >>>>> >>>>> -- >>>>> Joe Schaefer, Ph.D. >>>>> We only build what you need built. >>>>> <j...@sunstarsys.com> >>>>> 954.253.3732 <//954.253.3732> >>>>> >>>>> >>>>> >>>> >>>> -- >>>> Joe Schaefer, Ph.D. >>>> We only build what you need built. >>>> <j...@sunstarsys.com> >>>> 954.253.3732 <//954.253.3732> >>>> >>>> >>>> >>> >>> -- >>> Joe Schaefer, Ph.D. >>> We only build what you need built. >>> <j...@sunstarsys.com> >>> 954.253.3732 <//954.253.3732> >>> >>> >>> >> >> -- >> Joe Schaefer, Ph.D. >> We only build what you need built. >> <j...@sunstarsys.com> >> 954.253.3732 <//954.253.3732> >> >> >> > > -- > Joe Schaefer, Ph.D. > We only build what you need built. > <j...@sunstarsys.com> > 954.253.3732 <//954.253.3732> > > > -- Joe Schaefer, Ph.D. We only build what you need built. <j...@sunstarsys.com> 954.253.3732 <//954.253.3732>
