Comments/questions in-line. > On Jun 22, 2016, at 7:58 PM, [email protected] wrote: > > Here is a high level functional spec for the split image feature. It include > a motivation, overview of design decisions, and a set of functional changes > that need to happen. I’d love to get feedback if you are interested. > > Its not quite a complete design, but there are still a few unknowns to work > out. My though is to get feedback through this week and start the > implementation next week. I was thinking about starting with the Bootloader > Changes, then the Newt Changes, then writing the loader_app and new newtmgr > commands. Anyone up for helping? > > Paul > > > Split Image Bootloader (SIB) > > This document describes the split image bootloader design. > > Status > > This document is currently draft an anxiously awaiting review. > > Motivation > > The goal is to create an application in two pieces to fit into two image > banks such that one piece would contain the bluetooth stack and firmware > update application and the other would contain the customer application > (that’s the goal, but it would be defined generally to allow any split). > These two are linked together with a special property that the upgrade app > (also called loader_app below) could run without the customer app but not > vice versa. To give these names I call the independent one the AIIC > (Application independent image component) and the customer app the ADIC > (Application Dependent Image component). > > Consider this example, two 112k flash banks which we want to store a > bluetooth application. Assume the following code sizes: Application size 32k, > Bluetooth stack size 64k, Upgrade code size 16k. With two independent app > images, each would be 112k filling the available sectors. However, if we > split the image we would have an upgrade image of 80k and an app image of 32k > (since it uses bluetooth and upgrade from the AIIC) leaving tons more space > for sophisticated applications and more space for the upgrade as well. > > We decided to create this split image as a pair that are linked together > during the build process. There will be no dynamic bindings like SWI or > function table or anything like that. The goal is not to separate the > OS/stack from the app, but just to be a bit more efficient about duplicating > the large components like bluetooth. Thus, when up are updating one of the > split images, you MUST update the other. Splitting the image makes this > “safe" as there will always be a bank filled with an image that is capable of > performing an upgrade. > > You can see some discussion below from Contiki and NUT below about how they > did the same thing (although some were later replaced with dynamic loading) > > High Level Operation > > The basis principle is to create two image components that make up a single > image. One component would include an image loader and the components > required to load the image (e.g. bluetooth or serial). The other component > would include the remainder of the application. A small bootloader would > verify the two images and boot the one that was relevant. > > There are three image components to the SIB: > > 1. A bootloader > 2. An application independent image component (AIIC) > 3. An application dependent image component (ADIC) > > There are four flash regions required for the SIP > > 1. the boot image region (BIR) > 2. the primary image region (PIR) > 3. the secondary image region (SIR) > 4. a scratch area region (SAR) > > The bootloader is intended to be programmed at manufacturing and will not be > reprogrammable in the field. It is always loaded into the BIR.
Is there a fall-back position to be able to load a new bootloader? There are
instances where a new bootloader may be needed (bug fixes, major upgrades,
etc.). Return to Factory for bootloader upgrades will be expensive and
difficult for large-scale deployments. Even for small deployments.
>
> The AIIC has the ability to upgrade itself and download new AIIC or ADICs
> into the SIR. The AIIC always runs from the PIR, but can be loaded into the
> SIR and copied to the PIR by the bootloader.
>
> The ADIC always loads and runs in the SIR. The ADIC cannot upgrade itself
> (well it may have code to do so, but has no-where to write a new image since
> it uses the SIR and requires the PIR to run).
>
> The AIIC and ADIC are not independent components. An upgrade must upgrade
> both of these components. That is, they will be statically linked to run as a
> single unit. However, the AIIC will be linked to be able to run without the
> ADIC in order to perform an upgrade of the AIIC and ADIC.
>
> The scratch area region (SAR) is used when swapping AIIC images to/from the
> SIR to the PIR.
>
> Requirements
>
> The following lists the user requirements for the booting of split images.
>
> 1. Use the Nordic Soft Device as part of the AIIC
> 2. Update the ADIC with only an AIIC present
> 3. Update the AIIC with only an AIIC present
> 4. Update the ADIC with the AIIC and ADIC present
> 5. Must support protection of system to ensure that only approved images
> can be loaded on device
>
> Upgrade Procedure
>
> An upgrade would work as follows.
>
> 1. An upgrade would trigger a reboot into AIIC-mode
> 2. The bootloader would detect AIIC mode and run the AIIC in independent
> mode
> 3. The AIIC would download a new AIIC into the SIR and restart
> 4. The bootloader would copy the SIR AIIC into the PIR and boot the PIR AIIC
> 5. This new AIIC will detect absense of a SIR image and download an ADIC
> into the SIR.
> 6. The system will restart and the bootloader will run the ADIC.
>
> Fallback Procedure
>
> The following would list potential fallbacks during failed upgrade
>
> 1. If the AIIC in the PIR fails to download a new AIIC, it could always
> download the old ADIC into the PIR to resstore behavior.
> 2. If the AIIC is able to download a new AIIC but the new AIIC doesn't run,
> the bootloader could fallback to the old AIIC (since it swapped them) and the
> restore the old ADIC.
From whence would it get this old ADIC? If I make changes to the AIIC and the
ADIC, recompile and deploy them, the old AIIC (on the host) and the old ADIC
(on the host and the device) are blown away. While the old AIIC is still
present on-device, the old ADIC is no longer present on-device or on-host, and
so cannot be restored. If the old ADIC is archived as part of the build process
then this may be possible, but then we have versioning and “how many old ADICs
do we archive?” problems. Also, since ADICs are AIIC-dependent, we have to
archive them as pairs.
> 3. if the new AIIC loads and runs, but is unable to download the new ADIC,
> it could download the old AIIC and restore the old ADIC
See above.
>
> Out of Scope
>
> 1. Use of the Nordic Bootloader. Current design will replace the nordic
> bootloader.
> 2. Binary Bluetooth stack image to faciliate certification. This is a
> separate issue and not considered.
> 3. More that two image regions. There may be use cases to have more than
> two image regions. For example, there could be one AIIC region and two ADIC
> regions with different applications liked against the same AIIC. An example
> would be game1/game2. We will not support this today and do what our simple
> bootloader does which is to look at two regions (PIR and SIR)
>
> Current Design Discussion
>
> Here are some tradeoffs considered during the design phase.
>
> Versioning
>
> How do we differentiate between the different pieces of a split image?
>
> There are several places in the system that need to do this. To bootloader
> simply needs to know whether an image is directly bootable or not.
>
> The AIIC needs to determine whether it and the ADIC are a perfect match since
> symbols are pre-linked between them
>
> The user (via newtmgr) needs to be able to identify this package as a single
> UniqueName.
>
> The solution is desribed below.
>
> Starting an Application Image
>
> We have a design choice where the AIIC sometimes runs without the ADIC or the
> ADIC and AIIC may neen to be run together. There is an architecture decision
> about how this happens. Here are some choices
>
> In all cases, the bootloader can read the preferences like it does now (from
> FCB or NFFS). In all cases, the AIIC needs all the net_mgr code and bluetooth
> (or serial code) to perform an image update of the SIR.
>
> Split Image Aware Bootloader
>
> In this case, the Bootloader looks at three possible image types
> (stand_alone, split_indep, split_app) for each region and determines what it
> allows to copy or run. If would need additional (from what we have now)logic
> to validate that the split image components match and then run the
> appropriate one depending on whether they are both present and valid.
>
> For this model, the bootloader must understand the split image concept and
> must perform the validation of the component parts. It would make sense here
> (I think) for the bootloader to boot directly to the main of the AIIC or ADIC
> depending on whether they were both valid and the status of the boot request
> (whether the user wanted to boot the AIIC only).
I’m confused at this point. Up top now, I was assuming (perhaps incorrectly)
that the AIIC was a runnable application that is started by the bootloader.
That it must always be runnable. The ADIC is a separate runnable application
that DEPENDS on the AIIC and may, or may not, actually be run. From this, it
sounds like the AIIC may, in some instances, not be started as an application
itself but that the ADIC would be started and simply reference the AIIC for
BLE, console, or other library functions. In the second case, how would an
upgrade, where the AIIC succeeds and the ADIC fails, ever work? If the AIIC is
not started by the bootloader because the ADIC is the runnable app, then …
>
> This method has the advantage that we only perform the security checing in
> the bootloader. It also means that we don't have to do any startup magic in
> the AIIC to decide whether to run the AIIC main or look for and run the ADIC
> main. Of course that same code would happen in the bootloader which is
> not-upgradable.
I do still have some heartburn about a non-upgradeable bootloader. The
following may be out of scope for this proposal, but since it involves an AIIC
location, it might need to be considered.
Here’s a scenario for an upgradeable bootloader:
1) Bootloader is signed with a SHA or PKI from a host computer — only
bootloaders signed with this key will be accepted except over direct
serial/jtag connection. If you have physical access to the device, you win.
This also allows a new host to ‘take over’ responsibility for delivering
bootloaders to devices by providing a new bootloader with a new SHA/PKI key.
2) new bootloader image is sent to the device and stored in SAR — could
be stored in SIR
a) new bootloader key is checked against current bootloader
key. If keys don’t match, new bootloader image is erased otherwise continue
b) Keys match, so new bootlaoder is written to PIR and device
reboots to new bootloader
c) if boot to new bootlaoder in PIR succeeds, PIR region is
written to BIR and reboots — we have successfully updated the bootloader
d) if boot to new bootloader in PIR fails, reboots to old
bootloader, and PIR is erased.
This has the advantage — by first loading the bootloader into SAR for key
validation — of ensuring that a rogue bootloader does not wipe out an existing
AIIC/ADIC pair. It required 3 reboots to field-upgrade a bootloader, but it
maintains system integrity. Using this method, bootloaders could be updated
over bluetooth as well, as long as the keys match, making filed upgrades of
bootloaders possible.
Anyone paying attention will note that this is eerily similar to the AIIC
upgrade procedure. :-)
>
> Pros: 1. no security code in the AIIC 2. simpler linking of AIIC and ADIC. In
> this case, the AIIC would be a stand-alone linked-program and the ADIC would
> be special linkage with 3. more rigid
>
> Independent Bootloader
>
> In this case the bootloader looks at two possible image types (runnable,
> non-runnable) in each region and determines what it is allowed to copy and
> run. It does not need logic to "match" the split image components.
>
> In the stand-alone image case, the bootloader would look at the boot
> preference and try to validate, copy and run the image specified. This is no
> different than today, except that it would ensure that the image that it was
> asked to run is "runnable".
>
> In the split image case, the bootloader would look at the boot preference and
> try to validate, copy and run the image specified, This is no different than
> today, except that it would ensure that the image that it was asked to run is
> "runnable".
>
> This looks like the easy approach, as the bootloader would be identical for
> split images and dual-images. However, it would mean that the AIIC image
> would have to lauch the ADIC image as part of its startup. Depending on what
> information is passed to it from the bootloader, it may have to re-compute
> the security of the ADIC, and also understand the flashmap etc. Its likely
> that the AIIC would link-in many of the same code as in the bootloader.
>
> Pros: 1. Simple bootloader 2. Easily support dual and split image 3. more
> flexibility in AIIC for dual-image behavior (i.e. this allows future models
> to use syscall or linked or something else). it would also allow more
> flexibility for compatibility between split images.
>
> Bootloader Decision
>
> The decision moving forward (for now) is to use the independent bootloader
> model and have the AIIC validate the ADIC and run it. For now, we'll do this
> through duplicating the authentication code and public key code from the
> bootloader. This design will proceed with anindependent bootloader
>
> Linker Magic
>
> To build this split image, the following steps will be taken.
>
> 1. The AIIC image will be built and linked to run in the PIR. If this
> succeeds (no unresolved externals and fits into the memory). This is a normal
> linkage and guarantees that we have a stand-alone image with no unresolved
> externals.
> 2. We will run a command on the AIIC.elf file to generate a list of
> external symbols that are defined. For example, we may do nm -g
> --defined-only aiic.elf which will generate somethimg that has a list of
> symbols with their addresses.
> * 0000db90 T base64_decode
> * 0000dac4 T base64_encode
> * 0000db60 T base64_pad
> * 00015000 T ble_att_clt_rx_error
> * ...
> 3. We will remove a few key functions from that list depending on the
> decision for the bootloader above that we need to repeat in the second image.
> * main(), etc.
> 4. We will parse this output and generate linker command file split_app.ld
> like this
> * base64_decode = 0000db90 ;
> * base64_encode = 0000dac4 ;
> * ...
> 5. We will include this file as part of the linker script before the
> library includes. INCLUDE(split_app.ld)
> 6. The linker will then use the existing symbols definitions from the AIIC
> to build the ADIC.
>
> This has the following benefits: 1. The AIIC will always be runnable (it will
> always have exactly what it needs and no more) 2. The ADIC image will be
> complete as well. It establishes a rule for building independent and
> dependent images
>
> Drawbacks: 1. There is limited control over the size of the AIIC. You would
> have to tell the AIIC linker script to include stuff that was not used to
> balance size between AIIC and ADIC. 2. Its not clear where unused global
> variables will go. Will this force all global variables into the AIIC? 3.
> Individual packages will be split between ADIC and AIIC.
>
> Build Process
>
> The question I am investigating is how to formulate this in the newt build
> process. Here are two alternatives.
>
> Single Target build
>
> I was thinking that I would have a single target to produce both of these
> image components. The target would still have one bsp, one build profile, but
> I would define a new token for the target called loader=xxx. This would
> define the loader AIIC that would go into the first image bank, and if
> “loader" was defined, the app would be linked to run in the second image
> bank. Thus, if I were building a bluetooth app called foo, I would set
> loader=AIIC (contains upgrade and bluetooth) and then set app=foo. If
> “loader” was not defined, I would ensure the newt tool would do a complete
> build of foo and link it into the first image bank.
>
> Separate targets
>
> The alternative would be to have two separate targets. The ADIC target would
> have to reference the AIIC target and know that it was being built to be
> dependent on that bit. In this case, the AIIC target would be build as normal
> with no changes to the build system. Then the ADIC target would have to have
> a new tag that said that it was dependent on the other target. For example,
> in the app:foo target descriptor it would say split_image_dep=AIIC. This
> would cause newt to perform some special processing of the AIIC elf image,
> and ensure that we link the foo ADIC app to run out of the second image bank
> and reference common functions in the first image bank.
>
> In either case, the application will be build as a split image and will
> contain two elf files. One file will be the Application Independent AIIC)
> part, and another will be the application dependent (ADIC) part. These will
> both get converted to image files by newt create-image.
>
> Target Decision
>
> My thoughts are that IDEA1 is the better of these two. It would allow newt to
> be split-image-aware during create-image and during download and debug. This
> design will proceed with a single target build
>
> Image Verification
>
> We need a way to prove that the two images are indeed mates. Since the ADIC
> is essentially statically linked to the AIIC, the AIIC must be bit for bit
> exactly what the ADIC expects for code to run properly.
>
> To accomplish this requires a slight modification to the current design.
> Today each of the images has a SHA checksum that is used to validate that the
> image is unmodified from build time.
>
> The SHA in the form of a TLV that is added to the end of the image.
>
> AIICs will not change. Its SHA will be computed over its image and appended
> as it is now.
>
> When ADICs are build, their SHA checksum will be computed over the AIIC and
> ADIC image. This serves two purposes. First when the AIIC wants to run and
> ADIC it can verify 100% that the ADIC was meant to run with the exact AIIC
> image. Second, a bootloader that gets an ADIC image, will not run it since
> its SHA will not match (computed only over its image)
>
> Split Image Upgrade State
>
> It may take several system boots to perform a split image upgrade. State
> needs to be communicated between these boots. State will be communicated via
> a sysconfig variable.
>
> when newtmgr initiates and upgrade of a split image, it will set the
> sysconfig split_state to AIIC and then reboot the system. Upon rebooting, the
> bootloader will boot the AIIC which will see the split_state and run its
> internal application (rather than booting ADIC). In this state, new_mgr can
> erase the second image slot and download a new AIIC. Once complete
> successfully and the new AIIC is verified (SHA), the current AIIC will set
> the split state to AIIC and tell the bootloader to swap and run the alternate
> AIIC (in test mode) and reboot. When it reboots into AIIC mode, the newtmgr
> can check to ensure the right AIIC is loaded and, commit it (non-test-mode)
> then load the ADIC into the bank 2. Once loaded, it will set the split_state
> to 'ADIC'.
>
> Functional Spec
>
> This section describes the functional specification for the Split Image
> Bootloader. Its mean to describe the set of changes to each system component,
> without getting into the design of each component.
>
> Image Header
>
> The image header will be augmented with another flag in the ih_flags field:
>
> IMAGE_F_DEPENDENT
>
>
> This will signify to the bootloader that this is not a runnable image and
> that this image cannot be copied into the PIR of the flashmap.
>
> This will signify to the AIIC image that this image may be its co-image (if
> its version matches as well).
>
> When building an AIIC, the SHA will be calcualted over the contents of the
> AIIC image.
>
> When building a ADIC, the SHA will be calculated over the contents of the
> AIIC and ADIC.
>
> Bootloader/Boot Util
>
> The bootloader will be augmented to check the IMAGE_F_NON_BOOTABLE flag. It
> will not copy a non-bootable image into the PIR and will not boot a
> non-bootable image.
>
> If a non-bootable image winds up in the PIR, and there is a valid bootable
> iamge in the SIR, the bootloader will swap them (This should not happen)
>
> The bootloader will NOT verify SHA or PKI on non-bootable images since it
> won't boot them. It will only verify these on bootable images.
>
> NOTE: These changes are minor and intended to allow the bootloader to be
> simple and compatible with dual images or split images.
>
> The bootloader will not know about the split image scheme, but only know
> about two types of images, bootable and non-bootable.
>
> The bootloader current stores version numbers in the sysconfig. We can still
> use this method (but only boot images that also have BOOTABLE), but time
> permitting, I will make the sysconfig in the bootloader rely only on the hash
> values of the image TLV instead of the verison number. This would remove
> dependency on the uniqueness of the version number.
>
> boot Util Library
>
> The bootutil library will be augmented to allow the following. These may be
> used by the loader_app .
>
> 1) Ability to verify a combined AIIC and ADIC together. 2) Ability to boot
> into a ADIC
>
> Newt
>
> Newt is the system build utility and package manager.
>
> Target Descriptor
>
> Newt will be augmented with a additional field in the target.
>
> loader_app=@apache-mynewt-core/apps/loader
>
>
> This will tell newt that it is to split the image loader from the application
> and create a split image.
>
> Newt Build
>
> If loader_app is not defined in the target, the build will proceed as it does
> today. If loader_app is defined, the following additional steps will be taken:
>
> 1. Before the build of the app, newt will build the the loader_app
> application (with the same rules as the app). One exception is that the for
> loader_app it will use the primary.ld linker script, which links to the first
> flash bank for that target. The loader_app build components will be placed in
> the bin/<target>/apps/<loader_app_name> directory.
> 2. Then, newt will generate an loader_app.ld file which contains the
> address of all the symbols in loader_app_name except for a few key ones like
> main, startup and maybe a few others which will be removed so they are
> duplicated in the main app.
> 3. Now newt builds the app as before, but if loader_app is defined, uses
> the secondary.ld linker script.
> 4. This linker script has an INCLUDE(loader_app.ld) directive to use the
> addresses for the components already included in theloader_app. (NOTE: Not
> sure about paths and how to get the app to find the path to the other
> directory.
>
> Boot code will still be built with a special bootloader linker script.
>
> Newt Clean
>
> Will be expanded to ensure it cleans the loader and the the app both.
>
> Newt Create-Image
>
> This component needs to be modified to generate the appropriate SHA hashes on
> both the loader_app and app.
>
> Both the loader_app (optionally) and app will be built into images with the
> version presented to the create-image command. For each individual image, the
> SHA hash of the image will be appended as a TLV to the image.
>
> The a build ID hash will be generated from the hash of each image component.
> This hash will be appended to both images as aBUILD_ID_TLV.
>
> This is done to ensure that either image can be identified as being part of
> the build, and that the set of images can be identified as a perfect match
> before running them.
>
> The create image code will also be modified to create a donwload.json file
> that contains a small set of rules for downloading these image components to
> the target. For example, this may include something like: 1. {Boot PIR: AIIC}
> 1. {Load SIR: { Image:loader_app.img, hash:23738941793274912749327497 }} 2.
> {Boot SIR: AIIC} 3. {Load SIR: { Image:bletiny.img,
> hash:787658576876857687568765587 }} 4. {Boot PIR: ADIC}
>
> This rules file defines the set of rules that the loader must use when
> loading images into this device. This will be used by newt load to perform
> the load operation.
>
> For legacy dual images, the rules will be
>
> 1. {Load SIR { Image:loader_app.img, hash:23738941793274912749327497 }}
> 2. {Boot SIR: AIIC}
>
> Newt Load
>
> Newt Load will be modified to read the rules file and execute the rules to
> load the target.
>
> Newt Debug
>
> This part is still unkown. TODO.. Its not clear how to get the symbols for
> both images into GDB at the same time. I suspect we will have to loade the
> symbols from the AIIC and ADIC separately.
>
> Newt Run
>
> This will just roll up the changes from the other new commands.
>
> Newt Size
>
> This must report the size of each bank image separately if there are two
>
> Mynewt Loader (AIIC implementaiton)
>
> A Loader application will be built in the apache-mynewt-core which will allow
> upgrade over serial port newtmgr and bluetooth (via the Profile that Vipul is
> writing). The loader will boot and make a first decision (possible after
> starting mynewt) whether it will run as a stand alone applicaiton or whether
> it will bypass its main routing and continue to boot a companion ADIC.
>
> This will be a sample which can be used by mynewt users to create their own
> loader_app or they can use this app as a stock split image loader.
>
> It will contain the following existing components
>
> 1. The bluetooth stack as a periperhal
> 2. The newtmgr bluetooth profile
> 3. Optionally the newmgr serial protocol
> 4. imgmgr Library
> 5. sysconfig
>
> The mynewt loader will have the following abilities:
>
> 1. Read the flash images and determine whether the SIR is matching or not
> via SHA.
> 2. boot into the main of the second image depending on the split_state in
> sysconfig
> 3. Handle the newtmgr commands to: inspect, program, and erase the SIR.
> 4. Pass image boot information to the booloader to activate and swap images.
> 5. Expose newtmgr interface via bluetooth and/or serial
>
> NewtMgr
>
> Newtmgr will be expanded to handle new image commands related to split image.
>
> Specifically
>
> 1. Ability to tell the image, that upon boot, go into AIIC or ADIC mode.
>
> TODO
>
> These items are still TBD
>
> 1. How to performing this magic linking. Make sure to test before
> committing to this development path. The trick is going to be to ensure that
> we don't get duplicate symbols in the ADIC
> 2. How to debug a split image. See what we can get GDB to understand about
> two images. Look up how to GDB with ROM images. Seeadd-symbol-file for a good
> start.
>
> Geek Notes
>
> this links to a geeky paper on the topic
> http://dunkels.com/adam/dunkels06runtime.pdf<http://dunkels.com/adam/dunkels06runtime.pdf>
>
> NOTEs From Contiki
>
> There are two aspects I'd like to share with you:
>
> 1. The only actual implementation I know of was done for 6502 based
> platforms using the cc65 toolchain. I presume that this isn't the platform
> your're interested in ;-) However for your reference here's the URL:
> http://contiki.cvs.sourceforge.net/viewvc/checkout/contiki/contiki-cc65/make-labels
>
> 2. The approach works in general like this:
>
> * Link the system core in the "usual" manner.
> * Get yourself a list of all public symbols and their corresponding
> addresses - obviously this only works for a non-relocatable binary.
> * Write some script that converts this list to an assembler source file
> that exports those symbols as aliases to their addresses in the system core.
> * Create an object file from the assembler source.
> * Put that object file on the linker cmdline when linking binary
> modules.
>
> This approach has two effects:
>
> 1. The Contiki APIs implemented by the system core become available to the
> modules making them work.
> 2. The C-library / compiler-runtime-library functions already present in
> the system core don't get replicated but rather reused in the modules making
> them much smaller. This is because linkers look at object files on the
> cmdline before looking into archives when resolving symbols.
>
> Notes From NUT
>
> I did this for BTnut respectively Nut/OS (I sent Rick my diploma theses, but
> not to the list as it is in German). You forgot that the address space needs
> to be adjusted, as the code must not be linked into memory already used by
> the core. Also I didn't take the detour using the object file, but used a
> customized linkerscript, which also takes care of the some other adjustments
> which have to be taken into account.
>
> For those who are not able to read Moritz' superb thesis:
>
> 1. Build the kernel and link it
> 2. Extract the symbol table from the ELF file by using objdump -t
> 3. Convert the symbol table into the form SYMBOLNAME = ADDRESS
> 4. Add this file to the linkerscript by using the option -T in gnu-ld and
> link the desired source files
> 5. You may want to write a custom LinkerScript to link the code to a custom
> address space.
> 6. Tadaaa, you have a relocated executable module
--
David G. Simmons
(919) 534-5099
Web • Blog • Linkedin • Twitter • GitHub
/** Message digitally signed for security and authenticity.
* If you cannot read the PGP.sig attachment, please go to
* http://www.gnupg.com/ Secure your email!!!
* Public key available at keyserver.pgp.com
**/
♺ This email uses 100% recycled electrons. Don't blow it by printing!
There are only 2 hard things in computer science: Cache invalidation, naming
things, and off-by-one errors.
signature.asc
Description: Message signed with OpenPGP using GPGMail
