Re: [Discuss-gnuradio] [UHD] Notes on FPGA images packaging, and packaging in general

[Martin Braun]

On 03/08/2018 11:33 AM, Sid Hayn wrote:
> https://github.com/EttusResearch/uhddev/blob/master/images/manifest.txt
> 
> This address is invalid, assuming you meant this one?
> 
> https://github.com/EttusResearch/uhd/blob/master/images/manifest.txt

Yes! Thanks for pointing that out.

-- M
> 
> Thanks,
> Zero
> 
> On Wed, Mar 7, 2018 at 8:12 PM, Martin Braun  > wrote:
> 
> Hi all,
> 
> we've had a bunch of great discussions on the packaging of FPGA images
> (binaries) and on packaging UHD in general. Since there were many
> different discussions, I'd like to summarize them all in one big email.
> Thanks to Rick, Maitland, and Phil for providing feedback.
> 
> If you just use UHD via git, none of this affects you.
> 
> ## Release tags
> 
> Going forward, we'll change the format of the release tags. Before, it
> was release_003_011_000_000 and so on. Now, it'll be v3.11.0.0, which is
> a much more standard format. Personally, I'm happy to make this change
> -- we were hesitant to change it for consistency's sake, but it seems no
> one actually cares about the old format anyway. If you're pulling from
> github, this means you get nice, reproducible URLs like this:
> https://github.com/EttusResearch/uhd/releases/tag/v3.11.0.0
> 
> 
> ...and the tarball for the sources:
> https://github.com/EttusResearch/uhd/archive/v3.11.0.0.tar.gz
> 
> 
> ...and so on. I'd be interested to see who cares about the old tag
> format. For now, we'll double-tag (and we might re-tag old versions,
> too, if there's interest) to keep things easy.
> 
> ## Getting FPGA images for releases
> 
> We have two paths here. First, we have the uhd_images_downloader format
> (I'll talk more about that below). You can grab images from the
> images/manifest.txt file, but I understand that's cumbersome.
> 
> So, on top of the uhd_images_downloader format, we're considering
> bringing back an archive per release, but we're still figuring out the
> details of this.
> 
> We have for the sake of feedback uploaded an archive here:
> https://github.com/EttusResearch/uhd/releases/tag/v3.11.0.0
> 
> 
> I'd love to know if this format is suitable for people, and if things
> are missing. If this is all everyone needs, we'll find a way to automate
> this and make it error-proof.
> 
> However, this means that you will be downloading *all* default images
> (currently, that's over 200 MB after unpacking), some of which rarely or
> never change (e.g., the USRP1 image will always be the same).
> If you want to tailor the images you download (e.g., you might only care
> about the embedded images), pulling the individual packages might be a
> better option. The problem is, we don't have an easy way of identifying
> which ones there are without requiring to run a Python script at package
> build time.
> Something like this works, if you have access to the UHD source code
> (bash code):
> 
> SRC_URI=http://files.ettus.com/binaries/cache/`cat
>  images/manifest.txt |
> grep -v '#' | awk -F' ' '{print $3}' | grep b200mini`
> 
> ...and it would limit the images to the b200mini images, but it's not
> exactly handy. If people have suggestions, I'd be curious to hear them.
> 
> ## FPGA Versions
> 
> There was a question why we don't version our FPGA images (e.g., why
> does the X300 image not have version 1.2.3, and we say this version
> works with UHD 3.11.0.0). There are several problems with this: First,
> we have over 40 FPGA images (not all are packaged in the default image).
> Versioning them all correctly is already a hard logistical issue.
> Second, how do we create meaningful version numbers? The FPGA images
> consist of many subcomponents, all of which would need to go into an
> image version. The X300 and X310 images, for example, should always have
> similar versions when they have similar behaviour. But what if we fix
> something only for X300? Then, both the X310 and X300 images have 5
> variants (XG, HG, XA, HA, and AA). What if we fix a 10GigE issue? How do
> we keep the 10 version numbers for the {X300, X310} X {HG, XG, HA, AA,
> XA} in sync? Then of course, we have components on all X-Series that we
> also use on the E310. So that makes the version-number-space even more
> higher-dimensional. And so on.
> Finally, we consider the FPGA source code part of UHD. That means, for
> version 3.11.0.0 of UHD, there is exactly 1 commit on the FPGA
> repository that matches the source code. The images built off of this
> commit are the ones 

Re: [Discuss-gnuradio] [UHD] Notes on FPGA images packaging, and packaging in general

[Sid Hayn]

https://github.com/EttusResearch/uhddev/blob/master/images/manifest.txt

This address is invalid, assuming you meant this one?

https://github.com/EttusResearch/uhd/blob/master/images/manifest.txt

Thanks,
Zero

On Wed, Mar 7, 2018 at 8:12 PM, Martin Braun  wrote:

> Hi all,
>
> we've had a bunch of great discussions on the packaging of FPGA images
> (binaries) and on packaging UHD in general. Since there were many
> different discussions, I'd like to summarize them all in one big email.
> Thanks to Rick, Maitland, and Phil for providing feedback.
>
> If you just use UHD via git, none of this affects you.
>
> ## Release tags
>
> Going forward, we'll change the format of the release tags. Before, it
> was release_003_011_000_000 and so on. Now, it'll be v3.11.0.0, which is
> a much more standard format. Personally, I'm happy to make this change
> -- we were hesitant to change it for consistency's sake, but it seems no
> one actually cares about the old format anyway. If you're pulling from
> github, this means you get nice, reproducible URLs like this:
> https://github.com/EttusResearch/uhd/releases/tag/v3.11.0.0
>
> ...and the tarball for the sources:
> https://github.com/EttusResearch/uhd/archive/v3.11.0.0.tar.gz
>
> ...and so on. I'd be interested to see who cares about the old tag
> format. For now, we'll double-tag (and we might re-tag old versions,
> too, if there's interest) to keep things easy.
>
> ## Getting FPGA images for releases
>
> We have two paths here. First, we have the uhd_images_downloader format
> (I'll talk more about that below). You can grab images from the
> images/manifest.txt file, but I understand that's cumbersome.
>
> So, on top of the uhd_images_downloader format, we're considering
> bringing back an archive per release, but we're still figuring out the
> details of this.
>
> We have for the sake of feedback uploaded an archive here:
> https://github.com/EttusResearch/uhd/releases/tag/v3.11.0.0
>
> I'd love to know if this format is suitable for people, and if things
> are missing. If this is all everyone needs, we'll find a way to automate
> this and make it error-proof.
>
> However, this means that you will be downloading *all* default images
> (currently, that's over 200 MB after unpacking), some of which rarely or
> never change (e.g., the USRP1 image will always be the same).
> If you want to tailor the images you download (e.g., you might only care
> about the embedded images), pulling the individual packages might be a
> better option. The problem is, we don't have an easy way of identifying
> which ones there are without requiring to run a Python script at package
> build time.
> Something like this works, if you have access to the UHD source code
> (bash code):
>
> SRC_URI=http://files.ettus.com/binaries/cache/`cat images/manifest.txt |
> grep -v '#' | awk -F' ' '{print $3}' | grep b200mini`
>
> ...and it would limit the images to the b200mini images, but it's not
> exactly handy. If people have suggestions, I'd be curious to hear them.
>
> ## FPGA Versions
>
> There was a question why we don't version our FPGA images (e.g., why
> does the X300 image not have version 1.2.3, and we say this version
> works with UHD 3.11.0.0). There are several problems with this: First,
> we have over 40 FPGA images (not all are packaged in the default image).
> Versioning them all correctly is already a hard logistical issue.
> Second, how do we create meaningful version numbers? The FPGA images
> consist of many subcomponents, all of which would need to go into an
> image version. The X300 and X310 images, for example, should always have
> similar versions when they have similar behaviour. But what if we fix
> something only for X300? Then, both the X310 and X300 images have 5
> variants (XG, HG, XA, HA, and AA). What if we fix a 10GigE issue? How do
> we keep the 10 version numbers for the {X300, X310} X {HG, XG, HA, AA,
> XA} in sync? Then of course, we have components on all X-Series that we
> also use on the E310. So that makes the version-number-space even more
> higher-dimensional. And so on.
> Finally, we consider the FPGA source code part of UHD. That means, for
> version 3.11.0.0 of UHD, there is exactly 1 commit on the FPGA
> repository that matches the source code. The images built off of this
> commit are the ones that matter, and so they already have a version (the
> UHD version, in this case, 3.11.0.0).
> For any given UHD commit, the correct set of FPGA images is the one
> tracked in images/manifest.txt.
>
> ## The new images downloader
>
> If you're used to using uhd_images_downloader, nothing changes for you.
> Here's the contents of an older email explaining the motivation and
> changes in greater detail.
> Note that we'll start using this tool to distribute even more things
> going forward. We have a bunch of things where building binaries is
> getting more and more complicated for end users, and we'll try and make
> it easy to update devices.
>
>
> ```
> Subject: 

[Discuss-gnuradio] [UHD] Notes on FPGA images packaging, and packaging in general

[Martin Braun]

Hi all,

we've had a bunch of great discussions on the packaging of FPGA images
(binaries) and on packaging UHD in general. Since there were many
different discussions, I'd like to summarize them all in one big email.
Thanks to Rick, Maitland, and Phil for providing feedback.

If you just use UHD via git, none of this affects you.

## Release tags

Going forward, we'll change the format of the release tags. Before, it
was release_003_011_000_000 and so on. Now, it'll be v3.11.0.0, which is
a much more standard format. Personally, I'm happy to make this change
-- we were hesitant to change it for consistency's sake, but it seems no
one actually cares about the old format anyway. If you're pulling from
github, this means you get nice, reproducible URLs like this:
https://github.com/EttusResearch/uhd/releases/tag/v3.11.0.0

...and the tarball for the sources:
https://github.com/EttusResearch/uhd/archive/v3.11.0.0.tar.gz

...and so on. I'd be interested to see who cares about the old tag
format. For now, we'll double-tag (and we might re-tag old versions,
too, if there's interest) to keep things easy.

## Getting FPGA images for releases

We have two paths here. First, we have the uhd_images_downloader format
(I'll talk more about that below). You can grab images from the
images/manifest.txt file, but I understand that's cumbersome.

So, on top of the uhd_images_downloader format, we're considering
bringing back an archive per release, but we're still figuring out the
details of this.

We have for the sake of feedback uploaded an archive here:
https://github.com/EttusResearch/uhd/releases/tag/v3.11.0.0

I'd love to know if this format is suitable for people, and if things
are missing. If this is all everyone needs, we'll find a way to automate
this and make it error-proof.

However, this means that you will be downloading *all* default images
(currently, that's over 200 MB after unpacking), some of which rarely or
never change (e.g., the USRP1 image will always be the same).
If you want to tailor the images you download (e.g., you might only care
about the embedded images), pulling the individual packages might be a
better option. The problem is, we don't have an easy way of identifying
which ones there are without requiring to run a Python script at package
build time.
Something like this works, if you have access to the UHD source code
(bash code):

SRC_URI=http://files.ettus.com/binaries/cache/`cat images/manifest.txt |
grep -v '#' | awk -F' ' '{print $3}' | grep b200mini`

...and it would limit the images to the b200mini images, but it's not
exactly handy. If people have suggestions, I'd be curious to hear them.

## FPGA Versions

There was a question why we don't version our FPGA images (e.g., why
does the X300 image not have version 1.2.3, and we say this version
works with UHD 3.11.0.0). There are several problems with this: First,
we have over 40 FPGA images (not all are packaged in the default image).
Versioning them all correctly is already a hard logistical issue.
Second, how do we create meaningful version numbers? The FPGA images
consist of many subcomponents, all of which would need to go into an
image version. The X300 and X310 images, for example, should always have
similar versions when they have similar behaviour. But what if we fix
something only for X300? Then, both the X310 and X300 images have 5
variants (XG, HG, XA, HA, and AA). What if we fix a 10GigE issue? How do
we keep the 10 version numbers for the {X300, X310} X {HG, XG, HA, AA,
XA} in sync? Then of course, we have components on all X-Series that we
also use on the E310. So that makes the version-number-space even more
higher-dimensional. And so on.
Finally, we consider the FPGA source code part of UHD. That means, for
version 3.11.0.0 of UHD, there is exactly 1 commit on the FPGA
repository that matches the source code. The images built off of this
commit are the ones that matter, and so they already have a version (the
UHD version, in this case, 3.11.0.0).
For any given UHD commit, the correct set of FPGA images is the one
tracked in images/manifest.txt.

## The new images downloader

If you're used to using uhd_images_downloader, nothing changes for you.
Here's the contents of an older email explaining the motivation and
changes in greater detail.
Note that we'll start using this tool to distribute even more things
going forward. We have a bunch of things where building binaries is
getting more and more complicated for end users, and we'll try and make
it easy to update devices.


```
Subject: [UHD] Changes to uhd_images_downloader
Date: Tue, 23 Jan 2018 15:33:14 -0800
From: Martin Braun 

For the next UHD version, we will be updating uhd_images_downloader. The
problem with the old uhd_images_downloader is that it refers to one big
zip file for all the images. As we add more and more devices, we not
only have to update the zip file more often, it also becomes bigger and
bigger. If you're running uhd_images_downloader on an E310