Hi all, we've had a bunch of great discussions on the packaging of FPGA images (binaries) and on packaging UHD in general. Since there were many different discussions, I'd like to summarize them all in one big email. Thanks to Rick, Maitland, and Phil for providing feedback.
If you just use UHD via git, none of this affects you. ## Release tags Going forward, we'll change the format of the release tags. Before, it was release_003_011_000_000 and so on. Now, it'll be v3.11.0.0, which is a much more standard format. Personally, I'm happy to make this change -- we were hesitant to change it for consistency's sake, but it seems no one actually cares about the old format anyway. If you're pulling from github, this means you get nice, reproducible URLs like this: https://github.com/EttusResearch/uhd/releases/tag/v3.11.0.0 ...and the tarball for the sources: https://github.com/EttusResearch/uhd/archive/v3.11.0.0.tar.gz ...and so on. I'd be interested to see who cares about the old tag format. For now, we'll double-tag (and we might re-tag old versions, too, if there's interest) to keep things easy. ## Getting FPGA images for releases We have two paths here. First, we have the uhd_images_downloader format (I'll talk more about that below). You can grab images from the images/manifest.txt file, but I understand that's cumbersome. So, on top of the uhd_images_downloader format, we're considering bringing back an archive per release, but we're still figuring out the details of this. We have for the sake of feedback uploaded an archive here: https://github.com/EttusResearch/uhd/releases/tag/v3.11.0.0 I'd love to know if this format is suitable for people, and if things are missing. If this is all everyone needs, we'll find a way to automate this and make it error-proof. However, this means that you will be downloading *all* default images (currently, that's over 200 MB after unpacking), some of which rarely or never change (e.g., the USRP1 image will always be the same). If you want to tailor the images you download (e.g., you might only care about the embedded images), pulling the individual packages might be a better option. The problem is, we don't have an easy way of identifying which ones there are without requiring to run a Python script at package build time. Something like this works, if you have access to the UHD source code (bash code): SRC_URI=http://files.ettus.com/binaries/cache/`cat images/manifest.txt | grep -v '#' | awk -F' ' '{print $3}' | grep b200mini` ...and it would limit the images to the b200mini images, but it's not exactly handy. If people have suggestions, I'd be curious to hear them. ## FPGA Versions There was a question why we don't version our FPGA images (e.g., why does the X300 image not have version 1.2.3, and we say this version works with UHD 3.11.0.0). There are several problems with this: First, we have over 40 FPGA images (not all are packaged in the default image). Versioning them all correctly is already a hard logistical issue. Second, how do we create meaningful version numbers? The FPGA images consist of many subcomponents, all of which would need to go into an image version. The X300 and X310 images, for example, should always have similar versions when they have similar behaviour. But what if we fix something only for X300? Then, both the X310 and X300 images have 5 variants (XG, HG, XA, HA, and AA). What if we fix a 10GigE issue? How do we keep the 10 version numbers for the {X300, X310} X {HG, XG, HA, AA, XA} in sync? Then of course, we have components on all X-Series that we also use on the E310. So that makes the version-number-space even more higher-dimensional. And so on. Finally, we consider the FPGA source code part of UHD. That means, for version 3.11.0.0 of UHD, there is exactly 1 commit on the FPGA repository that matches the source code. The images built off of this commit are the ones that matter, and so they already have a version (the UHD version, in this case, 3.11.0.0). For any given UHD commit, the correct set of FPGA images is the one tracked in images/manifest.txt. ## The new images downloader If you're used to using uhd_images_downloader, nothing changes for you. Here's the contents of an older email explaining the motivation and changes in greater detail. Note that we'll start using this tool to distribute even more things going forward. We have a bunch of things where building binaries is getting more and more complicated for end users, and we'll try and make it easy to update devices. ``` Subject: [UHD] Changes to uhd_images_downloader Date: Tue, 23 Jan 2018 15:33:14 -0800 From: Martin Braun <martin.br...@ettus.com> For the next UHD version, we will be updating uhd_images_downloader. The problem with the old uhd_images_downloader is that it refers to one big zip file for all the images. As we add more and more devices, we not only have to update the zip file more often, it also becomes bigger and bigger. If you're running uhd_images_downloader on an E310, it's most likely that you only want the E310 images, and not the megabytes of other FPGA images. The new tool fixes all those issues by breaking up the zip file into more, smaller zip files. On top of that, the new tool has a bunch of other improvements: - It maintains an inventory of downloaded files, and skips downloading them again if they didn't change. This means if we roll out a fix for the X300 only, and you run uhd_images_downloader without any arguments, it won't try and update anything else. Should you delete the inventory, it'll simply get everything, like it did before. - You can now filter by type. Use the -t switch to match against a regular expression, e.g. '-t b200mini' will only download a b200mini file. - We'll be adding more image types soon that are not FPGA images, such as filesystem updates. - 'uhd_images_downloader -l' will list all the targets you can download. In general, this should improve things and make other things easier. However, since we had to basically rewrite the tool, which had otherwise matured, it's likely that the new tool has shortcomings. We'd like to hear from you regarding those. There are a few problems we are already tracking: - We currently have no archives other than zip files. We're intending to add .tar.xz in the future, and you will be able to use uhd_images_downloader to fetch those (assuming your system can unzip xz archives) - The tool output changed, and we're still fine tuning the output. If you're a package maintainer, you probably hard-coded the URL for the FPGA images in your package creation script. We track the URLs of the individual archives, along with their SHA256 hash, in a manifest that is checked into git: https://github.com/EttusResearch/uhddev/blob/master/images/manifest.txt The base URL for all those is http://files.ettus.com/binaries/cache, so http://files.ettus.com/binaries/cache/usrp1/fpga-6bea23d/usrp1_usrp1_fpga_default.zip will fetch the USRP1 FPGA images. We're happy for any feedback -- ideally before we release the next version. Cheers, Martin ``` Hopefully this is helpful. Again, I'd like to thank everyone for providing feedback. Cheers, Martin _______________________________________________ USRP-users mailing list USRP-users@lists.ettus.com http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com