Re: [Semi-OT] Cross-Platform GitHub Action 0.3.0 - NetBSD
On Wednesday, 17 November 2021 at 17:08:00 UTC, kinke wrote: On Wednesday, 17 November 2021 at 15:43:55 UTC, Imperatorn wrote: On Wednesday, 17 November 2021 at 09:03:09 UTC, Jacob Carlborg wrote: On Tuesday, 16 November 2021 at 14:04:12 UTC, Imperatorn wrote: [...] I can add older versions of FreeBSD (currently 12.2 and 13 are supported) if there's a need for that. /Jacob Carlborg Thanks, currently I think it's enough with 12 and 13 tho, but ideally someone actively using FreeBSD (I'm not) could chime in on it. FreeBSD has been CI-tested for years, if not decades; not just by autotester, but also by Cirrus CI, which provides 'native' FreeBSD VMs (with 4 CPU cores), incl. v11/12/13/14 images, see https://cirrus-ci.org/guide/FreeBSD/. Oh, I just saw v12.2 or smth like that
Re: [Semi-OT] Cross-Platform GitHub Action 0.3.0 - NetBSD
On Wednesday, 17 November 2021 at 15:43:55 UTC, Imperatorn wrote: On Wednesday, 17 November 2021 at 09:03:09 UTC, Jacob Carlborg wrote: On Tuesday, 16 November 2021 at 14:04:12 UTC, Imperatorn wrote: Oh, nice to see support for FreeBSD. I just added a version for it in druntime 4 days ago. Now maybe we can test it lol I can add older versions of FreeBSD (currently 12.2 and 13 are supported) if there's a need for that. /Jacob Carlborg Thanks, currently I think it's enough with 12 and 13 tho, but ideally someone actively using FreeBSD (I'm not) could chime in on it. FreeBSD has been CI-tested for years, if not decades; not just by autotester, but also by Cirrus CI, which provides 'native' FreeBSD VMs (with 4 CPU cores), incl. v11/12/13/14 images, see https://cirrus-ci.org/guide/FreeBSD/.
Re: [Semi-OT] Cross-Platform GitHub Action 0.3.0 - NetBSD
On Wednesday, 17 November 2021 at 09:03:09 UTC, Jacob Carlborg wrote: On Tuesday, 16 November 2021 at 14:04:12 UTC, Imperatorn wrote: Oh, nice to see support for FreeBSD. I just added a version for it in druntime 4 days ago. Now maybe we can test it lol I can add older versions of FreeBSD (currently 12.2 and 13 are supported) if there's a need for that. /Jacob Carlborg Thanks, currently I think it's enough with 12 and 13 tho, but ideally someone actively using FreeBSD (I'm not) could chime in on it.
Re: [Semi-OT] Cross-Platform GitHub Action 0.3.0 - NetBSD
On Tuesday, 16 November 2021 at 14:38:48 UTC, Sebastiaan Koppe wrote: I was reading https://github.com/marketplace/actions/cross-platform-action#under-the-hood and have to say I am impressed. Looks like a very well done library. Thanks. Yeah, it turned out to be quite complex to support everything. Multiple platforms, multiple hypervisors, multiple hosts platforms etc. -- /Jacob Carlborg
Re: [Semi-OT] Cross-Platform GitHub Action 0.3.0 - NetBSD
On Tuesday, 16 November 2021 at 14:04:12 UTC, Imperatorn wrote: Oh, nice to see support for FreeBSD. I just added a version for it in druntime 4 days ago. Now maybe we can test it lol I can add older versions of FreeBSD (currently 12.2 and 13 are supported) if there's a need for that. /Jacob Carlborg
Re: [Semi-OT] Cross-Platform GitHub Action 0.3.0 - NetBSD
On Tuesday, 16 November 2021 at 13:34:49 UTC, Jacob Carlborg wrote: # Cross-Platform GitHub Action 0.3.0 I would like to announce a new release of [Cross-Platform GitHub Action](https://github.com/marketplace/actions/cross-platform-action), [0.3.0](https://github.com/cross-platform-actions/action/releases/tag/v0.3.0). I was reading https://github.com/marketplace/actions/cross-platform-action#under-the-hood and have to say I am impressed. Looks like a very well done library.
Re: [Semi-OT] Cross-Platform GitHub Action 0.3.0 - NetBSD
On Tuesday, 16 November 2021 at 13:34:49 UTC, Jacob Carlborg wrote: # Cross-Platform GitHub Action 0.3.0 I would like to announce a new release of [Cross-Platform GitHub Action](https://github.com/marketplace/actions/cross-platform-action), [0.3.0](https://github.com/cross-platform-actions/action/releases/tag/v0.3.0). [...] Oh, nice to see support for FreeBSD. I just added a version for it in druntime 4 days ago. Now maybe we can test it lol
[Semi-OT] Cross-Platform GitHub Action 0.3.0 - NetBSD
# Cross-Platform GitHub Action 0.3.0 I would like to announce a new release of [Cross-Platform GitHub Action](https://github.com/marketplace/actions/cross-platform-action), [0.3.0](https://github.com/cross-platform-actions/action/releases/tag/v0.3.0). For those not familiar with this project, it provides a GitHub action for running GitHub Action workflows on multiple platforms. This includes platforms that GitHub Actions don't natively support (for more information see the original announcement of the project [1]). The major new feature in this release is the support for a new platform: NetBSD. The only supported version so far is 9.2. Other minor features since the last announcement are support for FreeBSD 13 and OpenBSD 6.9. ## Under the Hood For those interested in what's going on under the hood. This release contains some major refactoring to add support for the QEMU hypervisor. Up until now the xhyve hypervisor has been used for the FreeBSD and OpenBSD platforms. Unfortunately NetBSD doesn't run on the xhyve hypervisor, therefore support for QEMU was added. In this release it's only possible to run NetBSD on Linux hosts. The GitHub Linux runners doesn't support hardware accelerated nested virtualization. This unfortunately means that the QEMU hypervisor will be a bit slower than the xhyve hypervisor. [1] https://forum.dlang.org/post/jhasyhuvcxoqhldlb...@forum.dlang.org
Re: [Semi-OT] Cross-Platform GitHub Action
On Tuesday, 8 June 2021 at 20:39:45 UTC, Steven Schveighoffer wrote: I might have a need for it. When I moved mysql-native to github actions, I could no longer run mysql integration tests on MacOS or Windows, since there is no docker support for a mysql instance on those platforms. I can probably install mysql manually at some point, but I haven't looked into it. At least for MacOS, this sounds like a way I can run a mysql instance that the MacOS host can talk to. I don't think that would work. The VM is only running during one step. When the step is done, the VM is terminated. Also, Docker doesn't support FreeBSD or OpenBSD. I don't plan to add platforms which GitHub Actions natively support. On the other hand, it seems pretty straightforward to install MySQL natively on macOS: ``` brew install mysql brew services start mysql ``` There are also several GitHub Actions that will setup MySQL: https://github.com/marketplace?type=actions=mysql -- /Jacob Carlborg
Re: [Semi-OT] Cross-Platform GitHub Action
On Wednesday, 9 June 2021 at 14:05:33 UTC, Steven Schveighoffer wrote: On 6/9/21 6:49 AM, Steven Schveighoffer wrote: On 6/9/21 4:17 AM, evilrat wrote: Just a note from terms of service: you get 2000 minutes available for Github Actions every month for free, however for using Windows hosts it takes 2x minutes and Mac hosts takes 5x minutes. I think this only applies to private repositories: Confirmed: https://github.community/t/for-public-repositories-is-there-a-monthly-limit-on-minutes/129017 -Steve Ok then, good to know. Must be overly paranoid ToS checking kicked in :(
Re: [Semi-OT] Cross-Platform GitHub Action
On Tuesday, 8 June 2021 at 19:10:41 UTC, Jacob Carlborg wrote: # Cross-Platform GitHub Action I would like to announce the first version of a project I've been working on for a while. It's not anything D specific or implemented in D, but it can be used with D projects. This project provides a GitHub action for running GitHub Action workflows on multiple platforms. This includes platforms that GitHub Actions don't natively support. It currently supports FreeBSD and OpenBSD. [...] Neat. ~Brian
Re: [Semi-OT] Cross-Platform GitHub Action
On 6/9/21 6:49 AM, Steven Schveighoffer wrote: On 6/9/21 4:17 AM, evilrat wrote: Just a note from terms of service: you get 2000 minutes available for Github Actions every month for free, however for using Windows hosts it takes 2x minutes and Mac hosts takes 5x minutes. I think this only applies to private repositories: Confirmed: https://github.community/t/for-public-repositories-is-there-a-monthly-limit-on-minutes/129017 -Steve
Re: [Semi-OT] Cross-Platform GitHub Action
On 6/9/21 4:17 AM, evilrat wrote: On Wednesday, 9 June 2021 at 05:20:14 UTC, Jacob Carlborg wrote: On Tuesday, 8 June 2021 at 19:40:01 UTC, kinke wrote: Thx for sharing! Interesting; I've recently worked on something similar, but on Linux hosts and using a kvm/qemu/libvirt stack for running CI jobs in Windows VMs. Yeah, this is running on macOS instead because the Linux and the Windows runners on GitHub actions don't support nested virtualization. Just a note from terms of service: you get 2000 minutes available for Github Actions every month for free, however for using Windows hosts it takes 2x minutes and Mac hosts takes 5x minutes. I think this only applies to private repositories: ``` *GitHub Actions usage is free for both public repositories and self-hosted runners.* For private repositories, each GitHub account receives a certain amount of free minutes and storage, depending on the product used with the account. ``` (emphasis mine) I do not see any balance of minutes spent on github actions on mysql-native, and I'm pretty sure I would have exhausted 2000 minutes already. -Steve
Re: [Semi-OT] Cross-Platform GitHub Action
On Wednesday, 9 June 2021 at 05:20:14 UTC, Jacob Carlborg wrote: On Tuesday, 8 June 2021 at 19:40:01 UTC, kinke wrote: Thx for sharing! Interesting; I've recently worked on something similar, but on Linux hosts and using a kvm/qemu/libvirt stack for running CI jobs in Windows VMs. Yeah, this is running on macOS instead because the Linux and the Windows runners on GitHub actions don't support nested virtualization. Just a note from terms of service: you get 2000 minutes available for Github Actions every month for free, however for using Windows hosts it takes 2x minutes and Mac hosts takes 5x minutes.
Re: [Semi-OT] Cross-Platform GitHub Action
On Tuesday, 8 June 2021 at 19:40:01 UTC, kinke wrote: Thx for sharing! Interesting; I've recently worked on something similar, but on Linux hosts and using a kvm/qemu/libvirt stack for running CI jobs in Windows VMs. Yeah, this is running on macOS instead because the Linux and the Windows runners on GitHub actions don't support nested virtualization. The Hypervisor framework is something similar to KVM. The VM images are actually created using QEMU (on Linux hosts), because Packer doesn't have any support for Xhyve. Packer will create a qcow2 VM image. At run time, the qcow2 image will be converted to the "raw" format, which is the only format that Xhyve supports. qcow2 is used up until runtime because it natively supports compression. I do want to support other operating systems going forward, but unfortunately, it's only FreeBSD and OpenBSD that work in Xhyve. For other operating systems I will have to use QEMU. QEMU does support the Hypervisor framework as an accelerator, but I don't think it will be as fast as Xhyve. When QEMU is supported, it will hopefully be trivial to add support for non-native architectures. I've already built the OpenBSD image for ARM64. -- /Jacob Carlborg
Re: [Semi-OT] Cross-Platform GitHub Action
On 6/8/21 3:10 PM, Jacob Carlborg wrote: # Cross-Platform GitHub Action I would like to announce the first version of a project I've been working on for a while. It's not anything D specific or implemented in D, but it can be used with D projects. This project provides a GitHub action for running GitHub Action workflows on multiple platforms. This includes platforms that GitHub Actions don't natively support. It currently supports FreeBSD and OpenBSD. https://github.com/cross-platform-actions/action Very cool! I might have a need for it. When I moved mysql-native to github actions, I could no longer run mysql integration tests on MacOS or Windows, since there is no docker support for a mysql instance on those platforms. I can probably install mysql manually at some point, but I haven't looked into it. At least for MacOS, this sounds like a way I can run a mysql instance that the MacOS host can talk to. At some point, I will give it a try! -Steve
Re: [Semi-OT] Cross-Platform GitHub Action
Thx for sharing! Interesting; I've recently worked on something similar, but on Linux hosts and using a kvm/qemu/libvirt stack for running CI jobs in Windows VMs.
[Semi-OT] Cross-Platform GitHub Action
# Cross-Platform GitHub Action
I would like to announce the first version of a project I've been
working on for a while. It's not anything D specific or
implemented in D, but it can be used with D projects. This
project provides a GitHub action for running GitHub Action
workflows on multiple platforms. This includes platforms that
GitHub Actions don't natively support. It currently supports
FreeBSD and OpenBSD.
https://github.com/cross-platform-actions/action
## Features
Some of the features that are supported include:
* Multiple operating system with one single action
* Multiple versions of each operating system
* Allows to use default shell or Bash shell
* Low boot overhead
* Fast execution
Compared to
[vmactions/freebsd-vm](https://github.com/vmactions/freebsd-vm),
the boot time is around a fifth and the full execution time for
the same job is around half of freebsd-vm.
## Usage
Here's a sample workflow file which will setup a matrix resulting
in two jobs.
One which will run on FreeBSD 12.2 and one which runs on OpenBSD
6.8.
```yaml
name: CI
on: [push]
jobs:
test:
runs-on: macos-10.15
strategy:
matrix:
os:
- name: freebsd
version: 12.2
- name: openbsd
version: 6.8
steps:
- uses: actions/checkout@v2
- name: Test on ${{ matrix.os.name }}
uses: cross-platform-actions/action@v0.0.1
env:
MY_ENV1: MY_VALUE1
MY_ENV2: MY_VALUE2
with:
environment_variables: MY_ENV1 MY_ENV2
operating_system: ${{ matrix.os.name }}
version: ${{ matrix.os.version }}
shell: bash
run: |
uname -a
echo $SHELL
pwd
ls -lah
whoami
env | sort
```
I've been using this action for one of my own projects
([DLP](https://github.com/jacob-carlborg/dlp/runs/2759807903))
for now close to a week and it works fine. It's mostly FreeBSD
that has been tested.
If you're interested in how the sausage is made, read on. Also
see the readmes of the builder repositories:
https://github.com/cross-platform-actions/freebsd-builder
https://github.com/cross-platform-actions/openbsd-builder
## Under the Hood
GitHub Actions currently only support the following platforms:
macOS, Linux and
Windows. To be able to run other platforms, this GitHub action
runs the commands
inside a virtual machine (VM). macOS is used as the host platform
because it
supports nested virtualization.
The VMs run on the [xhyve][xhyve] hypervisor, which is built on
top of Apple's
[Hypervisor][hypervisor_framework] framework. The Hypervisor
framework allows
to implement hypervisors with support for hardware acceleration
without the
need for kernel extensions. xhyve is a lightweight hypervisor
that boots the
guest operating systems quickly and requires no dependencies
outside of what's
provided by the system.
The VM images running inside the hypervisor are built using
[Packer][packer].
It's a tool for automatically creating VM images, installing the
guest
operating system and doing any final provisioning.
The GitHub action uses SSH to communicate and execute commands
inside the VM.
It uses [rsync][rsync] to share files between the guest VM and
the host. xhyve
does not have any native support for sharing files. To
authenticate the SSH
connection a unique key pair is used. This pair is generated each
time the
action is run. The public key is added to the VM image and the
private key is
stored on the host. Since xhyve does not support file sharing, a
secondar hard
drive, which is backed by a file, is created. The public key is
stored on this
hard drive, which is then mounted by the VM. At boot time, the
secondary hard
drive will be identified and the public key will be copied to the
appropriate
location.
To reduce the time it takes for the GitHub action to start
executing the
commands specified by the user, it aims to boot the guest
operating systems as
fast as possible. This is achieved in a couple of ways:
* By downloading [resources][resources], like xhyve and a few
other tools,
instead of installing them through a package manager
* No compression is used for the resources that are downloaded.
The size is
small enough anyway and it's faster to download the
uncompressed data than
it is to download compressed data and then uncompress it.
* It leverages `async`/`await` to perform tasks asynchronously.
Like
downloading the VM image and other resources at the same time
* It performs as much as possible of the setup ahead of time when
the VM image
is provisioned
[xhyve]: https://github.com/machyve/xhyve
[hypervisor_framework]:
https://developer.apple.com/library/mac/documentation/DriversKernelHardware/Reference/Hypervisor/index.html
[rsync]: https://en.wikipedia.org/wiki/Rsync
[resources]: https://github.com/cross-platform-actions/resources
[packer]:
