Re: D for microservices: ldc, rdmd, dub now available on Alpine x86_64
On Tuesday, 5 November 2019 at 02:16:28 UTC, Mathias Lang wrote: Hi all, Recently there have been inquiries about support for D on Alpine Linux, a distribution mostly used in combination with Docker to create lightweight container images for microservices. At BPF Korea, we're working on a blockchain written in D, and wanted to be able to easily test and distribute our node using Alpine images, but there was no package for it yet. However, thanks to the work of many contributors before (Joakim Noah, yshui, Petar Kirov/zombinedev, and many others), most of the porting was already done and it was just a matter of fixing a few small issues and and creating the package definitions. A package for `dub` (v1.18.0), `dtools` (ddemangle & rdmd), and `ldc` (v1.18.0) are now available in the `testing` repository of Alpine Linux edge. As `testing` is not enabled by default, you will need to specify the repository (or add it to your `/etc/apk/repositories`) when installing the packages. For example: ``` apk --no-cache add -X http://dl-cdn.alpinelinux.org/alpine/edge/testing ldc ldc-static dtools-rdmd dub ``` this not install the lto static lib, how can i build lto static lib for alpine ?
Re: D for microservices: ldc, rdmd, dub now available on Alpine x86_64
On Wednesday, 15 January 2020 at 11:48:29 UTC, kinke wrote: On Wednesday, 15 January 2020 at 04:00:26 UTC, Mathias Lang wrote: The LDC package is not going to be cross-architecture in the near future, but it should be able to correctly cross-compile once LDC a version matching 2.090.1 is released (most likely LDC 1.20.0). What's the reason for that requirement? I'll soon release LDC v1.20.0-beta1, so is there something from DMD stable you'd need as prerequisite? While I've never fully understood the popularity of Alpine-based docker images (okay, the base image is tiny, but if all images are based on the same, say, Ubuntu image, the few hundred MB only need to be stored once on each host), it'd probably be nice to have the official prebuilt LDC Linux packages linked against musl, to get rid of the glibc dependency. I assume that would be enough to make it runnable on almost all Linux x64 hosts. Nothing from DMD, but some fixes in Druntime. The `stat` struct definition was broken (among other things), so anything involving files was failing on other architecture. This was fixed in https://github.com/dlang/druntime/pull/2899 . Regarding Alpine's popularity: I think it's a matter of convenience. Just like distributing a single binary (or, like in your case, a standalone package) is easier to deal with, being able to distribute a tiny image packaging your application is great (because the economy of scale argument only holds if you have many images on the same host and don't wipe it frequently). Also, the Ubuntu base image is much slower to build, and just I like my compilation time to be low, I like my image build time to be low. I think the README of alpine covers it pretty well: https://github.com/alpinelinux/docker-alpine#why .
Re: D for microservices: ldc, rdmd, dub now available on Alpine x86_64
On Wednesday, 15 January 2020 at 04:00:26 UTC, Mathias Lang wrote: On Tuesday, 5 November 2019 at 02:16:28 UTC, Mathias Lang wrote: [...] Time for an update! The GDC PR have been merged, and followed by another large fix to make it work on most architectures. So if you use Alpine edge, you can just run `apk add gcc-gdc` and you'll get GDC-9.2.0 (with a lot of patches), which is a 2.076.0 frontend, but allow you to bootstrap efficiently. The package is present on ARMv7, AArch64, x86, x86_64, and s390x. Only ppc64le is not supported as GDC hasn't been ported to it (at least not as of the time of 9.2.0 release). [...] It would be nice to have PBF Korea listed on https://dlang.org/orgs-using-d.html (organizations using D)
Re: D for microservices: ldc, rdmd, dub now available on Alpine x86_64
On Wednesday, 15 January 2020 at 04:00:26 UTC, Mathias Lang wrote: The LDC package is not going to be cross-architecture in the near future, but it should be able to correctly cross-compile once LDC a version matching 2.090.1 is released (most likely LDC 1.20.0). What's the reason for that requirement? I'll soon release LDC v1.20.0-beta1, so is there something from DMD stable you'd need as prerequisite? While I've never fully understood the popularity of Alpine-based docker images (okay, the base image is tiny, but if all images are based on the same, say, Ubuntu image, the few hundred MB only need to be stored once on each host), it'd probably be nice to have the official prebuilt LDC Linux packages linked against musl, to get rid of the glibc dependency. I assume that would be enough to make it runnable on almost all Linux x64 hosts.
Re: D for microservices: ldc, rdmd, dub now available on Alpine x86_64
On Wednesday, 15 January 2020 at 11:46:21 UTC, aberba wrote: On Wednesday, 15 January 2020 at 04:00:26 UTC, Mathias Lang wrote: On Tuesday, 5 November 2019 at 02:16:28 UTC, Mathias Lang wrote: [...] Time for an update! The GDC PR have been merged, and followed by another large fix to make it work on most architectures. So if you use Alpine edge, you can just run `apk add gcc-gdc` and you'll get GDC-9.2.0 (with a lot of patches), which is a 2.076.0 frontend, but allow you to bootstrap efficiently. The package is present on ARMv7, AArch64, x86, x86_64, and s390x. Only ppc64le is not supported as GDC hasn't been ported to it (at least not as of the time of 9.2.0 release). [...] It would be nice to have PBF Korea listed on https://dlang.org/orgs-using-d.html (organizations using D) Oops, BPF, my bad
Re: D for microservices: ldc, rdmd, dub now available on Alpine x86_64
On Tuesday, 5 November 2019 at 02:16:28 UTC, Mathias Lang wrote: What's next ? 1) There is a pending PR (https://github.com/alpinelinux/aports/pull/12006) to have GDC working on all architectures alpine supports, not just x86_64. 2) Adding a package for gdmd 3) Rebuild packages based on GDC, so that all architectures are supported. 4) Move the packages to community so they are available out of the box. It would be great for it to happen by the end of the month, as the next alpine release would be around end of December according to their schedule, but that depends on how long PR take to be reviewed. 5) A DMD package for x86 and x86_64 shouldn't be hard to make either Time for an update! The GDC PR have been merged, and followed by another large fix to make it work on most architectures. So if you use Alpine edge, you can just run `apk add gcc-gdc` and you'll get GDC-9.2.0 (with a lot of patches), which is a 2.076.0 frontend, but allow you to bootstrap efficiently. The package is present on ARMv7, AArch64, x86, x86_64, and s390x. Only ppc64le is not supported as GDC hasn't been ported to it (at least not as of the time of 9.2.0 release). Additionally, `gdmd` has been added to `testing` to make it easier to build packages. LDC has been updated to v1.19.0 and the package definition fixed (only `ldc` and `ldc-runtime` present now, no more `ldc-static`). Dub v1.19.0 is also available, also built with GDC, on all the architectures GDC supports. The LDC package is not going to be cross-architecture in the near future, but it should be able to correctly cross-compile once LDC a version matching 2.090.1 is released (most likely LDC 1.20.0). I am now working on finalizing the LDC package so all tests pass on x86 & x86_64, and to provide a DMD package. While there are still a few details to flesh out, most things should work perfectly on x86_64 (we've been using it for months), so feel free to test it and report back. I also hope we can use it in some of the official dlang repositories (e.g. dlang-tour and the dub testsuite). The best place to report issues with the packages is on the Alpine Gitlab ( https://gitlab.alpinelinux.org/alpine/aports/issues) and to tag @Geod24.
Re: D for microservices: ldc, rdmd, dub now available on Alpine x86_64
On Wednesday, 13 November 2019 at 12:27:52 UTC, user wrote: On Tuesday, 5 November 2019 at 02:16:28 UTC, Mathias Lang wrote: ``` apk --no-cache add -X http://dl-cdn.alpinelinux.org/alpine/edge/testing ldc ldc-static dtools-rdmd dub ``` A hello world vibe project doesn't build for me using a Dockerfile using your template. I tried to add missing deps, but I couldn't really figure out what's missing. I get an error during the `dub build` step: ``` eventcore 0.8.48: building configuration "epoll"... /root/.dub/packages/eventcore-0.8.48/eventcore/source/eventcore/drivers/posix/processes.d(316,10): Error: module `core.sys.posix.sys.wait` import `idtype_t` not found` ``` My full Dockefile: ``` FROM alpine:edge as builder RUN apk --no-cache add build-base git RUN apk --no-cache add -X http://dl-cdn.alpinelinux.org/alpine/edge/testing ldc ldc-static dtools-rdmd dub RUN apk --no-cache add -X http://dl-cdn.alpinelinux.org/alpine/edge/testing libevent ibevent-dev WORKDIR /tmp/app ADD source ./source ADD dub.json ./ RUN dub build --compiler=ldc2 ``` Unfortunately recent versions of Vibe.d seems to be broken with Alpine. We use eventcore v0.8.43, Vibe.d v0.8.6, vibe-core v1.7.0. I'll see if I can get a CI in place for Vibe (and fix the issues). Thanks for reporting!
Re: D for microservices: ldc, rdmd, dub now available on Alpine x86_64
On Tuesday, 5 November 2019 at 02:16:28 UTC, Mathias Lang wrote: ``` apk --no-cache add -X http://dl-cdn.alpinelinux.org/alpine/edge/testing ldc ldc-static dtools-rdmd dub ``` A hello world vibe project doesn't build for me using a Dockerfile using your template. I tried to add missing deps, but I couldn't really figure out what's missing. I get an error during the `dub build` step: ``` eventcore 0.8.48: building configuration "epoll"... /root/.dub/packages/eventcore-0.8.48/eventcore/source/eventcore/drivers/posix/processes.d(316,10): Error: module `core.sys.posix.sys.wait` import `idtype_t` not found` ``` My full Dockefile: ``` FROM alpine:edge as builder RUN apk --no-cache add build-base git RUN apk --no-cache add -X http://dl-cdn.alpinelinux.org/alpine/edge/testing ldc ldc-static dtools-rdmd dub RUN apk --no-cache add -X http://dl-cdn.alpinelinux.org/alpine/edge/testing libevent ibevent-dev WORKDIR /tmp/app ADD source ./source ADD dub.json ./ RUN dub build --compiler=ldc2 ```
Re: D for microservices: ldc, rdmd, dub now available on Alpine x86_64
On 2019-11-06 02:02, sarn wrote: And the neat way to do that is with a multi-stage build: one Dockerfile, with an Alpine container building the binary, then copying to a FROM scratch container I've used the "smith" tool as well [1]. It has some additonal help with dynamically linked code. It will use "ldd", recursively, to track dependencies and automatically add those. It will also do a couple of other things, like making the filesystem read only. The musl build is practically necessary because glibc has effectively given up standalone static binary support. So, thanks BPF Korea :) Yeah, it helps. Although it still possible to statically link with glibc, although you might get some warnings. [1] https://github.com/oracle/smith -- /Jacob Carlborg
Re: D for microservices: ldc, rdmd, dub now available on Alpine x86_64
On Tuesday, 5 November 2019 at 12:20:04 UTC, Jacob Carlborg wrote: On Tuesday, 5 November 2019 at 11:49:20 UTC, Daniel Kozak wrote: Generally no, because Apline use musl libc instead of glibc, so there are some issues with that The correct way is to use static linking and putting only the binary in a Docker image, i.e. "from scratch" [1] ;). But using Alpine and musl will help with building the binary. [1] https://hub.docker.com/_/scratch And the neat way to do that is with a multi-stage build: one Dockerfile, with an Alpine container building the binary, then copying to a FROM scratch container: https://docs.docker.com/develop/develop-images/multistage-build/ The musl build is practically necessary because glibc has effectively given up standalone static binary support. So, thanks BPF Korea :)
Re: D for microservices: ldc, rdmd, dub now available on Alpine x86_64
On Tuesday, 5 November 2019 at 02:16:28 UTC, Mathias Lang wrote: At BPF Korea, we're working on a blockchain written in D Hello, Sorry if this has been said already: would you consider being listed in https://dlang.org/orgs-using-d.html? Thanks
Re: D for microservices: ldc, rdmd, dub now available on Alpine x86_64
Am 05.11.2019 um 10:48 schrieb Petar Kirov [ZombineDev]: On Tuesday, 5 November 2019 at 02:16:28 UTC, Mathias Lang wrote: [..] That's great news! Thanks a lot for your hard work! # Note: This will redownload your dependencies every time, which doesn't play well with docker I have been meaning to add a docker and CI friendly command to dub that fetches all dependencies without building them. In the Node.js world they have the `npm ci` [0] command. [0]: https://docs.npmjs.com/cli/ci.html Sounds like "dub upgrade --missing-only" will do that, as it implicitly downloads all dependencies in addition to filling the (non-existent in this case) gaps.
Re: D for microservices: ldc, rdmd, dub now available on Alpine x86_64
On Tuesday, 5 November 2019 at 11:49:20 UTC, Daniel Kozak wrote: Generally no, because Apline use musl libc instead of glibc, so there are some issues with that The correct way is to use static linking and putting only the binary in a Docker image, i.e. "from scratch" [1] ;). But using Alpine and musl will help with building the binary. [1] https://hub.docker.com/_/scratch
Re: D for microservices: ldc, rdmd, dub now available on Alpine x86_64
On Tue, Nov 5, 2019 at 12:05 PM Dejan Lekic via Digitalmars-d-announce wrote: > > On Tuesday, 5 November 2019 at 02:16:28 UTC, Mathias Lang wrote: > > Hi all, > > Recently there have been inquiries about support for D on > > Alpine Linux, a distribution mostly used in combination with > > Docker to create lightweight container images for microservices. > > > > At BPF Korea, we're working on a blockchain written in D, and > > wanted to be able to easily test and distribute our node using > > Alpine images, but there was no package for it yet. > > I am confused... Why do you need LDC, DMD, dub, etc on Alpine? > Can't you build it anywhere and just put the final artifact (or > set of artifacts) inside an Alpine-based container? Generally no, because Apline use musl libc instead of glibc, so there are some issues with that
Re: D for microservices: ldc, rdmd, dub now available on Alpine x86_64
On Tuesday, 5 November 2019 at 02:16:28 UTC, Mathias Lang wrote: Hi all, Recently there have been inquiries about support for D on Alpine Linux, a distribution mostly used in combination with Docker to create lightweight container images for microservices. At BPF Korea, we're working on a blockchain written in D, and wanted to be able to easily test and distribute our node using Alpine images, but there was no package for it yet. I am confused... Why do you need LDC, DMD, dub, etc on Alpine? Can't you build it anywhere and just put the final artifact (or set of artifacts) inside an Alpine-based container?
Re: D for microservices: ldc, rdmd, dub now available on Alpine x86_64
On Tuesday, 5 November 2019 at 02:16:28 UTC, Mathias Lang wrote: [..] That's great news! Thanks a lot for your hard work! # Note: This will redownload your dependencies every time, which doesn't play well with docker I have been meaning to add a docker and CI friendly command to dub that fetches all dependencies without building them. In the Node.js world they have the `npm ci` [0] command. [0]: https://docs.npmjs.com/cli/ci.html
Re: D for microservices: ldc, rdmd, dub now available on Alpine x86_64
On Tuesday, 5 November 2019 at 02:16:28 UTC, Mathias Lang wrote: Hi all, Recently there have been inquiries about support for D on Alpine Linux, a distribution mostly used in combination with Docker to create lightweight container images for microservices. [...] This is great! Much thanks to all for all the work towards this!
