Re: CI for prototype/fdb-layer branch

[Alex Miller]

Thanks Ronny!  Then it looks like EL7 RPMs should indeed work fine for Centos8
going forward, and I’ll update FDB's documentation and downloads page text
accordingly at some point.

(And look into UBI for future docker struggles.  Thanks for the pointer!)

> On Feb 20, 2020, at 03:19, Ronny Berndt  wrote:
> 
> Hi Alex,
> 
> seems to work, here is the output of the command ' fdbcli —exec status ':
> 
> 
> Am Do., 20. Feb. 2020 um 11:43 Uhr schrieb Will Holley > :
> 
>> For running in a container, you could also try using the Red Hat UBI
>> instead of CentOS. There is a ubi-init variant which runs systemd (see
>> 
>> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/building_running_and_managing_containers/index#using_init_red_hat_base_images
>> and
>> https://www.redhat.com/en/blog/introducing-red-hat-universal-base-image
>> ).
>> 
>> On Thu, 20 Feb 2020 at 10:30, Ronny Berndt  wrote:
>> 
>>> @Alex
>>> 
>>> I have a running centos8 vm. Maybe i can help…
>>> 
>>> 
 Am 20.02.2020 um 11:13 schrieb Alex Miller
>> >>> :
 
 
> On Feb 19, 2020, at 20:09, Joan Touzet  wrote:
> On 2020-02-19 23:00, Alex Miller wrote:
>>> On Feb 19, 2020, at 16:07, Paul Davis 
>>> wrote:
>>> 
>>> foundationdb does take a while to build though, so finding binaries
>>> might short circuit everything to be even a single apt-get line or
>>> w/e.
>> The build is both slow and quite memory hungry.
>> In addition to FROM + COPY in docker, foundationdb.org hosts
>>> downloads in both a tarball-of-binaries form and a .deb of the server.
>>> Though that papers over CentOS support and the like. Dunno what that
>>> story is like.
>> RPMs for RHEL6 and RHEL7 are also published (which I think should
>>> correlate to centos6 and centos7).
> 
> Are there plans for a CentOS 8 RPM? CentOS 8 has been out since
>>> September 2019, and is the only CentOS that we support with SpiderMonkey
>> 60
>>> today.
 
 I don't think anyone in FDB realized Centos 8 is out, so that's a good
>>> question.
 
 After digging through packaging code, the only difference between the
>> EL6
 and EL7 RPMs is that EL6 installs a sysv init script, and el7 installs
>> a
>>> systemd
 unit file.  The binaries in both cases are built on centos6 and the
>>> build system
 jumps through all the hoops of statically linking a C++ binary, so that
 fdbserver will run on anything centos6 or newer just fine.  This should
 mean that EL7 RPMs are for EL7+, or at least, until Centos changes init
 systems again.
 
 But, that's just theory, and doing a quick install on a centos8 VM
>>> sounded like
 it'd be qick and simple...
 
 Except parallels doesn't support centos8 out of the box yet, and I
>> broke
>>> a
 centos7 install trying to do an (unsupported) upgrade to centos8.  So
>>> that's
 out.
 
 Docker should save the day here, but it turns out that running systemd
>>> in a
 docker container is nontrivial.  Even when I did get systemd running as
>>> PID 1,
 FoundationDB didn't start automatically for me, and systemctl doesn't
>>> work,
 because centos:8 gives you a half baked systemd install that somehow
>>> lacks dbus.
 
 So I'm out of easy options.  fdbserver still runs manually just fine,
>>> and all
 the files _look_ like they got installed in the right place.  So if
>>> someone has
 an actual running VM of Centos 8, it _seems_ like things should still
>>> start fine
 when installing the EL7 RPM.
 
 This exercise did point out that centos8 intentionally doesn't provide
>> a
>>> /usr/bin/python,
 which FDB's RPM packages accidentally depend on, so I've posted
 https://github.com/apple/foundationdb/pull/2700 to get rid of that.
>> One
 will have to use `rpm -i —force foundationdb-server*.rpm` until the
 next 6.2 release.
>>> 
>>> 
>> 

Re: CI for prototype/fdb-layer branch

[Ronny Berndt]

 Hi Alex,

seems to work, here is the output of the command ' fdbcli —exec status ':

[centos@localhost foundationdb]$ fdbcli --exec status
Using cluster file `/etc/foundationdb/fdb.cluster'.

Configuration:
  Redundancy mode- single
  Storage engine - memory-2
  Coordinators   - 1

Cluster:
  FoundationDB processes - 1
  Zones  - 1
  Machines   - 1
  Memory availability- 1.9 GB per process on machine with least
available
   > (WARNING: 4.0 GB recommended) <
  Fault Tolerance- 0 machines
  Server time- 02/20/20 12:08:10

Data:
  Replication health - Healthy
  Moving data- 0.000 GB
  Sum of key-value sizes - 0 MB
  Disk space used- 105 MB

Operating space:
  Storage server - 1.0 GB free on most full server
  Log server - 10.6 GB free on most full server

Workload:
  Read rate  - 7 Hz
  Write rate - 0 Hz
  Transactions started   - 3 Hz
  Transactions committed - 0 Hz
  Conflict rate  - 0 Hz

Backup and DR:
  Running backups- 0
  Running DRs- 0

Client time: 02/20/20 12:08:10

One additional info:

The first run of

sudo rpm -i --force --nodeps ./foundationdb-server-6.2.15-1.el7.x86_64.rpm

gives me some errors (german locale) and i don't know why:

tr: Schreibfehler: Datenübergabe unterbrochen (broken pipe)
tr: Schreibfehler
tr: Schreibfehler: Datenübergabe unterbrochen (broken pipe)
tr: Schreibfehler

I tried the command again, which gives me no errors...

Ronny


Florian Beckert & Ronny Berndt GbR
Saalstr. 3
07743 Jena

Tel.03641 - 6391110
Fax.03641 - 219637
E-Mail: p...@kioskkinder.com


Am Do., 20. Feb. 2020 um 11:43 Uhr schrieb Will Holley :

> For running in a container, you could also try using the Red Hat UBI
> instead of CentOS. There is a ubi-init variant which runs systemd (see
>
> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/building_running_and_managing_containers/index#using_init_red_hat_base_images
> and
> https://www.redhat.com/en/blog/introducing-red-hat-universal-base-image
> ).
>
> On Thu, 20 Feb 2020 at 10:30, Ronny Berndt  wrote:
>
> > @Alex
> >
> > I have a running centos8 vm. Maybe i can help…
> >
> >
> > > Am 20.02.2020 um 11:13 schrieb Alex Miller
>  > >:
> > >
> > >
> > >> On Feb 19, 2020, at 20:09, Joan Touzet  wrote:
> > >> On 2020-02-19 23:00, Alex Miller wrote:
> >  On Feb 19, 2020, at 16:07, Paul Davis 
> > wrote:
> > 
> >  foundationdb does take a while to build though, so finding binaries
> >  might short circuit everything to be even a single apt-get line or
> >  w/e.
> > >>> The build is both slow and quite memory hungry.
> > >>> In addition to FROM + COPY in docker, foundationdb.org hosts
> > downloads in both a tarball-of-binaries form and a .deb of the server.
> >  Though that papers over CentOS support and the like. Dunno what that
> >  story is like.
> > >>> RPMs for RHEL6 and RHEL7 are also published (which I think should
> > correlate to centos6 and centos7).
> > >>
> > >> Are there plans for a CentOS 8 RPM? CentOS 8 has been out since
> > September 2019, and is the only CentOS that we support with SpiderMonkey
> 60
> > today.
> > >
> > > I don't think anyone in FDB realized Centos 8 is out, so that's a good
> > question.
> > >
> > > After digging through packaging code, the only difference between the
> EL6
> > > and EL7 RPMs is that EL6 installs a sysv init script, and el7 installs
> a
> > systemd
> > > unit file.  The binaries in both cases are built on centos6 and the
> > build system
> > > jumps through all the hoops of statically linking a C++ binary, so that
> > > fdbserver will run on anything centos6 or newer just fine.  This should
> > > mean that EL7 RPMs are for EL7+, or at least, until Centos changes init
> > > systems again.
> > >
> > > But, that's just theory, and doing a quick install on a centos8 VM
> > sounded like
> > > it'd be qick and simple...
> > >
> > > Except parallels doesn't support centos8 out of the box yet, and I
> broke
> > a
> > > centos7 install trying to do an (unsupported) upgrade to centos8.  So
> > that's
> > > out.
> > >
> > > Docker should save the day here, but it turns out that running systemd
> > in a
> > > docker container is nontrivial.  Even when I did get systemd running as
> > PID 1,
> > > FoundationDB didn't start automatically for me, and systemctl doesn't
> > work,
> > > because centos:8 gives you a half baked systemd install that somehow
> > lacks dbus.
> > >
> > > So I'm out of easy options.  fdbserver still runs manually just fine,
> > and all
> > > the files _look_ like they got installed in the right place.  So if
> > someone has
> > > an actual running VM of Centos 8, it _seems_ like things should still
> > start fine
> > > when installing the EL7 RPM.
> > >
> > > This exercise did point out that centos8 intentionally doesn't provide
> 

Re: CI for prototype/fdb-layer branch

[Will Holley]

For running in a container, you could also try using the Red Hat UBI
instead of CentOS. There is a ubi-init variant which runs systemd (see
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/building_running_and_managing_containers/index#using_init_red_hat_base_images
and https://www.redhat.com/en/blog/introducing-red-hat-universal-base-image
).

On Thu, 20 Feb 2020 at 10:30, Ronny Berndt  wrote:

> @Alex
>
> I have a running centos8 vm. Maybe i can help…
>
>
> > Am 20.02.2020 um 11:13 schrieb Alex Miller  >:
> >
> >
> >> On Feb 19, 2020, at 20:09, Joan Touzet  wrote:
> >> On 2020-02-19 23:00, Alex Miller wrote:
>  On Feb 19, 2020, at 16:07, Paul Davis 
> wrote:
> 
>  foundationdb does take a while to build though, so finding binaries
>  might short circuit everything to be even a single apt-get line or
>  w/e.
> >>> The build is both slow and quite memory hungry.
> >>> In addition to FROM + COPY in docker, foundationdb.org hosts
> downloads in both a tarball-of-binaries form and a .deb of the server.
>  Though that papers over CentOS support and the like. Dunno what that
>  story is like.
> >>> RPMs for RHEL6 and RHEL7 are also published (which I think should
> correlate to centos6 and centos7).
> >>
> >> Are there plans for a CentOS 8 RPM? CentOS 8 has been out since
> September 2019, and is the only CentOS that we support with SpiderMonkey 60
> today.
> >
> > I don't think anyone in FDB realized Centos 8 is out, so that's a good
> question.
> >
> > After digging through packaging code, the only difference between the EL6
> > and EL7 RPMs is that EL6 installs a sysv init script, and el7 installs a
> systemd
> > unit file.  The binaries in both cases are built on centos6 and the
> build system
> > jumps through all the hoops of statically linking a C++ binary, so that
> > fdbserver will run on anything centos6 or newer just fine.  This should
> > mean that EL7 RPMs are for EL7+, or at least, until Centos changes init
> > systems again.
> >
> > But, that's just theory, and doing a quick install on a centos8 VM
> sounded like
> > it'd be qick and simple...
> >
> > Except parallels doesn't support centos8 out of the box yet, and I broke
> a
> > centos7 install trying to do an (unsupported) upgrade to centos8.  So
> that's
> > out.
> >
> > Docker should save the day here, but it turns out that running systemd
> in a
> > docker container is nontrivial.  Even when I did get systemd running as
> PID 1,
> > FoundationDB didn't start automatically for me, and systemctl doesn't
> work,
> > because centos:8 gives you a half baked systemd install that somehow
> lacks dbus.
> >
> > So I'm out of easy options.  fdbserver still runs manually just fine,
> and all
> > the files _look_ like they got installed in the right place.  So if
> someone has
> > an actual running VM of Centos 8, it _seems_ like things should still
> start fine
> > when installing the EL7 RPM.
> >
> > This exercise did point out that centos8 intentionally doesn't provide a
> /usr/bin/python,
> > which FDB's RPM packages accidentally depend on, so I've posted
> > https://github.com/apple/foundationdb/pull/2700 to get rid of that.  One
> > will have to use `rpm -i —force foundationdb-server*.rpm` until the
> > next 6.2 release.
>
>

Re: CI for prototype/fdb-layer branch

[Ronny Berndt]

@Alex

I have a running centos8 vm. Maybe i can help…


> Am 20.02.2020 um 11:13 schrieb Alex Miller :
> 
> 
>> On Feb 19, 2020, at 20:09, Joan Touzet  wrote:
>> On 2020-02-19 23:00, Alex Miller wrote:
 On Feb 19, 2020, at 16:07, Paul Davis  wrote:
 
 foundationdb does take a while to build though, so finding binaries
 might short circuit everything to be even a single apt-get line or
 w/e.
>>> The build is both slow and quite memory hungry.
>>> In addition to FROM + COPY in docker, foundationdb.org hosts downloads in 
>>> both a tarball-of-binaries form and a .deb of the server.
 Though that papers over CentOS support and the like. Dunno what that
 story is like.
>>> RPMs for RHEL6 and RHEL7 are also published (which I think should correlate 
>>> to centos6 and centos7).
>> 
>> Are there plans for a CentOS 8 RPM? CentOS 8 has been out since September 
>> 2019, and is the only CentOS that we support with SpiderMonkey 60 today.
> 
> I don't think anyone in FDB realized Centos 8 is out, so that's a good 
> question.
> 
> After digging through packaging code, the only difference between the EL6
> and EL7 RPMs is that EL6 installs a sysv init script, and el7 installs a 
> systemd
> unit file.  The binaries in both cases are built on centos6 and the build 
> system
> jumps through all the hoops of statically linking a C++ binary, so that
> fdbserver will run on anything centos6 or newer just fine.  This should
> mean that EL7 RPMs are for EL7+, or at least, until Centos changes init
> systems again.
> 
> But, that's just theory, and doing a quick install on a centos8 VM sounded 
> like
> it'd be qick and simple...
> 
> Except parallels doesn't support centos8 out of the box yet, and I broke a
> centos7 install trying to do an (unsupported) upgrade to centos8.  So that's
> out.
> 
> Docker should save the day here, but it turns out that running systemd in a
> docker container is nontrivial.  Even when I did get systemd running as PID 1,
> FoundationDB didn't start automatically for me, and systemctl doesn't work,
> because centos:8 gives you a half baked systemd install that somehow lacks 
> dbus.
> 
> So I'm out of easy options.  fdbserver still runs manually just fine, and all
> the files _look_ like they got installed in the right place.  So if someone 
> has
> an actual running VM of Centos 8, it _seems_ like things should still start 
> fine
> when installing the EL7 RPM.
> 
> This exercise did point out that centos8 intentionally doesn't provide a 
> /usr/bin/python,
> which FDB's RPM packages accidentally depend on, so I've posted
> https://github.com/apple/foundationdb/pull/2700 to get rid of that.  One
> will have to use `rpm -i —force foundationdb-server*.rpm` until the
> next 6.2 release.

Re: CI for prototype/fdb-layer branch

[Alex Miller]

> On Feb 19, 2020, at 20:09, Joan Touzet  wrote:
> On 2020-02-19 23:00, Alex Miller wrote:
>>> On Feb 19, 2020, at 16:07, Paul Davis  wrote:
>>> 
>>> foundationdb does take a while to build though, so finding binaries
>>> might short circuit everything to be even a single apt-get line or
>>> w/e.
>> The build is both slow and quite memory hungry.
>> In addition to FROM + COPY in docker, foundationdb.org hosts downloads in 
>> both a tarball-of-binaries form and a .deb of the server.
>>> Though that papers over CentOS support and the like. Dunno what that
>>> story is like.
>> RPMs for RHEL6 and RHEL7 are also published (which I think should correlate 
>> to centos6 and centos7).
> 
> Are there plans for a CentOS 8 RPM? CentOS 8 has been out since September 
> 2019, and is the only CentOS that we support with SpiderMonkey 60 today.

I don't think anyone in FDB realized Centos 8 is out, so that's a good question.

After digging through packaging code, the only difference between the EL6
and EL7 RPMs is that EL6 installs a sysv init script, and el7 installs a systemd
unit file.  The binaries in both cases are built on centos6 and the build system
jumps through all the hoops of statically linking a C++ binary, so that
fdbserver will run on anything centos6 or newer just fine.  This should
mean that EL7 RPMs are for EL7+, or at least, until Centos changes init
systems again.

But, that's just theory, and doing a quick install on a centos8 VM sounded like
it'd be qick and simple...

Except parallels doesn't support centos8 out of the box yet, and I broke a
centos7 install trying to do an (unsupported) upgrade to centos8.  So that's
out.

Docker should save the day here, but it turns out that running systemd in a
docker container is nontrivial.  Even when I did get systemd running as PID 1,
FoundationDB didn't start automatically for me, and systemctl doesn't work,
because centos:8 gives you a half baked systemd install that somehow lacks dbus.

So I'm out of easy options.  fdbserver still runs manually just fine, and all
the files _look_ like they got installed in the right place.  So if someone has
an actual running VM of Centos 8, it _seems_ like things should still start fine
when installing the EL7 RPM.

This exercise did point out that centos8 intentionally doesn't provide a 
/usr/bin/python,
which FDB's RPM packages accidentally depend on, so I've posted
https://github.com/apple/foundationdb/pull/2700 to get rid of that.  One
will have to use `rpm -i —force foundationdb-server*.rpm` until the
next 6.2 release.

Re: CI for prototype/fdb-layer branch

[Joan Touzet]

Hey Alex, thanks for this. See below...

On 2020-02-19 23:00, Alex Miller wrote:



On Feb 19, 2020, at 16:07, Paul Davis  wrote:

foundationdb does take a while to build though, so finding binaries
might short circuit everything to be even a single apt-get line or
w/e.


The build is both slow and quite memory hungry.

In addition to FROM + COPY in docker, foundationdb.org hosts downloads in both 
a tarball-of-binaries form and a .deb of the server.


Though that papers over CentOS support and the like. Dunno what that
story is like.


RPMs for RHEL6 and RHEL7 are also published (which I think should correlate to 
centos6 and centos7).


Are there plans for a CentOS 8 RPM? CentOS 8 has been out since 
September 2019, and is the only CentOS that we support with SpiderMonkey 
60 today.



Which I think is to overall say that basically all possible options are open, 
and updated as part of the FDB release process.


-Joan

Re: CI for prototype/fdb-layer branch

[Alex Miller]

> On Feb 19, 2020, at 16:07, Paul Davis  wrote:
> 
> foundationdb does take a while to build though, so finding binaries
> might short circuit everything to be even a single apt-get line or
> w/e.

The build is both slow and quite memory hungry.

In addition to FROM + COPY in docker, foundationdb.org hosts downloads in both 
a tarball-of-binaries form and a .deb of the server.

> Though that papers over CentOS support and the like. Dunno what that
> story is like.

RPMs for RHEL6 and RHEL7 are also published (which I think should correlate to 
centos6 and centos7).

Which I think is to overall say that basically all possible options are open, 
and updated as part of the FDB release process.

Re: CI for prototype/fdb-layer branch

[Paul Davis]

On Wed, Feb 19, 2020 at 4:51 AM Ilya Khlopotov  wrote:
>
> Sounds harder than I hoped. :-(
>

I was a bit out of it yesterday so dunno if my email went through
properly. Though I was trying to say the crazy docker bits should not
be an issue. dev/run and eunit already have the smarts to start fdb if
the fdbserver binary is available. We should be able to just add
multi-stage docker builds that create the fdbserver binary and copy
into our existing builds without much change to the existing stuff.

foundationdb does take a while to build though, so finding binaries
might short circuit everything to be even a single apt-get line or
w/e.

Though that papers over CentOS support and the like. Dunno what that
story is like.

> On 2020/02/18 20:06:48, Joan Touzet  wrote:
> > What Adam said.
> >
> > Reminder that to get CI builds working someone's going to need to do the
> > legwork to add an FDB instance to the container-based build. I'm happy
> > to grant docker hub credentials to couchdbdev for whomever takes this
> > work on.
> >
> > -Joan
> >
> > On 2020-02-18 14:15, Adam Kocoloski wrote:
> > > There’s nothing blocking us from using master for FDB development now 
> > > that we’ve got a 3.x branch — wouldn’t it be better to just make that 
> > > switch?
> > >
> > > Adam
> > >
> > >> On Feb 18, 2020, at 2:13 PM, Ilya Khlopotov  wrote:
> > >>
> > >> Currently we only trigger CI on attempts to merge to master branch. With 
> > >> ongoing effort to rebase CouchDB on top of FoundationDB it seems like we 
> > >> would be running two projects in parallel for quite some time. The lack 
> > >> of CI on merge to prototype/fdb-layer causes merges of broken code.
> > >>
> > >> I am curious how hard it would be to enable CI for prototype/fdb-layer 
> > >> branch in addition to master branch.
> > >>
> > >> Best regards,
> > >> iilyak
> > >
> >

Re: CI for prototype/fdb-layer branch

[Joan Touzet]

If no one else offers to help, I can look at this late next week, after 
3.0.0 is released.


-Joan

On 2020-02-18 9:14 p.m., Paul Davis wrote:

If memory serves, fdbserver is statically linked by default which should
save some work.

On Tue, Feb 18, 2020 at 8:13 PM Paul Davis 
wrote:


We probably don’t need fancy  Docker here. Eunit and dev/run already setup
fdbserver automatically if the binary is found. A two stage build that
copies fdbserver into the existing image is probably all that needs to
happen.

I’m limited to typing one handed for the forseeable future so I won’t get
to it myself. But i’m more than happy to chat on email or irc if someone
has questions.

Here’s build notes for macOS:

https://gist.github.com/davisp/aa6f526b8fd0441f2035c93774828798


On Tue, Feb 18, 2020 at 5:08 PM Joan Touzet  wrote:


Thanks Alex!

The Jenkins setup currently runs each CI job in a container per host,
and Docker-in-Docker can be a bit challenging. Hopefully the alternative
"expose your parent Docker socket inside of Docker with -v" approach, as
described at
https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/
will be sufficient. We'd also need the docker cli inside the CI image,
which shouldn't be too hard to add.

For builds on master itself (post-merge, to check a wider set of
platforms), we currently have FreeBSD builds where there is no Docker.
We'll need to sort out a solution for that as well. I can preinstall FDB
on those hosts but the Jenkinsfile will have to take care of starting
and stopping FDB as pre/post actions.

-Joan

On 2020-02-18 17:22, Alex Miller wrote:

And for whoever does pick it up, there's some examples of

docker-ify-ing FDB and running it via docker-compose in
apple/foundationdb's packaging/docker/<
https://github.com/apple/foundationdb/tree/master/packaging/docker>  ,
and the Dockerfile therein is built and pushed as foundationdb/foundationdb<
https://hub.docker.com/r/foundationdb/foundationdb/tags>  on docker hub.

Re: CI for prototype/fdb-layer branch

[Ilya Khlopotov]

Sounds harder than I hoped. :-(

On 2020/02/18 20:06:48, Joan Touzet  wrote: 
> What Adam said.
> 
> Reminder that to get CI builds working someone's going to need to do the 
> legwork to add an FDB instance to the container-based build. I'm happy 
> to grant docker hub credentials to couchdbdev for whomever takes this 
> work on.
> 
> -Joan
> 
> On 2020-02-18 14:15, Adam Kocoloski wrote:
> > There’s nothing blocking us from using master for FDB development now that 
> > we’ve got a 3.x branch — wouldn’t it be better to just make that switch?
> > 
> > Adam
> > 
> >> On Feb 18, 2020, at 2:13 PM, Ilya Khlopotov  wrote:
> >>
> >> Currently we only trigger CI on attempts to merge to master branch. With 
> >> ongoing effort to rebase CouchDB on top of FoundationDB it seems like we 
> >> would be running two projects in parallel for quite some time. The lack of 
> >> CI on merge to prototype/fdb-layer causes merges of broken code.
> >>
> >> I am curious how hard it would be to enable CI for prototype/fdb-layer 
> >> branch in addition to master branch.
> >>
> >> Best regards,
> >> iilyak
> > 
> 

Re: CI for prototype/fdb-layer branch

[Paul Davis]

If memory serves, fdbserver is statically linked by default which should
save some work.

On Tue, Feb 18, 2020 at 8:13 PM Paul Davis 
wrote:

> We probably don’t need fancy  Docker here. Eunit and dev/run already setup
> fdbserver automatically if the binary is found. A two stage build that
> copies fdbserver into the existing image is probably all that needs to
> happen.
>
> I’m limited to typing one handed for the forseeable future so I won’t get
> to it myself. But i’m more than happy to chat on email or irc if someone
> has questions.
>
> Here’s build notes for macOS:
>
> https://gist.github.com/davisp/aa6f526b8fd0441f2035c93774828798
>
>
> On Tue, Feb 18, 2020 at 5:08 PM Joan Touzet  wrote:
>
>> Thanks Alex!
>>
>> The Jenkins setup currently runs each CI job in a container per host,
>> and Docker-in-Docker can be a bit challenging. Hopefully the alternative
>> "expose your parent Docker socket inside of Docker with -v" approach, as
>> described at
>> https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/
>> will be sufficient. We'd also need the docker cli inside the CI image,
>> which shouldn't be too hard to add.
>>
>> For builds on master itself (post-merge, to check a wider set of
>> platforms), we currently have FreeBSD builds where there is no Docker.
>> We'll need to sort out a solution for that as well. I can preinstall FDB
>> on those hosts but the Jenkinsfile will have to take care of starting
>> and stopping FDB as pre/post actions.
>>
>> -Joan
>>
>> On 2020-02-18 17:22, Alex Miller wrote:
>> > And for whoever does pick it up, there's some examples of
>> docker-ify-ing FDB and running it via docker-compose in
>> apple/foundationdb's packaging/docker/<
>> https://github.com/apple/foundationdb/tree/master/packaging/docker>  ,
>> and the Dockerfile therein is built and pushed as foundationdb/foundationdb<
>> https://hub.docker.com/r/foundationdb/foundationdb/tags>  on docker hub.
>>
>

Re: CI for prototype/fdb-layer branch

[Paul Davis]

We probably don’t need fancy  Docker here. Eunit and dev/run already setup
fdbserver automatically if the binary is found. A two stage build that
copies fdbserver into the existing image is probably all that needs to
happen.

I’m limited to typing one handed for the forseeable future so I won’t get
to it myself. But i’m more than happy to chat on email or irc if someone
has questions.

Here’s build notes for macOS:

https://gist.github.com/davisp/aa6f526b8fd0441f2035c93774828798


On Tue, Feb 18, 2020 at 5:08 PM Joan Touzet  wrote:

> Thanks Alex!
>
> The Jenkins setup currently runs each CI job in a container per host,
> and Docker-in-Docker can be a bit challenging. Hopefully the alternative
> "expose your parent Docker socket inside of Docker with -v" approach, as
> described at
> https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/
> will be sufficient. We'd also need the docker cli inside the CI image,
> which shouldn't be too hard to add.
>
> For builds on master itself (post-merge, to check a wider set of
> platforms), we currently have FreeBSD builds where there is no Docker.
> We'll need to sort out a solution for that as well. I can preinstall FDB
> on those hosts but the Jenkinsfile will have to take care of starting
> and stopping FDB as pre/post actions.
>
> -Joan
>
> On 2020-02-18 17:22, Alex Miller wrote:
> > And for whoever does pick it up, there's some examples of docker-ify-ing
> FDB and running it via docker-compose in apple/foundationdb's
> packaging/docker/<
> https://github.com/apple/foundationdb/tree/master/packaging/docker>  ,
> and the Dockerfile therein is built and pushed as foundationdb/foundationdb<
> https://hub.docker.com/r/foundationdb/foundationdb/tags>  on docker hub.
>

Re: CI for prototype/fdb-layer branch

[Joan Touzet]

Thanks Alex!

The Jenkins setup currently runs each CI job in a container per host, 
and Docker-in-Docker can be a bit challenging. Hopefully the alternative 
"expose your parent Docker socket inside of Docker with -v" approach, as 
described at 
https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/ 
will be sufficient. We'd also need the docker cli inside the CI image, 
which shouldn't be too hard to add.


For builds on master itself (post-merge, to check a wider set of 
platforms), we currently have FreeBSD builds where there is no Docker. 
We'll need to sort out a solution for that as well. I can preinstall FDB 
on those hosts but the Jenkinsfile will have to take care of starting 
and stopping FDB as pre/post actions.


-Joan

On 2020-02-18 17:22, Alex Miller wrote:

And for whoever does pick it up, there's some examples of docker-ify-ing FDB and running it 
via docker-compose in apple/foundationdb's 
packaging/docker/ 
 , and the Dockerfile therein is built and pushed as 
foundationdb/foundationdb  
on docker hub.

Re: CI for prototype/fdb-layer branch

[Alex Miller]

And for whoever does pick it up, there's some examples of docker-ify-ing FDB 
and running it via docker-compose in apple/foundationdb's packaging/docker/ 
<https://github.com/apple/foundationdb/tree/master/packaging/docker> , and the 
Dockerfile therein is built and pushed as foundationdb/foundationdb 
<https://hub.docker.com/r/foundationdb/foundationdb/tags> on docker hub.

> On Feb 18, 2020, at 12:06, Joan Touzet  wrote:
> 
> What Adam said.
> 
> Reminder that to get CI builds working someone's going to need to do the 
> legwork to add an FDB instance to the container-based build. I'm happy to 
> grant docker hub credentials to couchdbdev for whomever takes this work on.
> 
> -Joan
> 
> On 2020-02-18 14:15, Adam Kocoloski wrote:
>> There’s nothing blocking us from using master for FDB development now that 
>> we’ve got a 3.x branch — wouldn’t it be better to just make that switch?
>> Adam
>>> On Feb 18, 2020, at 2:13 PM, Ilya Khlopotov  wrote:
>>> 
>>> Currently we only trigger CI on attempts to merge to master branch. With 
>>> ongoing effort to rebase CouchDB on top of FoundationDB it seems like we 
>>> would be running two projects in parallel for quite some time. The lack of 
>>> CI on merge to prototype/fdb-layer causes merges of broken code.
>>> 
>>> I am curious how hard it would be to enable CI for prototype/fdb-layer 
>>> branch in addition to master branch.
>>> 
>>> Best regards,
>>> iilyak

Re: CI for prototype/fdb-layer branch

[Joan Touzet]

What Adam said.

Reminder that to get CI builds working someone's going to need to do the 
legwork to add an FDB instance to the container-based build. I'm happy 
to grant docker hub credentials to couchdbdev for whomever takes this 
work on.


-Joan

On 2020-02-18 14:15, Adam Kocoloski wrote:

There’s nothing blocking us from using master for FDB development now that 
we’ve got a 3.x branch — wouldn’t it be better to just make that switch?

Adam


On Feb 18, 2020, at 2:13 PM, Ilya Khlopotov  wrote:

Currently we only trigger CI on attempts to merge to master branch. With 
ongoing effort to rebase CouchDB on top of FoundationDB it seems like we would 
be running two projects in parallel for quite some time. The lack of CI on 
merge to prototype/fdb-layer causes merges of broken code.

I am curious how hard it would be to enable CI for prototype/fdb-layer branch 
in addition to master branch.

Best regards,
iilyak

Re: CI for prototype/fdb-layer branch

[Adam Kocoloski]

There’s nothing blocking us from using master for FDB development now that 
we’ve got a 3.x branch — wouldn’t it be better to just make that switch?

Adam

> On Feb 18, 2020, at 2:13 PM, Ilya Khlopotov  wrote:
> 
> Currently we only trigger CI on attempts to merge to master branch. With 
> ongoing effort to rebase CouchDB on top of FoundationDB it seems like we 
> would be running two projects in parallel for quite some time. The lack of CI 
> on merge to prototype/fdb-layer causes merges of broken code. 
> 
> I am curious how hard it would be to enable CI for prototype/fdb-layer branch 
> in addition to master branch. 
> 
> Best regards,
> iilyak

CI for prototype/fdb-layer branch

[Ilya Khlopotov]

Currently we only trigger CI on attempts to merge to master branch. With 
ongoing effort to rebase CouchDB on top of FoundationDB it seems like we would 
be running two projects in parallel for quite some time. The lack of CI on 
merge to prototype/fdb-layer causes merges of broken code. 

I am curious how hard it would be to enable CI for prototype/fdb-layer branch 
in addition to master branch. 

Best regards,
iilyak