Re: [Nix-dev] State database in nixops

[Ben Moseley]

FWIW, we're currently deploying from VMs on developers own machines.

We ensure that we have the same directory structures on our deployment VMs - so 
we're not being bitten by the problem mentioned by the OP.

After deploying we use nix-copy-closure to sync the deployments between 
developer machines and also `tar` up the profile so that any developer can 
perform a rollback.

It's not totally ideal - but it's a reasonable stop-gap until we move to a 
dedicated deployment machine.

--
Ben Moseley - Engineering
b...@borde.rs
www.borde.rs
M: +44 7788 138855

> On 23 Feb 2015, at 15:59, Rob Vermaas  wrote:
> 
> Hi,
> 
> at LogicBlox we use NixOS and NixOps extensively to deploy all sorts of 
> clusters.
> 
> We are using dedicated deployment machines, from which different users can 
> deploy
> the applications by sudo'ing to a shared user account. We back up all nix 
> paths
> that nixops deployments use (using the 'nixops dump-nix-paths' command) to 
> another
> standby machine as well as a S3 backed (private) binary cache. Also we backup 
> the
> nixops state file every 15 minutes. This has been sufficient for our purposes 
> until now.
> This way, in case of emergency, we can switch to standby server, or when shit 
> hits
> the fan even more, we can have up a replacement deployment server within 
> minutes.
> 
> We also have continuously deployed systems, however these are also deployed 
> from the
> shared machines. These deployments pull latest builds from Hydra (our CI 
> system),
> and deploy them automatically.
> 
> With regards to trace-ability / audit trail, NixOps logs to syslog which user 
> (also
> via sudo) runs a nixops command, which might be helpful for that purpose.
> 
> One thing to keep in mind is that the NixOps state file contains secrets, e.g.
> encryption or generated ssh keys, so make sure you keep it safe. I would only 
> put
> it in a git repository if only you have full control over the repository, or 
> a way
> to encrypt it before you push, or both :-).
> 
> I think it would indeed be nice if we would have a way to choose the format 
> for
> the NixOps state, and we would definitely welcome a PR for such a feature. 
> Also,
> it would be good to hear from people, what kind of requirements they would 
> have
> for such a feature.
> 
> Cheers,
> Rob
> 
> 
> On Sun, Feb 22, 2015 at 8:11 PM, Thomas Hunger  > wrote:
> Could you expand on this a bit? I've been using nixops for a while, but only 
> recently set up a Hydra server to run tests automatically. I'm now 
> considering doing automated deployments out of hydra, but not quite sure how 
> that should work. It would be simple to have a hydra job that runs "nixops 
> deploy" but having a build with external side-effects like that seems 
> problematic.
> 
> We're running Jenkins for historical reasons. Jenkins allows executing 
> arbitrary shell scripts after a successful build / test. We run the tests on 
> Jenkins in the same nix-shell environment that we're using for development.
> Jenkins uses an exceedingly terrible XML config format but the files can be 
> generated which allows us to set up projects via nixops. Jenkins also has 
> some hooks and can e.g. be pinged by github to trigger a build.
> 
> We briefly looked at Hydra but could not figure out how to configure it via 
> files (it looks like a point-and-click interface backed by a database). Also, 
> because we have a working system switching is very low priority for us. There 
> are some other open source CI systems like travis and drone which we know of 
> but haven't yet investigated.
> 
> Even though it's very off-topic I'd definitely be interested in reading more 
> about how other companies are using nixops!
> 
> ~
> 
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl 
> http://lists.science.uu.nl/mailman/listinfo/nix-dev 
> 
> 
> 
> 
> 
> --
> Rob Vermaas
> 
> [email] rob.verm...@gmail.com 
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev



signature.asc
Description: Message signed with OpenPGP using GPGMail
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] State database in nixops

[Rob Vermaas]

Hi,

at LogicBlox we use NixOS and NixOps extensively to deploy all sorts of
clusters.

We are using dedicated deployment machines, from which different users can
deploy
the applications by sudo'ing to a shared user account. We back up all nix
paths
that nixops deployments use (using the 'nixops dump-nix-paths' command) to
another
standby machine as well as a S3 backed (private) binary cache. Also we
backup the
nixops state file every 15 minutes. This has been sufficient for our
purposes until now.
This way, in case of emergency, we can switch to standby server, or when
shit hits
the fan even more, we can have up a replacement deployment server within
minutes.

We also have continuously deployed systems, however these are also deployed
from the
shared machines. These deployments pull latest builds from Hydra (our CI
system),
and deploy them automatically.

With regards to trace-ability / audit trail, NixOps logs to syslog which
user (also
via sudo) runs a nixops command, which might be helpful for that purpose.

One thing to keep in mind is that the NixOps state file contains secrets,
e.g.
encryption or generated ssh keys, so make sure you keep it safe. I would
only put
it in a git repository if only you have full control over the repository,
or a way
to encrypt it before you push, or both :-).

I think it would indeed be nice if we would have a way to choose the format
for
the NixOps state, and we would definitely welcome a PR for such a feature.
Also,
it would be good to hear from people, what kind of requirements they would
have
for such a feature.

Cheers,
Rob


On Sun, Feb 22, 2015 at 8:11 PM, Thomas Hunger  wrote:

> Could you expand on this a bit? I've been using nixops for a while, but
>> only recently set up a Hydra server to run tests automatically. I'm now
>> considering doing automated deployments out of hydra, but not quite sure
>> how that should work. It would be simple to have a hydra job that runs
>> "nixops deploy" but having a build with external side-effects like that
>> seems problematic.
>>
>
> We're running Jenkins for historical reasons. Jenkins allows executing
> arbitrary shell scripts after a successful build / test. We run the tests
> on Jenkins in the same nix-shell environment that we're using for
> development.
> Jenkins uses an exceedingly terrible XML config format but the files can
> be generated which allows us to set up projects via nixops. Jenkins also
> has some hooks and can e.g. be pinged by github to trigger a build.
>
> We briefly looked at Hydra but could not figure out how to configure it
> via files (it looks like a point-and-click interface backed by a database).
> Also, because we have a working system switching is very low priority for
> us. There are some other open source CI systems like travis and drone which
> we know of but haven't yet investigated.
>
> Even though it's very off-topic I'd definitely be interested in reading
> more about how other companies are using nixops!
>
> ~
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>


-- 
Rob Vermaas

[email] rob.verm...@gmail.com
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] State database in nixops

[Thomas Hunger]

>
> Could you expand on this a bit? I've been using nixops for a while, but
> only recently set up a Hydra server to run tests automatically. I'm now
> considering doing automated deployments out of hydra, but not quite sure
> how that should work. It would be simple to have a hydra job that runs
> "nixops deploy" but having a build with external side-effects like that
> seems problematic.
>

We're running Jenkins for historical reasons. Jenkins allows executing
arbitrary shell scripts after a successful build / test. We run the tests
on Jenkins in the same nix-shell environment that we're using for
development.
Jenkins uses an exceedingly terrible XML config format but the files can be
generated which allows us to set up projects via nixops. Jenkins also has
some hooks and can e.g. be pinged by github to trigger a build.

We briefly looked at Hydra but could not figure out how to configure it via
files (it looks like a point-and-click interface backed by a database).
Also, because we have a working system switching is very low priority for
us. There are some other open source CI systems like travis and drone which
we know of but haven't yet investigated.

Even though it's very off-topic I'd definitely be interested in reading
more about how other companies are using nixops!

~
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] State database in nixops

[Colin Putney]

On Sat, Feb 21, 2015 at 4:06 PM, Thomas Hunger  wrote:


> We also have a CI server which deploys for us, but that's not the same
> server as the common one we use for manual deploys (which are unfortunately
> necessary on occasion). So we have two copies of the state which has
> already caused some problems.
>

Could you expand on this a bit? I've been using nixops for a while, but
only recently set up a Hydra server to run tests automatically. I'm now
considering doing automated deployments out of hydra, but not quite sure
how that should work. It would be simple to have a hydra job that runs
"nixops deploy" but having a build with external side-effects like that
seems problematic.

-Colin
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] State database in nixops

[Domen Kožar]

On the long run NixOps needs a cleaner API to handle the state.

The result could be centralized deployments (via web api) or just a
plaintext file stored in git.

PS: Making sure you're aware of https://github.com/zalora/upcast


On Sat, Feb 21, 2015 at 4:06 PM, Thomas Hunger  wrote:

>
> What I'd like much better is an option to use an external database; then I
>> could use a replicated cluster or something like that to eliminate the
>> single point of failure.  The last thing I want is my ops team being locked
>> out of nixops during an emergency.
>>
>
> nixops accesses the .nix config file via an absolute path stored in the
> sqlite database  ("nixExprs"). An external database doesn't help unless we
> remove that absolute path restriction somehow and also distribute the
> nixExprs file to every machine that might run nixops.
>
> I'm not too fond of a central (potentially replicated) DB because it comes
> with its own set of problems like permission management, requiring
> bootstrapping (where to store the nixops state for the database server?)
> etc.
>
> We'd be interested in 1) an audit trail, 2) no single point of failure, 3)
> light infrastructure.
>
> On top of the 3 points I mentioned above we'd also be interested in
> avoiding storing ephemeral state like ec2.backups and configsPath.
>
> I think keeping state in a text file in git could achieve the above
> requirements but I'm also sure that there are many other good ways to solve
> this.
>
> @Domen: In our experience deploying from a shared machine doesn't work
> well [1].
>
> best,
> Tom
>
> [1]
> Long-ish:
> E.g. we have an always-online server with all our SSH and Amazon
> credentials on it. We're using instance IAM roles so it's not all bad.
> Also, who's deploying to that server?
> To deploy we SSH into that server, pull the latest git version and then
> call nixops. If there is an issue with the config we fix it locally, push
> to git, pull on server, deploy. That's a bit tiresome.
> We also have a CI server which deploys for us, but that's not the same
> server as the common one we use for manual deploys (which are unfortunately
> necessary on occasion). So we have two copies of the state which has
> already caused some problems.
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] State database in nixops

[Thomas Hunger]

> What I'd like much better is an option to use an external database; then I
> could use a replicated cluster or something like that to eliminate the
> single point of failure.  The last thing I want is my ops team being locked
> out of nixops during an emergency.
>

nixops accesses the .nix config file via an absolute path stored in the
sqlite database  ("nixExprs"). An external database doesn't help unless we
remove that absolute path restriction somehow and also distribute the
nixExprs file to every machine that might run nixops.

I'm not too fond of a central (potentially replicated) DB because it comes
with its own set of problems like permission management, requiring
bootstrapping (where to store the nixops state for the database server?)
etc.

We'd be interested in 1) an audit trail, 2) no single point of failure, 3)
light infrastructure.

On top of the 3 points I mentioned above we'd also be interested in
avoiding storing ephemeral state like ec2.backups and configsPath.

I think keeping state in a text file in git could achieve the above
requirements but I'm also sure that there are many other good ways to solve
this.

@Domen: In our experience deploying from a shared machine doesn't work well
[1].

best,
Tom

[1]
Long-ish:
E.g. we have an always-online server with all our SSH and Amazon
credentials on it. We're using instance IAM roles so it's not all bad.
Also, who's deploying to that server?
To deploy we SSH into that server, pull the latest git version and then
call nixops. If there is an issue with the config we fix it locally, push
to git, pull on server, deploy. That's a bit tiresome.
We also have a CI server which deploys for us, but that's not the same
server as the common one we use for manual deploys (which are unfortunately
necessary on occasion). So we have two copies of the state which has
already caused some problems.
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] State database in nixops

[Domen Kožar]

It shouldn't be much work to support any SQL db by refactoring nixops to
use sqlalhemy orm.
On 21 Feb 2015 13:10, "Ryan Trinkle"  wrote:

> At skedge.me, we've recently switched over to nixops for all deployments,
> so we've run into this issue as well.  We're moving towards the shared
> machine approach, but it's not completely satisfying, because it creates a
> single point of failure.
>
> What I'd like much better is an option to use an external database; then I
> could use a replicated cluster or something like that to eliminate the
> single point of failure.  The last thing I want is my ops team being locked
> out of nixops during an emergency.
>
> On Sat, Feb 21, 2015 at 3:52 PM, Domen Kožar  wrote:
>
>> Best way is to have a shared machine to deploy from.
>>
>> Another option would be to create a web interface for nixops.
>>
>> On Fri, Feb 20, 2015 at 7:05 AM, Thomas Hunger 
>> wrote:
>>
>>> Hi,
>>>
>>> I've been a happy user of nixops for my own projects for a while. It
>>> works fine as a single user tool but we found it to be tricky to use with
>>> multiple developers, or even just a CI system that calls nixops deploy.
>>>
>>> One issue we had is absolute paths in the state. I.e. if I "nixops
>>> export" my state and then import and use it on e.g. the jenkins account I
>>> need to adjust the absolute paths.
>>>
>>> Another issue we have is that checking a sqlite database into git isn't
>>> great for reviews.
>>>
>>> We have a semi-working system now where jenkins calls "nixops deploy -s
>>> /var/common-state/project.sqlite ..." on our deploy server, and we have
>>> local copies of that state for emergency deploys.
>>>
>>> First: How are other people solving collaboration on nixops state?
>>>
>>> Secondly: Is there any interest in extending nixops to have a text (e.g.
>>> protobuf-ascii) state file with relative paths that could be checked into
>>> git? There are a few unclear design choices, e.g. what to do with ec2
>>> backups. But for our purposes it would be better to use AWSs list of
>>> volumes as the source-of-truth for backups (e.g. by adding more tags).
>>>
>>> best,
>>> Tom
>>>
>>> ___
>>> nix-dev mailing list
>>> nix-dev@lists.science.uu.nl
>>> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>>>
>>>
>>
>> ___
>> nix-dev mailing list
>> nix-dev@lists.science.uu.nl
>> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>>
>>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] State database in nixops

[Ryan Trinkle]

At skedge.me, we've recently switched over to nixops for all deployments,
so we've run into this issue as well.  We're moving towards the shared
machine approach, but it's not completely satisfying, because it creates a
single point of failure.

What I'd like much better is an option to use an external database; then I
could use a replicated cluster or something like that to eliminate the
single point of failure.  The last thing I want is my ops team being locked
out of nixops during an emergency.

On Sat, Feb 21, 2015 at 3:52 PM, Domen Kožar  wrote:

> Best way is to have a shared machine to deploy from.
>
> Another option would be to create a web interface for nixops.
>
> On Fri, Feb 20, 2015 at 7:05 AM, Thomas Hunger  wrote:
>
>> Hi,
>>
>> I've been a happy user of nixops for my own projects for a while. It
>> works fine as a single user tool but we found it to be tricky to use with
>> multiple developers, or even just a CI system that calls nixops deploy.
>>
>> One issue we had is absolute paths in the state. I.e. if I "nixops
>> export" my state and then import and use it on e.g. the jenkins account I
>> need to adjust the absolute paths.
>>
>> Another issue we have is that checking a sqlite database into git isn't
>> great for reviews.
>>
>> We have a semi-working system now where jenkins calls "nixops deploy -s
>> /var/common-state/project.sqlite ..." on our deploy server, and we have
>> local copies of that state for emergency deploys.
>>
>> First: How are other people solving collaboration on nixops state?
>>
>> Secondly: Is there any interest in extending nixops to have a text (e.g.
>> protobuf-ascii) state file with relative paths that could be checked into
>> git? There are a few unclear design choices, e.g. what to do with ec2
>> backups. But for our purposes it would be better to use AWSs list of
>> volumes as the source-of-truth for backups (e.g. by adding more tags).
>>
>> best,
>> Tom
>>
>> ___
>> nix-dev mailing list
>> nix-dev@lists.science.uu.nl
>> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>>
>>
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] State database in nixops

[Domen Kožar]

Best way is to have a shared machine to deploy from.

Another option would be to create a web interface for nixops.

On Fri, Feb 20, 2015 at 7:05 AM, Thomas Hunger  wrote:

> Hi,
>
> I've been a happy user of nixops for my own projects for a while. It works
> fine as a single user tool but we found it to be tricky to use with
> multiple developers, or even just a CI system that calls nixops deploy.
>
> One issue we had is absolute paths in the state. I.e. if I "nixops export"
> my state and then import and use it on e.g. the jenkins account I need to
> adjust the absolute paths.
>
> Another issue we have is that checking a sqlite database into git isn't
> great for reviews.
>
> We have a semi-working system now where jenkins calls "nixops deploy -s
> /var/common-state/project.sqlite ..." on our deploy server, and we have
> local copies of that state for emergency deploys.
>
> First: How are other people solving collaboration on nixops state?
>
> Secondly: Is there any interest in extending nixops to have a text (e.g.
> protobuf-ascii) state file with relative paths that could be checked into
> git? There are a few unclear design choices, e.g. what to do with ec2
> backups. But for our purposes it would be better to use AWSs list of
> volumes as the source-of-truth for backups (e.g. by adding more tags).
>
> best,
> Tom
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

[Nix-dev] State database in nixops

[Thomas Hunger]

Hi,

I've been a happy user of nixops for my own projects for a while. It works
fine as a single user tool but we found it to be tricky to use with
multiple developers, or even just a CI system that calls nixops deploy.

One issue we had is absolute paths in the state. I.e. if I "nixops export"
my state and then import and use it on e.g. the jenkins account I need to
adjust the absolute paths.

Another issue we have is that checking a sqlite database into git isn't
great for reviews.

We have a semi-working system now where jenkins calls "nixops deploy -s
/var/common-state/project.sqlite ..." on our deploy server, and we have
local copies of that state for emergency deploys.

First: How are other people solving collaboration on nixops state?

Secondly: Is there any interest in extending nixops to have a text (e.g.
protobuf-ascii) state file with relative paths that could be checked into
git? There are a few unclear design choices, e.g. what to do with ec2
backups. But for our purposes it would be better to use AWSs list of
volumes as the source-of-truth for backups (e.g. by adding more tags).

best,
Tom
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev