Re: [Tails-dev] Feature #5301 - Clone or Backup Persistent Volume
Hi, all. Sorry it has been so long since I've written. I have just uploaded v0.2.0 of tails-clone-persistent[1], which is memory-safe and relies entirely on polkit and sudoers to manage privilege escalation. There are still a couple of minor niggles, but I'm hoping someone on the list can help with them. Firstly, I had some trouble finding the correct polkit action for unlocking the target crypted drive. As far as I can tell from the docs, this should be `org.freedesktop.udisks2.encrypted-unlock-system` - but this is already in the tails polkit configuration and seems to have no effect. I got it to work by enabling `org.freedesktop.udisks2.*`, but this is excessive. Secondly, I have a subroutine that duplicates the functionality of `tails-persistence-setup --step bootstrap --override-boot-drive `, but clumsily and less safely. Ideally, I would replace this by a call to tails-persistence-setup itself, but I can't get `--override-boot-drive` to work. Any ideas or support would be most welcome. ;-) [1] https://github.com/andrewgdotcom/tails-clone-persistent Andrew. signature.asc Description: OpenPGP digital signature ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Feature #5301 - Clone or Backup Persistent Volume
Andrew Gallagher wrote (10 Jan 2016 01:38:20 GMT) : >> On 10 Jan 2016, at 00:01, intrigeri wrote: >> >> In Tails, we also directly access the block device as the amnesia >> user, since >> /etc/udev/rules.d/99-make-removable-devices-user-writable.rules allows >> us to do that. > Ah, this could be the game changer. I'll look into that and see if it gives > me the > powers I need to avoid setuid (which is the source of all the problems). Cool :) >> On Debian/Ubuntu, we are more limited so we use some operations that >> require administrator credentials: >> >> * opening the block device with udisks2, to get a filehandle for >> writing the MBR; >> * running syslinux as root, using pkexec. > From what little I know of policykit, the same security caveats as setuid > would usually apply...? polkit has some minor security advantages, such as allowing us to grant the privileges we need to the active session user only, and requiring user consent in a way that's integrated in the desktop. Cheers, -- intrigeri ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Feature #5301 - Clone or Backup Persistent Volume
Andrew Gallagher > On 10 Jan 2016, at 08:40, emmapeel wrote: > > Ok, I didn't tried Andrew's script, but I can say that the 'permissions > issue' was for the users that attended the most difficult part to follow. > > As in after backing up they lose the configurations, > live-additional-software, etc. Yes, the tool performs the equivalent of tails-fix-persistent-volume-permissions on the target partition after rsyncing. I had to code it separately as t-f-p-v-p hardcodes the location of the source partition. Solving that was particularly urgent, as I use apt persistence to pull my custom software onto the tails drive for testing, which is much easier than rebuilding images. (The only wrinkle that still trips me up is that persisting single files is not supported, so I have to persist the entire /etc/apt/sources.list.d directory, thus tails upgrades require manual intervention - but that's a separate issue) A ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Feature #5301 - Clone or Backup Persistent Volume
sajolida: > Spencer: >>> Emma Peel (in copy) could be >>> interested as she's been giving a workshop on Tails backups last week >>> and 32C3. >> Ok, I didn't tried Andrew's script, but I can say that the 'permissions issue' was for the users that attended the most difficult part to follow. As in after backing up they lose the configurations, live-additional-software, etc. signature.asc Description: OpenPGP digital signature ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Feature #5301 - Clone or Backup Persistent Volume
> On 10 Jan 2016, at 00:01, intrigeri wrote: > > In Tails, we also directly access the block device as the amnesia > user, since > /etc/udev/rules.d/99-make-removable-devices-user-writable.rules allows > us to do that. Ah, this could be the game changer. I'll look into that and see if it gives me the powers I need to avoid setuid (which is the source of all the problems). > On Debian/Ubuntu, we are more limited so we use some operations that > require administrator credentials: > > * opening the block device with udisks2, to get a filehandle for > writing the MBR; > * running syslinux as root, using pkexec. >From what little I know of policykit, the same security caveats as setuid >would usually apply...? Thanks! A ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Feature #5301 - Clone or Backup Persistent Volume
anonym wrote (08 Jan 2016 17:43:04 GMT) : > I leave it to sajolida and u to decide on how the backup tool should > integrate into the installer (or if it should be separate), Same here. > and for u and intrigeri to clarify the privileges separation situation > in the installer (I think it's run as a normal users, and udisks is used > to partition, format, luksOpen etc without any risky setuid business). This is correct. There's no luksOpen involved since Tails Installer does not handle encrypted partitions, though. In Tails, we also directly access the block device as the amnesia user, since /etc/udev/rules.d/99-make-removable-devices-user-writable.rules allows us to do that. On Debian/Ubuntu, we are more limited so we use some operations that require administrator credentials: * opening the block device with udisks2, to get a filehandle for writing the MBR; * running syslinux as root, using pkexec. Cheers! -- intrigeri ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Feature #5301 - Clone or Backup Persistent Volume
Andrew Gallagher: > On 07/01/16 19:41, anonym wrote: >> >> I haven't tested it, but I did have a look at src/tcp-helper.c since the >> necessity of memory unsafe code surprised me: > > For the record, I never intended to use C. It was originally written in > perl (like the GUI wrapper), but setuidperl is no longer supported and I > needed to drop any requirement for sudo. The perl documentation suggests > writing a setuid C wrapper that does (more or less) > "setreuid(0,0); system(MY_PROGRAM);" > and then lists all the reasons why you shouldn't. Ok. I'm not entirely sure why you'd need setuid. > So I reluctantly rewrote the helper in C, and after getting angry with C > I gave in and #included (forgetting to rename the source > accordingly). Much of my time since has been spent trying to extricate > myself from the inevitable foot-shooting. ;-) Ack! > Rest of your comments fully understood, :) > hands in the air bang to rights guv. What does this mean? I'm not a native English speaker, which possibly explains it. >> Best practice is to drop the input sanitation and pick alternatives to >> `system()`/`popen()` that avoids invoking the shell and calls `exec()` >> or similar. I'm too ignorant of C/C++ to recommend you anything. > > Yes, these were originally perl system(,,,) and open(,,,) calls, which > are safe as they don't invoke a shell. Ok. Still, I'm confused by the need for setuid and going C(++). For the record, Python's subprocess module would do the same job: https://docs.python.org/3/library/subprocess.html > C has no simple equivalent that I > am aware of, I guess that'd be a combo of fork() + exec() + IPC to get the results (return code, stdout/err) back to the parent. :) Or, preferably, using some existing library which makes this easier. > so I cheated by adding the overkill input sanitation > (bearing in mind that in normal use the input parameters will in 99.9% > of cases be "/live/persistence/TailsData_unlocked" and "/dev/sd[a-z]"). Understood. > 1. is it worth spending any more time fixing the dodgy C++, or is it > just throwing good effort after bad? It's not worth the effort. Sorry, but let's drop the C/C++. > 2. is the general structure of userspace GUI + setuid helper still the > way to go? > > 3. Since tails-installer-invoker and tails-installer remain distinct > programs (and tails-installer closely tracks liveusb-creator), is it > necessary/desirable to merge persistence cloning into either of them, or > should/could it remain a separate (linked) executable? In other words, > do I really need to learn Python? ;-) I mainly threw myself in the discussion to avert us from adding any C/C++ code. I leave it to sajolida and u to decide on how the backup tool should integrate into the installer (or if it should be separate), and for u and intrigeri to clarify the privileges separation situation in the installer (I think it's run as a normal users, and udisks is used to partition, format, luksOpen etc without any risky setuid business). All this will influence the answers to these questions. Cheers! ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Feature #5301 - Clone or Backup Persistent Volume
On 07/01/16 19:41, anonym wrote: > > I haven't tested it, but I did have a look at src/tcp-helper.c since the > necessity of memory unsafe code surprised me: For the record, I never intended to use C. It was originally written in perl (like the GUI wrapper), but setuidperl is no longer supported and I needed to drop any requirement for sudo. The perl documentation suggests writing a setuid C wrapper that does (more or less) "setreuid(0,0); system(MY_PROGRAM);" and then lists all the reasons why you shouldn't. So I reluctantly rewrote the helper in C, and after getting angry with C I gave in and #included (forgetting to rename the source accordingly). Much of my time since has been spent trying to extricate myself from the inevitable foot-shooting. ;-) Rest of your comments fully understood, hands in the air bang to rights guv. > Best practice is to drop the input sanitation and pick alternatives to > `system()`/`popen()` that avoids invoking the shell and calls `exec()` > or similar. I'm too ignorant of C/C++ to recommend you anything. Yes, these were originally perl system(,,,) and open(,,,) calls, which are safe as they don't invoke a shell. C has no simple equivalent that I am aware of, so I cheated by adding the overkill input sanitation (bearing in mind that in normal use the input parameters will in 99.9% of cases be "/live/persistence/TailsData_unlocked" and "/dev/sd[a-z]"). > So, I personally would not include this program. Hence I recommend you > to rewriting it in the same scripting language you end up using for the > GUI, so Python 3, it seems. I'd pay good money to have setuidperl back, but upstream had their reasons. Thanks for the thorough criticism, sincerely. So I have a few questions for the list: 1. is it worth spending any more time fixing the dodgy C++, or is it just throwing good effort after bad? 2. is the general structure of userspace GUI + setuid helper still the way to go? 3. Since tails-installer-invoker and tails-installer remain distinct programs (and tails-installer closely tracks liveusb-creator), is it necessary/desirable to merge persistence cloning into either of them, or should/could it remain a separate (linked) executable? In other words, do I really need to learn Python? ;-) A signature.asc Description: OpenPGP digital signature ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Feature #5301 - Clone or Backup Persistent Volume
Spencer: >> Emma Peel (in copy) could be >> interested as she's been giving a workshop on Tails backups last week >> and 32C3. > > I found this [0] but no record of anything here [1]. Are there any > recordings of this workshop, or workshops in general, from this event? It was an ad-hoc session with no recording. ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Feature #5301 - Clone or Backup Persistent Volume
On 07/01/16 14:16, sajolida wrote: > > But we're definitely lacking people writing code in Tails as well, > especially Perl, so it's a good news in and off itself that you can > write Perl and do Debian packages! You may have to retract that after you read the code. ;-) I'm a sysadmin more than a developer. Let's just say this has been a greater learning experience than I first anticipated... A signature.asc Description: OpenPGP digital signature ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Feature #5301 - Clone or Backup Persistent Volume
Andrew Gallagher: > On 13/12/15 17:46, sajolida wrote: >> >> If you're interested in plugging this in the current code [1] that would >> be awesome. But as we haven't really finished the design decision around >> the backup scenarios, I can't promise anything regarding it's official >> acceptance yet. > > I have a reasonably stable version of the code available here: > > https://github.com/andrewgdotcom/tails-clone-persistent/releases/tag/v0.1.1-beta1 Amazing! > The easiest way to install it is to download the .deb, but it should(?) > build cleanly from the source. There's nothing particularly fancy in it. I want to first have a look at what would be the user scenario for this from a broader perspective in our plans regarding backups in Tails, before looking at the code. But we're definitely lacking people writing code in Tails as well, especially Perl, so it's a good news in and off itself that you can write Perl and do Debian packages! I'm still way too busy with finishing deliverables to look at your tool this week again, but I though maybe Emma Peel (in copy) could be interested as she's been giving a workshop on Tails backups last week and 32C3. > Some notes/caveats: > > 1. The GUI half of the code is still perl, sorry. So it's not directly > pluggable into tails-installer (yet!). It works fine as a standalone > GTK2 app and invokes tails-installer as required. Working code >> proper > standards and all that. ;-) > > 2. The GUI is not particularly pretty. > > 3. My C++ string handling sucks. Please don't look at it. I'll tidy it > up later. > > 4. If you type in the wrong password when updating an existing > persistent partition, it will fail hard rather than reprompt. > > 5. I've been testing it against Tails 2.0beta. It used to work against > 1.x also, but I might have broken that when I migrated to udisks2. > Haven't got around to checking it since. ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Feature #5301 - Clone or Backup Persistent Volume
On 13/12/15 17:46, sajolida wrote: > > If you're interested in plugging this in the current code [1] that would > be awesome. But as we haven't really finished the design decision around > the backup scenarios, I can't promise anything regarding it's official > acceptance yet. I have a reasonably stable version of the code available here: https://github.com/andrewgdotcom/tails-clone-persistent/releases/tag/v0.1.1-beta1 The easiest way to install it is to download the .deb, but it should(?) build cleanly from the source. There's nothing particularly fancy in it. Some notes/caveats: 1. The GUI half of the code is still perl, sorry. So it's not directly pluggable into tails-installer (yet!). It works fine as a standalone GTK2 app and invokes tails-installer as required. Working code >> proper standards and all that. ;-) 2. The GUI is not particularly pretty. 3. My C++ string handling sucks. Please don't look at it. I'll tidy it up later. 4. If you type in the wrong password when updating an existing persistent partition, it will fail hard rather than reprompt. 5. I've been testing it against Tails 2.0beta. It used to work against 1.x also, but I might have broken that when I migrated to udisks2. Haven't got around to checking it since. Andrew. signature.asc Description: OpenPGP digital signature ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Feature #5301 - Clone or Backup Persistent Volume
Andrew Gallagher: > On 11/12/15 10:57, sajolida wrote: >> We have "backup system for persistence" on our roadmap for 2016 [1][2] >> and I'll be the one to work on the broad picture, UX design, etc. So I'm >> definitely interested in your work but I wanted to test it before >> answering (but couldn't find the time to do so), tough I won't be able >> review your Perl code. > > That's fine, I can refactor it in python - although I'm not a python dev > so the first attempt may be a little ropey. I don't code seriously myself so I won't be able to review your Python code either :) But as u pointed out, Tails Installer is written in Python and if we ever want to propose a "clone everything" feature, I think it should be integrated in Tails Installer. If you're interested in plugging this in the current code [1] that would be awesome. But as we haven't really finished the design decision around the backup scenarios, I can't promise anything regarding it's official acceptance yet. [1]: https://git-tails.immerda.ch/liveusb-creator/ >> Right. At the moment I'm playing with Deja-Dup [3] for the backups of my >> Tails. I really like the integration in the desktop but, as it's running >> as the amnesia user, it can't create a perfect clone of the persistent >> volume. So your approach is also something we've considered, see #8812 [4]. > > I saw you had been working on it, hence why I wanted to check before > spending too much time on potentially duplicated effort. I think there's > value in both methods - the bootable clone is more useful for me and I'm > sure I can't be alone. I also think there's value in both methods. But we also prefer having one tool for one thing and avoid overlapping features. That's for me one of the main issue regarding the design of the backup tool: - "Clone everything" allows for a bootable backup of literally everything in persistence, including files owed by root. - Deja-Dup allows incremental and distant backups with a very good integration on the desktop but only for the files owned by `amnesia`. I wonder whether we could have to best of both worlds in a single tool, but I haven't taken the time to investigate this in depth really. If it's impossible, then I might propose to have both options. > So I'll work on and get back to you when I have something worth testing? Sure :) ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Feature #5301 - Clone or Backup Persistent Volume
On 11/12/15 10:57, sajolida wrote: > > Hi Andrew, sorry for not answering earlier but as u said we're all too > busy right now. It's the end of the year and we have to finish many > projects. Completely understandable! Sorry for nagging. > We have "backup system for persistence" on our roadmap for 2016 [1][2] > and I'll be the one to work on the broad picture, UX design, etc. So I'm > definitely interested in your work but I wanted to test it before > answering (but couldn't find the time to do so), tough I won't be able > review your Perl code. That's fine, I can refactor it in python - although I'm not a python dev so the first attempt may be a little ropey. > Right. At the moment I'm playing with Deja-Dup [3] for the backups of my > Tails. I really like the integration in the desktop but, as it's running > as the amnesia user, it can't create a perfect clone of the persistent > volume. So your approach is also something we've considered, see #8812 [4]. I saw you had been working on it, hence why I wanted to check before spending too much time on potentially duplicated effort. I think there's value in both methods - the bootable clone is more useful for me and I'm sure I can't be alone. So I'll work on and get back to you when I have something worth testing? Thanks, Andrew signature.asc Description: OpenPGP digital signature ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Feature #5301 - Clone or Backup Persistent Volume
On 10/12/15 19:16, u wrote: > Hi Andrew, > > Andrew Gallagher: >> Nobody interested? > > I guess most of us are simply quite busy atm - but yes, as we created > this ticket initially, we are interested in such a solution. Thank you > for working on this. Sorry, that may have come out more sarcastic than I intended! I saw the ticket had been around for a while and someone did appear to be working on it so I wanted to check before duplicating effort. > I had a quick view, but I am not knowledgeable in Perl and just saw that > you use rsync, as suggested also in our blueprint on the matter: > https://tails.boum.org/blueprint/backups/ > Did you read it? Yes, the "Clone everything" option is sufficient for my needs, so I coded only for it. > The Installer is written in Python and uses GTK so I suppose that a > backup that wants to get integrated into the installer should probably > use Python (correct me if that is totally wrong, please). I'm assuming that also. I wrote it in perl because that's what I'm most comfortable with - and I initially discounted the likelihood of integrating it into the official code. Andrew. signature.asc Description: OpenPGP digital signature ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Feature #5301 - Clone or Backup Persistent Volume
u: > Andrew Gallagher: >> Nobody interested? > > I guess most of us are simply quite busy atm - but yes, as we created > this ticket initially, we are interested in such a solution. Thank you > for working on this. Hi Andrew, sorry for not answering earlier but as u said we're all too busy right now. It's the end of the year and we have to finish many projects. We have "backup system for persistence" on our roadmap for 2016 [1][2] and I'll be the one to work on the broad picture, UX design, etc. So I'm definitely interested in your work but I wanted to test it before answering (but couldn't find the time to do so), tough I won't be able review your Perl code. [1]: https://tails.boum.org/blueprint/roadmap_2016-2017/ [2]: https://labs.riseup.net/code/issues/5301 > I had a quick view, but I am not knowledgeable in Perl and just saw that > you use rsync, as suggested also in our blueprint on the matter: > https://tails.boum.org/blueprint/backups/ > Did you read it? > > The Installer is written in Python and uses GTK so I suppose that a > backup that wants to get integrated into the installer should probably > use Python (correct me if that is totally wrong, please). Right. At the moment I'm playing with Deja-Dup [3] for the backups of my Tails. I really like the integration in the desktop but, as it's running as the amnesia user, it can't create a perfect clone of the persistent volume. So your approach is also something we've considered, see #8812 [4]. [3]: https://wiki.gnome.org/action/show/Apps/DejaDup?action=show&redirect=DejaDup [4]: https://labs.riseup.net/code/issues/8812 ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Feature #5301 - Clone or Backup Persistent Volume
Hi Andrew, Andrew Gallagher: > Nobody interested? I guess most of us are simply quite busy atm - but yes, as we created this ticket initially, we are interested in such a solution. Thank you for working on this. I had a quick view, but I am not knowledgeable in Perl and just saw that you use rsync, as suggested also in our blueprint on the matter: https://tails.boum.org/blueprint/backups/ Did you read it? The Installer is written in Python and uses GTK so I suppose that a backup that wants to get integrated into the installer should probably use Python (correct me if that is totally wrong, please). Cheers! u. ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Feature #5301 - Clone or Backup Persistent Volume
Nobody interested? To be clear, I'm willing to put in the work. I just don't want to spend too much time on it if it isn't going to go anywhere. Andrew. On 03/12/15 12:51, Andrew Gallagher wrote: > Hi, all. > > I've written a tool to clone the contents of a mounted persistent volume > (or any other given directory tree) to a fresh or existing persistent > volume on another tails drive. It was done as part of a pet project, but > this functionality really belongs in tails itself. > > It consists of two parts: a perl wrapper that handles user prompts for > passphrases etc. and a small setuid helper that performs the actual > work. The setuid helper will create a partition/filesystem if necessary > and then mounts, rsyncs and unmounts - this requires the passphrase to > be entered multiple times, so the perl wrapper prompts for one itself > and uses expect to supply it to the helper. > > There are currently two main caveats. Firstly, the size of the tails > primary partition is hardcoded in the helper. It should be possible to > extract this instead from the output of parted, and is probably just a > matter of spending an hour or two on it. The other one is that the > wrapper depends on Term::Choose, which isn't packaged in either tails or > debian. I solved this by aliening a perl-term-choose deb but don't want > to be responsible for it, so the user interface really should be rebuilt > (suggestions welcome!). Of course it would probably be best if it could > be integrated into the tails installer instead... ;-) > > You can find the code here: > > https://github.com/andrewgdotcom/frith/tree/master/src/tcp-helper.c > https://github.com/andrewgdotcom/frith/tree/master/bin/tails-clone-persistent > > perl-term-choose can be found here: > > https://andrewg.com/debian/pool/main/p/perl-term-choose/perl-term-choose_1.202.0-2-abg2_all.deb > > Let me know if you think it is useful. > > Andrew. signature.asc Description: OpenPGP digital signature ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
