Re: [gentoo-user] File system meta-data indexer / checker
On Mon, Jun 17, 2013 at 8:52 AM, Florian Philipp li...@binarywings.net wrote: Am 16.06.2013 23:14, schrieb Ciprian Dorin Craciun: Hello all! While struggling with managing various old backups --- just imagine 10 or so copies of almost the same content, some with `rsync`, some with `rdiff-backup`, yet some others on plain ISO's, all over a range of a few years --- I stumbled upon the following missing piece in the Linux tools ecosystem: a file-system crawler that records **only** meta-data, like all the info available through `stat`, plus an assortment of hashes of the files (at least MD5 and SHA-1,2 family), and optionally ACL's and extended attributes. Thus I was wondering if someone knows a tool that fits this description. [...] Thus, are there any other alternatives? (Just to be clear, I don't need a backup solution, just something to record file-system meta-data. Maybe a meta-backup solution... :) ) Ciprian. Do you mean something like `cp --attributes-only`? Nop, wouldn't do the job, because `cp --attributes-only` requires a target file system, however I need something which records those attributes in a file that I can use afterwards to restore them (or compare them with further versions, etc.). Ciprian.
Re: [gentoo-user] File system meta-data indexer / checker
On Mon, Jun 17, 2013 at 10:21 AM, Peter Humphrey pe...@humphrey.ukfsn.org wrote: On Monday 17 Jun 2013 07:00:35 Ciprian Dorin Craciun wrote: On Mon, Jun 17, 2013 at 8:52 AM, Florian Philipp li...@binarywings.net wrote: Do you mean something like `cp --attributes-only`? Nop, wouldn't do the job, because `cp --attributes-only` requires a target file system, ... You could always cp to /dev/null. Sorry I don't follow... What does imply to cp to /dev/null, and what would be the outcome of that?
Re: [gentoo-user] File system meta-data indexer / checker
On Mon, Jun 17, 2013 at 6:56 PM, Peter Humphrey pe...@humphrey.ukfsn.org wrote: On Monday 17 Jun 2013 08:25:08 Ciprian Dorin Craciun wrote: On Mon, Jun 17, 2013 at 10:21 AM, Peter Humphrey pe...@humphrey.ukfsn.org wrote: On Monday 17 Jun 2013 07:00:35 Ciprian Dorin Craciun wrote: On Mon, Jun 17, 2013 at 8:52 AM, Florian Philipp li...@binarywings.net wrote: Do you mean something like `cp --attributes-only`? Nop, wouldn't do the job, because `cp --attributes-only` requires a target file system, ... You could always cp to /dev/null. Sorry I don't follow... What does imply to cp to /dev/null, and what would be the outcome of that? /dev/null would be the target file system you referred to. It's a bottomless empty pit, so no physically real copy would be made. I would be curious to find out how one can use `/dev/null` as a replacement for an **entire** file-system (i.e. not just a replacement for a stream sink)? And how would the `cp` command be used to obtain such an outcome?
[gentoo-user] File system meta-data indexer / checker
Hello all! While struggling with managing various old backups --- just imagine 10 or so copies of almost the same content, some with `rsync`, some with `rdiff-backup`, yet some others on plain ISO's, all over a range of a few years --- I stumbled upon the following missing piece in the Linux tools ecosystem: a file-system crawler that records **only** meta-data, like all the info available through `stat`, plus an assortment of hashes of the files (at least MD5 and SHA-1,2 family), and optionally ACL's and extended attributes. Thus I was wondering if someone knows a tool that fits this description. I must say I've tried to do my homework, and below are a few tools that come close, but not quite... (A) `rdup` is probably the closest to what I'm searching. However there are a few issues: * it's output format is not very parsable, especially in cases like symlinks, and a few other special cases; * it doesn't escape the file names --- and from some reason I have files containing escape sequences in them... * it records only SHA-1; * it doesn't handle ACL's or extended attributes; (B) `mtree` from FreeBSD. I found two ports of it for Linux, however my main concern is how parsable is the output... (C) `md5deep` (or `sha*deep`), which only records the checksum not other meta-data. Thus, are there any other alternatives? (Just to be clear, I don't need a backup solution, just something to record file-system meta-data. Maybe a meta-backup solution... :) ) Ciprian.
Re: [gentoo-user] How reliable is ext3?
On Tue, Apr 23, 2013 at 11:39 PM, Hilco Wijbenga hilco.wijbe...@gmail.com wrote: [...] So when I needed to install a new machine, I looked around and settled on JFS. This box has been running for about half a year now (so that includes several power failures) without any problems. I certainly am very pleased with JFS so perhaps you might want to consider it. I've also used (and still use) JFS on a lot of partitions (LVM actually), from my laptops (both rotating and SSD), desktop, VM's, etc. I've moved to it a few years ago after getting tired of all the Ext3 fsck's. Although JFS is quite efficient, and didn't create too much trouble --- never lost an entire file-system, never corrupted data, etc. --- it does have a few quirks: * empty files after panics --- I think in this regard it's not JFS's fault, but actually badly written software, because things go like this: say you edit a file, save it, and immediately (a few seconds) get either a panic or power failure, the result is an empty file; the technical details are like this: some software first truncate the file, write to it, and close it, but don't sync the data, thus you end up with an empty file; as said I think JFS is correct here, because you don't get a mix of old and new data, etc.; however I've encountered this behavior in quite a few instances... * no TRIM support --- obviously really useful on SSD and virtualized disks; (although I remember there was some work done in this respect;) * not enough tooling --- you get only the `jfs-utils`, and that's kind of it... * small community --- if you have a question, you can use the mailing list, it's quite responsive, but there aren't many data-points so that you can easily find someone in a similar situation, thus with a solution... All in all, I've started gradually migrating my partitions on Ext4. I stay away for Btrfs for now. And to be frank I don't quite like Btrfs's, and ZFS's for that matter, approach of throwing together all the layers, from the file-system, to the RAID, to the block management, etc. I find the layered approach more appealing --- as in if something goes wrong you can poke around --- of having completely separated block device management (LVM), RAID (MD), and file-system. A... and for backup file-systems, I use Ext2. Why? My take on this is: * I don't need write or read performance; I don't mind long fsck's; (thus any file-system could fit in here, however see below;) * I do really need reliability and, most importantly, recovery in case s**t... Therefore Ext2 is a perfect match: * it is so old, that I guess by now most bugs have been found and squashed; * it is so old, that virtually any Linux (or Windows, FreeBSD, or most other knows OS's) are able to at least read it; * it is so old, that by now I bet there are countless recovery tools; * it is so simple (compared with others), that someone could just re-implement a reader for it, or recovery tools; Any feedback about the Ext2 for backups? (Hope I'm not wrong on this one...) Ciprian.
Re: [gentoo-user] {OT} open-source: chat, tasks, resources, code
On Mon, Dec 17, 2012 at 9:26 AM, Grant emailgr...@gmail.com wrote: When I need a new web-based software tool, I consider writing it myself and if that isn't feasible I try to use something open-source and self-hosted. I need something for chat, task management, resource management, and code management, all for groups. I'm considering Campfire, Trello, Float, and GitHub respectively, but I thought I'd check with you guys to see if any of this is available in an open-source and self-hosted form, especially in portage. - Grant You should take a look at Fosil: http://www.fossil-scm.org/index.html/doc/trunk/www/index.wiki I haven't used it myself, but if you don't have strict preferences for the versioning system you could use that, as it's very self contained and easy to use. Ciprian.
Re: [gentoo-user] The LIGHTEST web server (just for serving files) ?
On Sat, Nov 12, 2011 at 10:24, Pandu Poluan pa...@poluan.info wrote: What is the *LIGHTEST* web server package you know for gentoo? I just want to serve the distfiles, so no CGI / PHP / whathaveyouscripting support is needed. Preferably, with logging so I can see which packages I missed, but not necessary. Rgds, We use for such purpose http://linux.bytesex.org/misc/webfs.html . It features both (reasonable) performance (threaded), security (chroot, SSL) and simplicity (can be statically linked, can be controlled solely from arguments). For example (or add -d for debugging): webfsd -F -p 80 -r . Ciprian.
Re: [gentoo-user] Re: initramfs RAID at boot time
On Sun, Apr 18, 2010 at 10:42, Jarry mr.ja...@gmail.com wrote: On 18. 4. 2010 8:57, Ciprian Dorin, Craciun wrote: * there is an option for the kernel that must be enabled at compile time that enables automatic RAID detection and assembly by the kernel before mounting /, but it works only for MD metadata 0.96 (see [1]); * the default metadata for `mdadm` is 1.2 (see `man mdadm`, and search for `--metadata`), so when creating the RAID you must explicitly select the metadata you want; [1] http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/md.txt;h=188f4768f1d58c013d962f993ae36483195fd288;hb=HEAD Which version of mdadm are you using? I have 3.0, and defalut metadata is 0.90: -e , --metadata= Declare the style of RAID metadata (superblock) to be used. The default is 0.90 for --create, and to guess for other operations. The default can be overridden by setting the metadata value for the CREATE keyword in mdadm.conf. BTW [1] says about kernel 2.6.9, things might have changed since then... Jarry On my laptop on which I've made the experiments I have ArchLinux (which always has the bleeding edge packages), I have 3.1.2. So maybe between 3.0 and 3.1 there was this switch from 0.90 to 1.2 default metadata. About the autodetection stuff I'm absolutely positive that it only handles 0.90 format as I've tried it and didn't work with the 1.x version of superblock format. Ciprian.
Re: [gentoo-user] Re: initramfs RAID at boot time
On Sun, Apr 18, 2010 at 11:16, Neil Bothwick n...@digimed.co.uk wrote: On Sun, 18 Apr 2010 09:57:38 +0300, Ciprian Dorin, Craciun wrote: Also a question for about /boot on RAID1... I didn't manage to make it work... Could you Neil please tell me exactly how you did this? I'm most interested in how you've convinced Grub to work... You just don't tell GRUB that it's working with one half of a RAID1 array. Unlike all other RAID level, with 1 you can also access the individual disks. -- Neil Bothwick Well I've tried exactly that: I've aggregated two partitions in RAID1, made the file system, then tried to install Grub on them (as in run grub-setup or grub-install or grub and then from the shell the setup)... And I didn't succeeded. I've tried the following: * try to install Grub on the MD as it would have been a partition -- failed (as expected as the MD device is not on a hard-drive); * stopped the MD, and then tried to install grub on each partition individually -- it worked to install, but from a reason I don't remember right now it failed to boot; So what intrigues me is how you've initialized the MBR, how you've runned grub-setup? (In the end I am more pleased with two boot partitions, as if I miss-configure one, I'll have the other one to boot from. I've also cross-referenced the grub menu to chain-load the other disk.) Thanks, Ciprian.
Re: [gentoo-user] Re: initramfs RAID at boot time
On Sun, Apr 18, 2010 at 1:01 AM, Neil Bothwick n...@digimed.co.uk wrote:
On Sat, 17 Apr 2010 14:36:39 -0700, Mark Knecht wrote:
Empirically any way there doesn't seem to be a problem. I built the
new kernel and it booted normally so I think I'm misinterpreting what
was written in the Wiki or the Wiki is wrong.
As long as /boot is not on RAID, or is on RAID1, you don't need an
initrd. I've been booting this system for years with / on RAID1 and
everything else on RAID5.
From my research on the topic (I also wanted to have both /boot
and / on RAID1) there are the following traps:
* there is an option for the kernel that must be enabled at
compile time that enables automatic RAID detection and assembly by the
kernel before mounting /, but it works only for MD metadata 0.96 (see
[1]);
* the default metadata for `mdadm` is 1.2 (see `man mdadm`, and
search for `--metadata`), so when creating the RAID you must
explicitly select the metadata you want;
* indeed the preferred may to do it is using an initramfs; (I've
posted below some shell snippets that create do exactly this: assemble
my RAID); (the code snippets are between {{{...}}}, it's from a
MoinMoin wiki page;)
Also a question for about /boot on RAID1... I didn't manage to
make it work... Could you Neil please tell me exactly how you did
this? I'm most interested in how you've convinced Grub to work...
Best,
Ciprian.
[1]
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=Documentation/md.txt;h=188f4768f1d58c013d962f993ae36483195fd288;hb=HEAD
Init-ramfs preparation
{{{
mkdir -p /usr/src/initramfs
cd /usr/src/initramfs
mkdir /usr/src/initramfs/bin
mkdir /usr/src/initramfs/dev
mkdir /usr/src/initramfs/proc
mkdir /usr/src/initramfs/rootfs
mkdir /usr/src/initramfs/sys
cp -a /bin/busybox /usr/src/initramfs/bin/busybox
cp -a /sbin/mdadm /usr/src/initramfs/bin/mdadm
cp -a /sbin/jfs_fsck /usr/src/initramfs/bin/jfs_fsck
cp -a /dev/console /usr/src/initramfs/dev/console
cp -a /dev/null /usr/src/initramfs/dev/null
cp -a /dev/sda2 /usr/src/initramfs/dev/sda2
cp -a /dev/sdc2 /usr/src/initramfs/dev/sdc2
cp -a /dev/md127 /usr/src/initramfs/dev/md127
}}}
{{{
cat /usr/src/initramfs/init 'EOS'
#!/bin/busybox ash
exec /dev/null /dev/null 2/dev/console
exec 12
/bin/busybox mount -n -t proc none /proc || exit 1
/bin/busybox mount -n -t sysfs none /sys || exit 1
/bin/mdadm -A /dev/md127 -R -a md /dev/sda2 /dev/sdc2 || exit 1
/bin/jfs_fsck -p /dev/md127 || true
/bin/busybox mount -n -t jfs /dev/md127 /rootfs -o
ro,exec,suid,dev,relatime,errors=remount-ro || exit 1
/bin/busybox umount -n /sys || exit 1
/bin/busybox umount -n /proc || exit 1
# /bin/busybox ash /dev/console /dev/console 2/dev/console || exit 1
exec /bin/busybox switch_root /rootfs /sbin/init || exit 1
exit 1
EOS
chmod +x /usr/src/initramfs/init
}}}
{{{
( cd /usr/src/initramfs ; find . | cpio --quiet -o -H newc | gzip -9
/boot/initramfs )
}}}
Re: [gentoo-user] OT:Choosing a filesystem
On Thu, Apr 1, 2010 at 10:09 PM, meino.cra...@gmx.de wrote: [ ... snip ... ] So I have a lot of docs (specs of microcontrollers, howtos, programm and source code docs...etc) on my disk. This one part. I've seen that nobody mentioned JFS yet... :) In some benchmarks the best FS for most tasks is either XFS and JFS, but it seems that JFS has less CPU and memory usage. So for small and medium files I would say it's best. (I think it was on Tom's Hardware site?) I'll also describe my history on the issue: initially I've only used ReiserFS until something (not the hard drive) just snapped and I've almost lost all my data. At that moment I've migrated to Ext3. But Ext3 has the problem of needing constant (usually once a moth) checking (I know this is optional or tunable but it seems it is recommended) which for large file systems takes incredibly long (60GB HDD takes about 2 or 3 minutes... So imagine what would to to 1TB...) So I got angry again and moved to JFS... And I'm using JFS for about two years without major incidents... (Only once I've lost the contents of a configuration file due to a power interruption but this is because of the editor.) So as a conclusion for this task I would recommend JFS (I also have 200GB of documentation which covers about 100 thousand files I guess.) Also see at the end for my notes on journaled file systems. Then: I often transer videos from my DVB-T-receiver/recorder to my harddisk to cut out the advertising and to transcode the videos to somethings better than ts (transport streams), This is another part. Although JFS could handle this, maybe a file system specially designed for this would do best: Ext4 with it's extent feature. (But be aware that by just using a file system is not enough... The software also has to be specially crafted if you want high performance. Just see the `fallocate` and `fadvise` system calls.) Then I plan to have two roots this time: One to experiment with and one good and stable-version which is used/updated/... strictly as recommended. Filesizes and usage do vary here...take a look at your own roots ;))) :) This sounds like my setup: 160GB HDD from my laptop has the following layout: * GPT partition table (not MBR) -- this gives me more partitions without needing the extended partition feature of MBR; * 2 boot partitions of 512MB (maybe 1GB would have been better) -- one for current usage (Grub 0.97 with GPT patches) and one for experimentation; these are Ext2 for safety and compatibility; * 3 root partitions of 4GB (I should have made them 8GB) -- one for the current operating system, and two for future upgrades / experimentation; currently JFS and maybe also so in the future; * 1 swap of 8GB (encrypted with random password with the help of dm-crypt); * rest of the HDD as one big partition with LVM; (large extents 256MB); * from the LVM I have partitions for personal data (/home) and other things -- everything is JFS; Then I want something encrypted, either as a partition or as a files (carrying a encrypted fs), which I can copy to dvd and will be able to mount this dvd and use it without to have to copy the whole dvd first to harddisk before using it... Currently I am using encfs...(outdated?). What can I do use instead? This is for personal things like letters, photos, texts ... etc. Files vary from some kb up to about 2GByte (guessed). Most of them smaller than 200MByte As someone noted maybe EncryptFS (in kernel one) would be better... (It's an install option in Ubuntu so I would say it's mature enough.) But for this encrypted purpose I would use dm-crypt with `aes-xts-essiv:sha256` encryption. (In the past I've used LoopAES but I had some minor issues with kernel building as it's not in the vanilla kernel...) Last thing: I have a lot iof copies of code from svn repositories because I like to have the bleeding edge of some projects (do you know the new Blender 2.50??? :O) I also have a lot of repositories on JFS and everything works nice. This implies a lot of compile work. This will be the only case where files are created as often as read. For temporary folders while compiling I would recommend to instruct your build scripts to build inside /tmp where you have tmpfs mounted... It's blazingly fast... And some notes about journaled file systems: they journal meta-data (that is file creation, deletion, rename, etc), not data (that is the contents)... (Of course there are a few (Ext3 maybe?) file systems that have the option to also journal data...) What does this mean: well when you edit a file and save it and then cut the power, the file still exists (the meta-data), but the contents could (and usually is) wrong: either no content (like I've encountered once with JFS), either mixed content (old and new)... So the fineprint here is: no journaled file system is safe... They are all safe if you
