Re: [gentoo-user] Recover on SSD
On 5 May 2013, at 17:16, Hinnerk van Bruinehsen wrote: ... The data on a SSD is not necessarily stored linar so it's not said that a new partition is using the same memory cells as the old one. … For a HDD I'd advise to create a copy using dd but from my understanding of SSD technology it's not guaranteed to copy the right (now unused marked) blocks. Is anyone able to elaborate on this, please? I think I've had a eureka! moment of understanding whilst preparing to compose this reply, but I've always been sceptical of these kinds of statements in the past. Surely flash memory devices must present themselves to the o/s as block devices, because that's how all storage devices work, right? If I'm now understanding correctly, SSDs present themselves to the o/s as block devices more or less as convenient or necessary. They can be treated as such as long as all the data required is listed in the file allocation table. I'm left wondering how the SSD knows that a file has been deleted, and whether this works for all conceivable file-systems. Stroller.
Re: [gentoo-user] Using date/time variables in cronjobs?
On Sun, 05 May 2013 19:21:18 -0400, Tanstaafl wrote: For example, I want to keep 17 hourlies, and 30 nightlies, so I have two cron jobs set up, the hourly, and the nightly. Each backs up to a separate dir. So each time your backup fails, you reduce the number of available backups by one. If you want to use this approach, delete one backup AFTER you have created AND tested a new backup. I'm thinking the easiest way would be to find and delete the oldest file in the backup target directory before executing the backup command. For the hourlies dir, I'd just find the files that are older than one day - so maybe: What about rm -f $(ls -1t backuppattern* | tail -n +numbertokeep) -- Neil Bothwick The Computer is the logical advancement of humankind: intelligence without morality. signature.asc Description: PGP signature
Re: [gentoo-user] Recover on SSD
On Mon, May 06, 2013 at 07:50:52AM +0100, Stroller wrote: On 5 May 2013, at 17:16, Hinnerk van Bruinehsen wrote: ... The data on a SSD is not necessarily stored linar so it's not said that a new partition is using the same memory cells as the old one. … For a HDD I'd advise to create a copy using dd but from my understanding of SSD technology it's not guaranteed to copy the right (now unused marked) blocks. Is anyone able to elaborate on this, please? I think I've had a eureka! moment of understanding whilst preparing to compose this reply, but I've always been sceptical of these kinds of statements in the past. Surely flash memory devices must present themselves to the o/s as block devices, because that's how all storage devices work, right? If I'm now understanding correctly, SSDs present themselves to the o/s as block devices more or less as convenient or necessary. They can be treated as such as long as all the data required is listed in the file allocation table. I'm left wondering how the SSD knows that a file has been deleted, and whether this works for all conceivable file-systems. The problem is that you can't delete on a flash cell. The process is simplified: read cell - delete to be deleted stuff in memory - write memory contents back. Since flash cells can only be written to a fixed amount of times (afterwards they become unreliable) there is a concept called wear leveling. This means essentially that your 128 GB flash drive in reality hasn't just 128 GB of storage but e.g. 256GB. To spread out the writes it reads one cell, does the memory operation and write the contents back to another cell while marking the old cell as unused. This means two things: you can't really delete something securely (noteven with tools like shred) and you can't access overwritten data (because it's now inside the unused-marked area). There is a special command (TRIM [1]) that does the marking after something was deleted to counter perfomance degradation. Fun fact: most SSDs that offer a secure delete feature (whole disk) don't really delete anything but are internally encrypted and throw away the encryption key and generate a new one on receiving the secure delete command which leads to unreadable data and therefore is a kind of secure deletion (unless the underlying crypto is broken). WKR Hinnerk [1] http://en.wikipedia.org/wiki/TRIM signature.asc Description: Digital signature
Re: [gentoo-user] Recover on SSD
On Sun, May 5, 2013 at 7:49 PM, Randolph Maaßen r.maasse...@gmail.com wrote: I'm so damn lucky I dd'ed the SSD onto an external drive and worked at first on the image with qemu. A simple recreation of the partition brought the system back to live on the image. I tried the same on the real machine and Gentoo works again. Wow, congratulations. In the old days when I first started using Linux, I used to keep a backup of my boot sector and partition table on a floppy disk. Maybe I should consider this practice again as part of my backup routine. :)
[gentoo-user] Re: Nautical Gentoo hardware suggestions?
Stroller stroller at stellar.eclipse.co.uk writes: … sci-geosciences/opencpn is in portage [1] However, much of the time, the Gentoo_nav_hadware will be on a 17 foot boat (damp and salty if not wet) … So I'm looking for marine grade hardware onto which installation of Gentoo is reasonably straightforward. I don't know, but I think I'd start by looking at Android tablets. Here is what I've decided on: ChromeOS Arm-15 lappy: Samsung Arm Chromebook: http://dev.chromium.org/chromium-os/developer-information-for-chrome-os-devices/samsung-arm-chromebook It pretty much runs Gentoo now, according to Vapier http://dev.chromium.org/ I'm sure there's an ARM version of Debian or Ubuntu that has been released for mobile devices. Better than that, there are linux distros for with a Nautical focus: http://marinux.pk973.org/ with some waterproof bags, duck tape /or rubber sealant you could get a very waterproof platform for not much money. Better idea: Disassemble the lappy and spray down the boards with a conformal coating: http://en.wikipedia.org/wiki/Conformal_coating thanks, James
Re: [gentoo-user] Recover on SSD
Am 06.05.2013 13:00, schrieb Hinnerk van Bruinehsen: On Mon, May 06, 2013 at 07:50:52AM +0100, Stroller wrote: On 5 May 2013, at 17:16, Hinnerk van Bruinehsen wrote: ... The data on a SSD is not necessarily stored linar so it's not said that a new partition is using the same memory cells as the old one. … For a HDD I'd advise to create a copy using dd but from my understanding of SSD technology it's not guaranteed to copy the right (now unused marked) blocks. Is anyone able to elaborate on this, please? I think I've had a eureka! moment of understanding whilst preparing to compose this reply, but I've always been sceptical of these kinds of statements in the past. Surely flash memory devices must present themselves to the o/s as block devices, because that's how all storage devices work, right? If I'm now understanding correctly, SSDs present themselves to the o/s as block devices more or less as convenient or necessary. They can be treated as such as long as all the data required is listed in the file allocation table. I'm left wondering how the SSD knows that a file has been deleted, and whether this works for all conceivable file-systems. The problem is that you can't delete on a flash cell. The process is simplified: read cell - delete to be deleted stuff in memory - write memory contents back. Since flash cells can only be written to a fixed amount of times (afterwards they become unreliable) there is a concept called wear leveling. This means essentially that your 128 GB flash drive in reality hasn't just 128 GB of storage but e.g. 256GB. To spread out the writes it reads one cell, does the memory operation and write the contents back to another cell while marking the old cell as unused. emm - no. Wear leveling does not need any spare blocks. A lot of drives do have spare blocks, but those are never the same size of the original size (at least not on drives you can buy for a sensible amount of money). More like 120+8 or 160+16 or 256+16. The spare blocks are used like on a hdd: some block goes bad, another one is mapped in. Since the sdd firmware does not know if something was deleted or not* - it does know shit about filesystems**, you can of course dd an image, if you want to. Just like on a hdd. *there are drives that do garbage collection without TRIM for fat and or ntfs.. so they seem to know a bit about filesystems. ** and this is why TRIM exists in the first place. To tell the drive: yes, this data is gone. You don't need to care about it anymore.
Re: [gentoo-user] Recover on SSD
On Mon, May 06, 2013 at 07:34:20PM +0200, Volker Armin Hemmann wrote: emm - no. Wear leveling does not need any spare blocks. A lot of drives do have spare blocks, but those are never the same size of the original size (at least not on drives you can buy for a sensible amount of money). More like 120+8 or 160+16 or 256+16. The spare blocks are used like on a hdd: some block goes bad, another one is mapped in. Since the sdd firmware does not know if something was deleted or not* - it does know shit about filesystems**, you can of course dd an image, if you want to. Just like on a hdd. *there are drives that do garbage collection without TRIM for fat and or ntfs.. so they seem to know a bit about filesystems. ** and this is why TRIM exists in the first place. To tell the drive: yes, this data is gone. You don't need to care about it anymore. The actual numbers were made up to make the point (maybe I should have stated that in my OP). According to [1] they are normally between 7% - 37%. Linux supports TRIM since Kernel 2.6.28. It's supported for several filesystems (Ext4, Btrfs, FAT, GFS2 and XFS) but must be enabled via the discard mount option. I don't have definitive information for Windows but it seems to be supported by at least Windows 7 (as far as I can tell without any user interaction). Since the deletion happened under Windows I made a guess that it is not totally unreasonable that dd may not work (if the deleted data would have been TRIMed). [1] http://www.lsi.com/downloads/Public/Flash%20Storage%20Processors/LSI_PRS_FMS2012_TE21_Smith.pdf signature.asc Description: Digital signature
Re: [gentoo-user] Recover on SSD
On 06/05/2013 20:36, Hinnerk van Bruinehsen wrote: On Mon, May 06, 2013 at 07:34:20PM +0200, Volker Armin Hemmann wrote: emm - no. Wear leveling does not need any spare blocks. A lot of drives do have spare blocks, but those are never the same size of the original size (at least not on drives you can buy for a sensible amount of money). More like 120+8 or 160+16 or 256+16. The spare blocks are used like on a hdd: some block goes bad, another one is mapped in. Since the sdd firmware does not know if something was deleted or not* - it does know shit about filesystems**, you can of course dd an image, if you want to. Just like on a hdd. *there are drives that do garbage collection without TRIM for fat and or ntfs.. so they seem to know a bit about filesystems. ** and this is why TRIM exists in the first place. To tell the drive: yes, this data is gone. You don't need to care about it anymore. The actual numbers were made up to make the point (maybe I should have stated that in my OP). According to [1] they are normally between 7% - 37%. Linux supports TRIM since Kernel 2.6.28. It's supported for several filesystems (Ext4, Btrfs, FAT, GFS2 and XFS) but must be enabled via the discard mount option. I don't have definitive information for Windows but it seems to be supported by at least Windows 7 (as far as I can tell without any user interaction). Since the deletion happened under Windows I made a guess that it is not totally unreasonable that dd may not work (if the deleted data would have been TRIMed). [1] http://www.lsi.com/downloads/Public/Flash%20Storage%20Processors/LSI_PRS_FMS2012_TE21_Smith.pdf A delete on an SSD is a very expensive operation, to my mind it seems completely unreasonable to think that Windows would try and clear many tens of GB just because it trashed a partition table. It would take _hours_ to clear those blocks. By far the easiest route would be to just do what is done for spinning disks - write the partition table, leave whatever junk is in the cells intact until the partition is formatted and actual data is written to the fs. As your results show, this is indeed what did happen. -- Alan McKinnon alan.mckin...@gmail.com
[gentoo-user] Calibre Update Problems
Hello, i run @world update. By calibre 0.9.29 broke the Process with follow message: quote calibre successfully installed. You can start it by running the command calibre There were 1 warning(s): * Setting up completion failed with error: install: der Aufruf von stat für „/var/tmp/portage/app-text/calibre-0.9.29/image/usr/etc/bash_completion.d/calibre“ ist nicht möglich: Datei oder Verzeichnis nicht gefunden !!! doins: /var/tmp/portage/app-text/calibre-0.9.29/image/usr/etc/bash_completion.d/calibre does not exist * ERROR: app-text/calibre-0.9.29 failed (install phase): * doins failed * * If you need support, post the output of `emerge --info '=app-text/calibre-0.9.29'`, * the complete build log and the output of `emerge -pqv '=app-text/calibre-0.9.29'`. * The complete build log is located at '/var/tmp/portage/app-text/calibre-0.9.29/temp/build.log'. * The ebuild environment file is located at '/var/tmp/portage/app-text/calibre-0.9.29/temp/environment'. * Working directory: '/var/tmp/portage/app-text/calibre-0.9.29/work/calibre' * S: '/var/tmp/portage/app-text/calibre-0.9.29/work/calibre' * QA Notice: file does not exist: * * doins: /var/tmp/portage/app-text/calibre-0.9.29/image/usr/etc/bash_completion.d/calibre does not exist Failed to emerge app-text/calibre-0.9.29, Log file: '/var/tmp/portage/app-text/calibre-0.9.29/temp/build.log' * Messages for package app-text/calibre-0.9.29: * ERROR: app-text/calibre-0.9.29 failed (install phase): * doins failed * * If you need support, post the output of `emerge --info '=app-text/calibre-0.9.29'`, * the complete build log and the output of `emerge -pqv '=app-text/calibre-0.9.29'`. * The complete build log is located at '/var/tmp/portage/app-text/calibre-0.9.29/temp/build.log'. * The ebuild environment file is located at '/var/tmp/portage/app-text/calibre-0.9.29/temp/environment'. * Working directory: '/var/tmp/portage/app-text/calibre-0.9.29/work/calibre' * S: '/var/tmp/portage/app-text/calibre-0.9.29/work/calibre' /quote gentoo-mobile siefke # emerge --info =app-text/calibre-0.9.29 - http://nopaste.info/166126bafc.html gentoo-mobile siefke # emerge -pqv =app-text/calibre-0.9.29 [ebuild R ] app-text/calibre-0.9.29 USE=udisks Has someone advice what is the Problem. The missing QT lib is present, but why is mistake? Missing Doins i not understand on my gentoo give package not. Thanks for help. Thank you Greetings Silvio
Re: [gentoo-user] Calibre Update Problems
On 05/06/2013 03:15 PM, Silvio Siefke wrote: Hello, i run @world update. By calibre 0.9.29 broke the Process with follow message: quote calibre successfully installed. You can start it by running the command calibre There were 1 warning(s): * Setting up completion failed with error: install: der Aufruf von stat für „/var/tmp/portage/app-text/calibre-0.9.29/image/usr/etc/bash_completion.d/calibre“ ist nicht möglich: Datei oder Verzeichnis nicht gefunden !!! doins: /var/tmp/portage/app-text/calibre-0.9.29/image/usr/etc/bash_completion.d/calibre does not exist * ERROR: app-text/calibre-0.9.29 failed (install phase): * doins failed * * If you need support, post the output of `emerge --info '=app-text/calibre-0.9.29'`, * the complete build log and the output of `emerge -pqv '=app-text/calibre-0.9.29'`. * The complete build log is located at '/var/tmp/portage/app-text/calibre-0.9.29/temp/build.log'. * The ebuild environment file is located at '/var/tmp/portage/app-text/calibre-0.9.29/temp/environment'. * Working directory: '/var/tmp/portage/app-text/calibre-0.9.29/work/calibre' * S: '/var/tmp/portage/app-text/calibre-0.9.29/work/calibre' * QA Notice: file does not exist: * *doins: /var/tmp/portage/app-text/calibre-0.9.29/image/usr/etc/bash_completion.d/calibre does not exist Failed to emerge app-text/calibre-0.9.29, Log file: '/var/tmp/portage/app-text/calibre-0.9.29/temp/build.log' * Messages for package app-text/calibre-0.9.29: * ERROR: app-text/calibre-0.9.29 failed (install phase): * doins failed * * If you need support, post the output of `emerge --info '=app-text/calibre-0.9.29'`, * the complete build log and the output of `emerge -pqv '=app-text/calibre-0.9.29'`. * The complete build log is located at '/var/tmp/portage/app-text/calibre-0.9.29/temp/build.log'. * The ebuild environment file is located at '/var/tmp/portage/app-text/calibre-0.9.29/temp/environment'. * Working directory: '/var/tmp/portage/app-text/calibre-0.9.29/work/calibre' * S: '/var/tmp/portage/app-text/calibre-0.9.29/work/calibre' /quote gentoo-mobile siefke # emerge --info =app-text/calibre-0.9.29 - http://nopaste.info/166126bafc.html gentoo-mobile siefke # emerge -pqv =app-text/calibre-0.9.29 [ebuild R ] app-text/calibre-0.9.29 USE=udisks Has someone advice what is the Problem. The missing QT lib is present, but why is mistake? Missing Doins i not understand on my gentoo give package not. Thanks for help. There's been a libicu update. Run revdep-rebuild, and then try updating calibre again. signature.asc Description: OpenPGP digital signature
Re: [gentoo-user] Recover on SSD
2013/5/6 Alan McKinnon alan.mckin...@gmail.com On 06/05/2013 20:36, Hinnerk van Bruinehsen wrote: On Mon, May 06, 2013 at 07:34:20PM +0200, Volker Armin Hemmann wrote: emm - no. Wear leveling does not need any spare blocks. A lot of drives do have spare blocks, but those are never the same size of the original size (at least not on drives you can buy for a sensible amount of money). More like 120+8 or 160+16 or 256+16. The spare blocks are used like on a hdd: some block goes bad, another one is mapped in. Since the sdd firmware does not know if something was deleted or not* - it does know shit about filesystems**, you can of course dd an image, if you want to. Just like on a hdd. *there are drives that do garbage collection without TRIM for fat and or ntfs.. so they seem to know a bit about filesystems. ** and this is why TRIM exists in the first place. To tell the drive: yes, this data is gone. You don't need to care about it anymore. The actual numbers were made up to make the point (maybe I should have stated that in my OP). According to [1] they are normally between 7% - 37%. Linux supports TRIM since Kernel 2.6.28. It's supported for several filesystems (Ext4, Btrfs, FAT, GFS2 and XFS) but must be enabled via the discard mount option. I don't have definitive information for Windows but it seems to be supported by at least Windows 7 (as far as I can tell without any user interaction). Since the deletion happened under Windows I made a guess that it is not totally unreasonable that dd may not work (if the deleted data would have been TRIMed). [1] http://www.lsi.com/downloads/Public/Flash%20Storage%20Processors/LSI_PRS_FMS2012_TE21_Smith.pdf A delete on an SSD is a very expensive operation, to my mind it seems completely unreasonable to think that Windows would try and clear many tens of GB just because it trashed a partition table. It would take _hours_ to clear those blocks. By far the easiest route would be to just do what is done for spinning disks - write the partition table, leave whatever junk is in the cells intact until the partition is formatted and actual data is written to the fs. As your results show, this is indeed what did happen. -- Alan McKinnon alan.mckin...@gmail.com Ok, let me sum up what I understood about the working of SSDs, please correct me if I'm wrong at some point. - The SSD stores what internal cell is allocated as a sector for the block device representation on the SATA port. - When a file is deleted the file system marks the block device sectors as free and sends the TRIM command to the SSD and the SSD really frees the underlying cell / breaks the cell - section allocation. - Some SSDs have idle TRIM as described in http://en.wikipedia.org/wiki/TRIM to use the advantage at systems that doesn't have the file system option - A write operation can write to sectors which are not TRIMed jet - When some program overwrites the partition table the sectors of the partition aren't touched, so the SSD must be aware of the partition table to trim these sectors - A new partition can be formatted without trimming the sectors - So when creating a new partition on the same sectors used before, the sector cell allocation in the SSD is still the same, and no data is lost, except the SSD is aware of the partition table to know which sectors can be TRIMed -- Mit freundlichen Grüßen / Best regards Randolph Maaßen
[gentoo-user] Re: Delays while building Libre Office.
On 05/05/2013 01:38 PM, Alan Mackenzie wrote: There doesn't appear to be any action in emerge which unpacks the build files (or even the entire source) of a package for perusal. This is a shame. You should become familiar with the 'ebuild' command, which I use very often when trying to debug a package, e.g. #ebuild /usr/portage/app-office/libreoffice/libreoffice-3.6.6.2.ebuild unpack To completely build and install a package you need to follow that command with the 'configure', 'compile', 'install', and 'qmerge' steps (man ebuild). This method allows you to resume an emerge without starting from zero. It saves a lot of pain when dealing with a package like libreoffice :)
Re: [gentoo-user] Re: Delays while building Libre Office.
Am 07.05.2013 01:22, schrieb walt: On 05/05/2013 01:38 PM, Alan Mackenzie wrote: There doesn't appear to be any action in emerge which unpacks the build files (or even the entire source) of a package for perusal. This is a shame. You should become familiar with the 'ebuild' command, which I use very often when trying to debug a package, e.g. #ebuild /usr/portage/app-office/libreoffice/libreoffice-3.6.6.2.ebuild unpack To completely build and install a package you need to follow that command with the 'configure', 'compile', 'install', and 'qmerge' steps (man ebuild). If you are lazy like my, and don't want to execute all steps manually, just use ebuild package.ebuild merge :)
[gentoo-user] Re: Delays while building Libre Office.
On 05/06/2013 04:37 PM, Michael Hampicke wrote: Am 07.05.2013 01:22, schrieb walt: On 05/05/2013 01:38 PM, Alan Mackenzie wrote: There doesn't appear to be any action in emerge which unpacks the build files (or even the entire source) of a package for perusal. This is a shame. You should become familiar with the 'ebuild' command, which I use very often when trying to debug a package, e.g. #ebuild /usr/portage/app-office/libreoffice/libreoffice-3.6.6.2.ebuild unpack To completely build and install a package you need to follow that command with the 'configure', 'compile', 'install', and 'qmerge' steps (man ebuild). If you are lazy like me, No way! I'm lazier than you :p and don't want to execute all steps manually, just use ebuild package.ebuild merge :) The reason it's nice to know the individual steps is that there can be bugs at any of those steps along the way (usually during the configure or compile phases) and if the bug is simple enough you can edit the broken files and then restart the merge. Often someone will add a patch to an existing bug report, which you can apply to a failed emerge and then restart the compile phase without starting from scratch. I'm guessing you know all of this already, but Alan and others may not know. The more debuggers, the merrier :)
