Re: Suggestion: bmap files and bmaptool

2013-09-17 Thread Artem Bityutskiy
On Wed, 2013-08-14 at 11:44 +0200, Till Maas wrote: On Wed, Aug 14, 2013 at 12:21:23PM +0300, Artem Bityutskiy wrote: On Wed, 2013-08-14 at 10:37 +0200, Till Maas wrote: On Wed, Aug 14, 2013 at 09:31:22AM +0300, Artem Bityutskiy wrote: Other things like reading from remote sites,

Re: Suggestion: bmap files and bmaptool

2013-09-17 Thread Artem Bityutskiy
On Wed, 2013-08-14 at 12:24 +0200, Björn Persson wrote: Artem Bityutskiy wrote: On Wed, 2013-08-14 at 11:44 +0200, Till Maas wrote: On Wed, Aug 14, 2013 at 12:21:23PM +0300, Artem Bityutskiy wrote: On Wed, 2013-08-14 at 10:37 +0200, Till Maas wrote: On Wed, Aug 14, 2013 at 09:31:22AM

Re: Suggestion: bmap files and bmaptool

2013-09-17 Thread Artem Bityutskiy
On Sun, 2013-08-18 at 16:43 +0100, Pádraig Brady wrote: You definitely need the fsync before doing the fiemap. We saw this on certain file systems including ext4 when adding fiemap support (efficient reading of holes) to cp. This is a bug in the fiemap interface IMHO in that it returns fairly

Re: Suggestion: bmap files and bmaptool

2013-08-19 Thread Artem Bityutskiy
On Fri, 2013-08-16 at 09:51 -0500, Eric Sandeen wrote: by single file I meant a _single_ file, not the original file and a mapping file. :) Oh, sorry, OK. I realize that you have a fully-fledged set of tools, and you're not looking for new directions, but I was thinking about encoding

Re: Suggestion: bmap files and bmaptool

2013-08-19 Thread Artem Bityutskiy
On Sat, 2013-08-17 at 17:09 +0100, Richard W.M. Jones wrote: On Thu, Aug 15, 2013 at 12:34:26PM -0500, Eric Sandeen wrote: But then there's the issue of transporting these sparse files around. We have had the same problem in the past with large e2image metadata image files, which may be

Re: Suggestion: bmap files and bmaptool

2013-08-19 Thread Artem Bityutskiy
On Sun, 2013-08-18 at 16:43 +0100, Pádraig Brady wrote: You definitely need the fsync before doing the fiemap. We saw this on certain file systems including ext4 when adding fiemap support (efficient reading of holes) to cp. This is a bug in the fiemap interface IMHO in that it returns fairly

Re: Suggestion: bmap files and bmaptool

2013-08-18 Thread Pádraig Brady
On 08/16/2013 09:46 AM, Artem Bityutskiy wrote: Hi Eric, thanks for the question. Sorry my answers contain extra information, but I assume that other people read this, and may benefit from the info. After all, I am trying to get people interested, this is a good tool IMO, and I would like

Re: Suggestion: bmap files and bmaptool

2013-08-17 Thread Richard W.M. Jones
On Thu, Aug 15, 2013 at 12:34:26PM -0500, Eric Sandeen wrote: But then there's the issue of transporting these sparse files around. We have had the same problem in the past with large e2image metadata image files, which may be terabytes in length, with only gigabytes or megabytes of real

Re: Suggestion: bmap files and bmaptool

2013-08-16 Thread Artem Bityutskiy
Hi Eric, thanks for the question. Sorry my answers contain extra information, but I assume that other people read this, and may benefit from the info. After all, I am trying to get people interested, this is a good tool IMO, and I would like to get more users and hopefully contributors. :-) On

Re: Suggestion: bmap files and bmaptool

2013-08-16 Thread Eric Sandeen
On 8/16/13 3:46 AM, Artem Bityutskiy wrote: Another approach which might (?) be more robust, is to somehow encode that sparseness in a single file format that can be transported/compressed/copied w/o losing the sparseness information, and another tool to operate efficiently on that format

Re: Suggestion: bmap files and bmaptool

2013-08-15 Thread Eric Sandeen
On 8/13/13 8:58 AM, Artem Bityutskiy wrote: # Make the image to be sparse $ cp --sparse=always Fedora-x86_64-19-20130627-sda.raw Fedora-x86_64-19-20130627-sda.raw.sparse # Generate the bmap file $ bmaptool create Fedora-x86_64-19-20130627-sda.raw.sparse -o

Re: Suggestion: bmap files and bmaptool

2013-08-14 Thread Artem Bityutskiy
On Tue, 2013-08-13 at 17:48 +0200, Jochen Schmitt wrote: On Tue, Aug 13, 2013 at 04:58:16PM +0300, Artem Bityutskiy wrote: Hi Fedora developers, $ dd if=Fedora-x86_64-19-20130627-sda.raw of=/dev/sdd 4194304+0 records in 4194304+0 records out 2147483648 bytes (2.1 GB) copied, 799.487

Re: Suggestion: bmap files and bmaptool

2013-08-14 Thread Artem Bityutskiy
On Wed, 2013-08-14 at 09:31 +0300, Artem Bityutskiy wrote: But this all is not the point, these are additional goodies. The main point of bmaptool is in having the bmap file, and _then_ you get real speed-up, based on the sparseness of the original image. Just as a rough demonstration, as I

Re: Suggestion: bmap files and bmaptool

2013-08-14 Thread Christopher Meng
在 2013-8-13 PM9:52,Artem Bityutskiy dedeki...@gmail.com写道: Hi Fedora developers, I would like suggest you to take a look at bmaptool, which you can use for flashing Fedora ISO images to USB sticks (or other block devices). I've read an article about this tool in Chinese months ago. I can

Re: Suggestion: bmap files and bmaptool

2013-08-14 Thread Till Maas
On Wed, Aug 14, 2013 at 09:31:22AM +0300, Artem Bityutskiy wrote: Other things like reading from remote sites, progress indicator, protecting your mounted disks, uncompressing on-the-fly, checking sha1 of the data ond of the bmap file itself - are goodies, although important ones. Why sha1?

Re: Suggestion: bmap files and bmaptool

2013-08-14 Thread Artem Bityutskiy
On Wed, 2013-08-14 at 16:35 +0800, Christopher Meng wrote: 在 2013-8-13 PM9:52,Artem Bityutskiy dedeki...@gmail.com写道: Hi Fedora developers, I would like suggest you to take a look at bmaptool, which you can use for flashing Fedora ISO images to USB sticks (or other block devices).

Re: Suggestion: bmap files and bmaptool

2013-08-14 Thread Till Maas
On Wed, Aug 14, 2013 at 12:21:23PM +0300, Artem Bityutskiy wrote: On Wed, 2013-08-14 at 10:37 +0200, Till Maas wrote: On Wed, Aug 14, 2013 at 09:31:22AM +0300, Artem Bityutskiy wrote: Other things like reading from remote sites, progress indicator, protecting your mounted disks,

Re: Suggestion: bmap files and bmaptool

2013-08-14 Thread Artem Bityutskiy
On Wed, 2013-08-14 at 11:44 +0200, Till Maas wrote: On Wed, Aug 14, 2013 at 12:21:23PM +0300, Artem Bityutskiy wrote: On Wed, 2013-08-14 at 10:37 +0200, Till Maas wrote: On Wed, Aug 14, 2013 at 09:31:22AM +0300, Artem Bityutskiy wrote: Other things like reading from remote sites,

Re: Suggestion: bmap files and bmaptool

2013-08-14 Thread Björn Persson
Artem Bityutskiy wrote: On Wed, 2013-08-14 at 11:44 +0200, Till Maas wrote: On Wed, Aug 14, 2013 at 12:21:23PM +0300, Artem Bityutskiy wrote: On Wed, 2013-08-14 at 10:37 +0200, Till Maas wrote: On Wed, Aug 14, 2013 at 09:31:22AM +0300, Artem Bityutskiy wrote: Other things like

Re: Suggestion: bmap files and bmaptool

2013-08-14 Thread Artem Bityutskiy
On Wed, 2013-08-14 at 12:24 +0200, Björn Persson wrote: Speaking of security, how is the integrity of the bmap file itself verified? This is not implemented, unfortunately. This is another thing which I probably would need to do, and this is a very good point. I will look at this, after I do

Re: Suggestion: bmap files and bmaptool

2013-08-14 Thread Samuel Sieb
On 08/14/2013 02:21 AM, Artem Bityutskiy wrote: I think I covered this part in the documentation. But here is a short description. 1. The bmap file should be created just after the image is generated. 2. The blocks where zeroes were explicitly written will be mapped to real sectors which will

Re: Suggestion: bmap files and bmaptool

2013-08-14 Thread Artem Bityutskiy
On Wed, 2013-08-14 at 21:16 -0700, Samuel Sieb wrote: On 08/14/2013 02:21 AM, Artem Bityutskiy wrote: I think I covered this part in the documentation. But here is a short description. 1. The bmap file should be created just after the image is generated. 2. The blocks where zeroes were

Re: Suggestion: bmap files and bmaptool

2013-08-13 Thread Jochen Schmitt
On Tue, Aug 13, 2013 at 04:58:16PM +0300, Artem Bityutskiy wrote: Hi Fedora developers, $ dd if=Fedora-x86_64-19-20130627-sda.raw of=/dev/sdd 4194304+0 records in 4194304+0 records out 2147483648 bytes (2.1 GB) copied, 799.487 s, 2.7 MB/s 1.) Seems to be a slow USB flash drive. 2.) What