Re: dd: copy blocks in reverse order

[Pádraig Brady]

On 06/12/15 18:32, Ivan Pozdeev wrote:
>> On 06/12/15 15:20, Ivan Pozdeev wrote:
>>> Hello Coreutils,
>>>
>>> The subj is a requirement if I'm copying between overlapping regions, and 
>>> the
>>> destination is further in the media than the source.
>>>
>>> The specific task I'm having is to move a partition a number of sectors 
>>> forward.
>>>
>>> To be completely clear, the algorithm is: copy a block, then seek to the
>>> previous block (i.e. 2*bs bytes back).
>>>
>>> I don't currently see a use case for setting this for input/output 
>>> independently.
>>>
>>> An idea for the option name is `d'.
>>>
>>> There is a workaround for my particular case - `dd if=<block_device> 
>>> bs=<2*shift_bytes> | dd
>>> of=<block_device> seek=<shift_blocks>' but it requires a few times 
>>> 2*shift_bytes of memory.
> 
>> I see dd_rescue (and ddrescue) have "operate in reverse" options.
>> The reason stated being:
> 
>> "If you have one spot of bad sectors within the partition,
>> it might be a good idea, to approach this spot from both sides."
> 
> I already looked into dd_rescue. It can only copy a full device,
> its purpose being data rescue rather than tinkering.
> 
>> For your use case you could compute the size and offset of the overlap,
>> and use 2 dd invocations to copy the overlap area, then the rest.
> 
> Not applicable. The overlap is >1/2, so its destination is also within the
> overlapped area.
> 
>> Or perhaps more simply use a loop to iterate in reverse
>> using an appropriately large blocksize to minimize dd invocations.
> 
> Looked into this, too. 78000000+ invocations in my case.
> I don't even want to think of how long this would take.
> 
>> So while this is useful it is a bit of an edge case
>> and there are alternatives using other tools or multiple dd invocations.
> 
> dd's sole job is to "take blocks from here, put them into there".
> Unless there's another tool whose sole job is to shuffle the input,
> the order of taking/putting seems to be an intrinsic trait of that job.
> Though I agree that it's the minority of cases where this trait matters.
> 
> There's yet another argument:
> As you outlined, currently one has to resort to different methods for 
> different
> data sizes/overlap percentages, which are far from being obvious (e.g. I came
> with my stated workaround only after having composed the letter). Basically,
> what I'm suggesting is the "one obvious way" ((c)Python Zen) - a method whose
> chief advantage is being immediately apparent (as well as being highly
> applicable and readable in code).
> 
>> I'd be 60:40 against adding it.

Good arguments thanks.  Worth considering.

Another option for this case where the read and write offsets are close,
would be to read into a circular buffer, and write while maintaining
a minimum number of blocks (the overlap) in the buffer.

We previously talked about adding such a util to coreutils:
http://lists.gnu.org/archive/html/bug-coreutils/2005-11/msg00190.html

A simple untested patch to debian's buffer(1) attached might achieve this
in a more optimum fashion by supporting read and write sizes > overlap size.
That would be used something like:

dd bs=... if=... | buffer -s 1m -P 50 | dd seek=... of=...

cheers,
Pádraig.

--- buffer-1.19/buffer.c	2015-12-06 22:36:43.364143317 +0000(B
+++ /tmp/buffer-new.c	2015-12-06 22:36:26.001894230 +0000(B
@@ -255,6 +255,7 @@(B
 int reader_pid	= 0;
 int free_shm	= 1;
 int percent	= 0;
+int min_percent	= 0;(B
 int debug	= 0;
 int Zflag	= 0;
 int writer_status = 0;
@@ -385,6 +386,16 @@(B
 			if( debug )
 				fprintf( stderr, "percent set to %d\n", percent );
 			break;
+		case 'P':	/* min percent to maintain in the buffer */(B
+			min_percent = atoi( optarg );(B
+(B
+			if( (min_percent < 0) || (100 < min_percent) ){(B
+				fprintf( stderr, "min_percent %d out of range\n", min_percent );(B
+				byee( -1 );(B
+			}(B
+			if( debug )(B
+				fprintf( stderr, "min_percent set to %d\n", min_percent );(B
+			break;(B
 		case 'z':
 			zflag++;
 			/* FALL THRU */
@@ -397,7 +408,7 @@(B
 			}
 			break;
 		default:
-			fprintf( stderr, "Usage: %s [-B] [-t] [-S size] [-m memsize] [-b blocks] [-p percent] [-s blocksize] [-u pause] [-i infile] [-o outfile] [-z size] [-Z] [-d]\n",(B
+			fprintf( stderr, "Usage: %s [-B] [-t] [-S size] [-m memsize] [-b blocks] [-p percent] [-P min_percent] [-s blocksize] [-u pause] [-i infile] [-o outfile] [-z size] [-Z] [-d]\n",(B
 				progname );
 			fprintf( stderr, "-B = blocked device - pad out last block\n" );
 			fprintf( stderr, "-t = show total amount written at end\n" );
@@ -405,6 +416,7 @@(B
 			fprintf( stderr, "-m size = size of shared mem chunk to grab\n" );
 			fprintf( stderr, "-b num = number of blocks in queue\n" );
 			fprintf( stderr, "-p percent = don't start writing until percent blocks filled\n" );
+			fprintf( stderr, "-P min_percent = min number of blocks to keep in the buffer\n" );(B
 			fprintf( stderr, "-s size = size of a block\n" );
 			fprintf( stderr, "-u usecs = microseconds to sleep after each write\n" );
 			fprintf( stderr, "-i infile = file to read from\n" );
@@ -421,6 +433,11 @@(B
 		byee( -1 );
 	}
 
+        if (min_percent > percent) {(B
+		fprintf( stderr, "min_percent > percent, aborting!\n" );(B
+		byee( -1 );(B
+	}(B
+(B
 	if (zflag) showevery = blocksize;
 
 	/* If -b was not given try and work out the max buffer size */
@@ -748,12 +765,14 @@(B
 {
 	int filled = 0;
 	int maxfilled = (blocks * percent) / 100;
+	int minfilled = (blocks * min_percent) / 100;(B
 	int first_block = 0;
 
 	if( debug )
-		fprintf( stderr, "\tW: Entering writer\n blocks = %d\n maxfilled = %d\n",(B
+		fprintf( stderr, "\tW: Entering writer\n blocks = %d\n"(B
+                                 " maxfilled = %d\n minfilled = %d\n",(B
 			blocks,
-			maxfilled );(B
+			maxfilled, minfilled );(B
 
 	while( 1 ){
 		if( !filled )
@@ -765,10 +784,10 @@(B
 		filled++;
 		if( debug > 1 )
 			fprintf( stderr, "W: filled = %d\n", filled );
-		if( filled >= maxfilled ){(B
+		if( filled >= maxfilled && filled > minfilled ){(B
 			if( debug > 1 )
 				fprintf( stderr, "W: writing\n" );
-			write_blocks_to_stdout( filled, first_block );(B
+			write_blocks_to_stdout( filled - minfilled, first_block );(B
 			filled = 0;
 		}
 	}