On 22/12/15 03:03, Ivan Pozdeev wrote:
>> On 06/12/15 18:32, Ivan Pozdeev wrote:
>>>> On 06/12/15 15:20, Ivan Pozdeev wrote:
>>>>> Hello Coreutils,
>>>>>
>>>>> The subj is a requirement if I'm copying between overlapping regions, and
>>>>> the
>>>>> destination is further in the media than the source.
>>>>>
>>>>> The specific task I'm having is to move a partition a number of sectors
>>>>> forward.
>>>>>
>>>>> To be completely clear, the algorithm is: copy a block, then seek to the
>>>>> previous block (i.e. 2*bs bytes back).
>>>>>
>>>>> I don't currently see a use case for setting this for input/output
>>>>> independently.
>>>>>
>>>>> An idea for the option name is `d'.
>>>>>
>>>>> There is a workaround for my particular case - `dd if=<block_device>
>>>>> bs=<2*shift_bytes> | dd
>>>>> of=<block_device> seek=<shift_blocks>' but it requires a few times
>>>>> 2*shift_bytes of memory.
>>>
>>>> I see dd_rescue (and ddrescue) have "operate in reverse" options.
>>>> The reason stated being:
>>>
>>>> "If you have one spot of bad sectors within the partition,
>>>> it might be a good idea, to approach this spot from both sides."
>>>
>>> I already looked into dd_rescue. It can only copy a full device,
>>> its purpose being data rescue rather than tinkering.
>>>
>>>> For your use case you could compute the size and offset of the overlap,
>>>> and use 2 dd invocations to copy the overlap area, then the rest.
>>>
>>> Not applicable. The overlap is >1/2, so its destination is also within the
>>> overlapped area.
>>>
>>>> Or perhaps more simply use a loop to iterate in reverse
>>>> using an appropriately large blocksize to minimize dd invocations.
>>>
>>> Looked into this, too. 78000000+ invocations in my case.
>>> I don't even want to think of how long this would take.
>>>
>>>> So while this is useful it is a bit of an edge case
>>>> and there are alternatives using other tools or multiple dd invocations.
>>>
>>> dd's sole job is to "take blocks from here, put them into there".
>>> Unless there's another tool whose sole job is to shuffle the input,
>>> the order of taking/putting seems to be an intrinsic trait of that job.
>>> Though I agree that it's the minority of cases where this trait matters.
>>>
>>> There's yet another argument:
>>> As you outlined, currently one has to resort to different methods for
>>> different
>>> data sizes/overlap percentages, which are far from being obvious (e.g. I
>>> came
>>> with my stated workaround only after having composed the letter). Basically,
>>> what I'm suggesting is the "one obvious way" ((c)Python Zen) - a method
>>> whose
>>> chief advantage is being immediately apparent (as well as being highly
>>> applicable and readable in code).
>>>
>>>> I'd be 60:40 against adding it.
>
>> Good arguments thanks. Worth considering.
>
>> Another option for this case where the read and write offsets are close,
>> would be to read into a circular buffer, and write while maintaining
>> a minimum number of blocks (the overlap) in the buffer.
>
>> We previously talked about adding such a util to coreutils:
>> http://lists.gnu.org/archive/html/bug-coreutils/2005-11/msg00190.html
>
>> A simple untested patch to debian's buffer(1) attached might achieve this
>> in a more optimum fashion by supporting read and write sizes > overlap size.
>> That would be used something like:
>
>> dd bs=... if=... | buffer -s 1m -P 50 | dd seek=... of=...
>
>> cheers,
>> Pádraig.
>
> My aforementioned workaround didn't work. The 2nd `dd' still overwrites
> some sectors before the 1st one reads them.
>
> The line with `buffer' just above (the System Rescue CD that I'm using has
> `mbuffer' instead) didn't work, either: -P delays writes initially but doesn't
> protect from buffer underruns.
>
> You didn't specify which specific source your patch is against so I couldn't
> test it.
> Yet, due to the previous paragraph, I suspect it won't help, either.
mkdir buffer
cd buffer
wget http://http.debian.net/debian/pool/main/b/buffer/buffer_1.19.orig.tar.gz
wget
http://http.debian.net/debian/pool/main/b/buffer/buffer_1.19-12.debian.tar.xz
tar -xf buffer_1.19.orig.tar.gz
tar -xf buffer_1.19-12.debian.tar.xz
(cd buffer-1.19 && for p in $(cat ../debian/patches/series); do patch -p1 <
../debian/patches/$p; done)
cd buffer-1.19/
patch -p1 < ../buffer-min_percent.diff
make
./buffer ...
Note I've attached the diff again to this email
as the previous one erroneously included ansi coloring sequences.
> Even `parted' and `gparted' are no good! The first one doesn't support
> moving, the
> 2nd one has 1M granilarity and doesn't show/allow to set actual sector
> numbers.
thanks for testing all the options,
Pádraig.
--- buffer-1.19/buffer.c 2015-12-06 22:36:43.364143317 +0000
+++ /tmp/buffer-new.c 2015-12-06 22:36:26.001894230 +0000
@@ -255,6 +255,7 @@
int reader_pid = 0;
int free_shm = 1;
int percent = 0;
+int min_percent = 0;
int debug = 0;
int Zflag = 0;
int writer_status = 0;
@@ -385,6 +386,16 @@
if( debug )
fprintf( stderr, "percent set to %d\n", percent );
break;
+ case 'P': /* min percent to maintain in the buffer */
+ min_percent = atoi( optarg );
+
+ if( (min_percent < 0) || (100 < min_percent) ){
+ fprintf( stderr, "min_percent %d out of range\n", min_percent );
+ byee( -1 );
+ }
+ if( debug )
+ fprintf( stderr, "min_percent set to %d\n", min_percent );
+ break;
case 'z':
zflag++;
/* FALL THRU */
@@ -397,7 +408,7 @@
}
break;
default:
- fprintf( stderr, "Usage: %s [-B] [-t] [-S size] [-m memsize] [-b blocks] [-p percent] [-s blocksize] [-u pause] [-i infile] [-o outfile] [-z size] [-Z] [-d]\n",
+ fprintf( stderr, "Usage: %s [-B] [-t] [-S size] [-m memsize] [-b blocks] [-p percent] [-P min_percent] [-s blocksize] [-u pause] [-i infile] [-o outfile] [-z size] [-Z] [-d]\n",
progname );
fprintf( stderr, "-B = blocked device - pad out last block\n" );
fprintf( stderr, "-t = show total amount written at end\n" );
@@ -405,6 +416,7 @@
fprintf( stderr, "-m size = size of shared mem chunk to grab\n" );
fprintf( stderr, "-b num = number of blocks in queue\n" );
fprintf( stderr, "-p percent = don't start writing until percent blocks filled\n" );
+ fprintf( stderr, "-P min_percent = min number of blocks to keep in the buffer\n" );
fprintf( stderr, "-s size = size of a block\n" );
fprintf( stderr, "-u usecs = microseconds to sleep after each write\n" );
fprintf( stderr, "-i infile = file to read from\n" );
@@ -421,6 +433,11 @@
byee( -1 );
}
+ if (min_percent > percent) {
+ fprintf( stderr, "min_percent > percent, aborting!\n" );
+ byee( -1 );
+ }
+
if (zflag) showevery = blocksize;
/* If -b was not given try and work out the max buffer size */
@@ -748,12 +765,14 @@
{
int filled = 0;
int maxfilled = (blocks * percent) / 100;
+ int minfilled = (blocks * min_percent) / 100;
int first_block = 0;
if( debug )
- fprintf( stderr, "\tW: Entering writer\n blocks = %d\n maxfilled = %d\n",
+ fprintf( stderr, "\tW: Entering writer\n blocks = %d\n"
+ " maxfilled = %d\n minfilled = %d\n",
blocks,
- maxfilled );
+ maxfilled, minfilled );
while( 1 ){
if( !filled )
@@ -765,10 +784,10 @@
filled++;
if( debug > 1 )
fprintf( stderr, "W: filled = %d\n", filled );
- if( filled >= maxfilled ){
+ if( filled >= maxfilled && filled > minfilled ){
if( debug > 1 )
fprintf( stderr, "W: writing\n" );
- write_blocks_to_stdout( filled, first_block );
+ write_blocks_to_stdout( filled - minfilled, first_block );
filled = 0;
}
}
