compiling static binaries

[Niv Waizer via rsync]

For anybody trying to build a static binary:

Thanks to Han Boetes for the post here, advising to use musl.
Thanks to Florian Dejonckheere for posting this script:
https://github.com/JBBgameich/rsync-static

Here is my small contrib on this, for the lazy folks.
I download the musl prebuild binaries for my arch from
skarnet.org/toolchains
My script is posted here:
https://github.com/nwaizer/rsync-static/blob/main/build_static.sh

or below:
#!/bin/bash
# based on the solution by Florian Dejonckheere <
flor...@floriandejonckheere.be>
# at https://github.com/JBBgameich/rsync-static

mkdir toolchain
echo "I: Downloading prebuilt toolchain"
wget --continue
https://skarnet.org/toolchains/native/x86_64-linux-musl_pc-11.3.0.tar.xz -O
/tmp/x86_64-linux-musl_pc.tar.xz|| echo "Failed to find
x86_64-linux-musl_pc-11.3.0.tar.xz. Please open your browser at
https://skarnet.org/toolchains/native and find the correct file to fix this"
tar -xf /tmp/x86_64-linux-musl_pc.tar.xz -C toolchain

echo "Native compiler is called gcc"
TOOLCHAIN_PATH="$(readlink -f $(dirname $(find . -name "gcc"))/..)"
echo "Use local gcc instead of the OS installed one"
export PATH=$TOOLCHAIN_PATH/bin:$PATH

echo "Get rsync source"
git clone https://github.com/WayneD/rsync.git

echo "Build rsync"
cd rsync/
make clean
export CC="gcc"
echo "Disable openssl xxhash zstd and lz4 as they did not configure even
after consulting the INSTALL"
./configure CFLAGS="-static" --host="x86" --disable-openssl
--disable-xxhash --disable-zstd --disable-lz4
make
strip rsync

Thanks for rsync,
Niv
-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Merging three slightly different directories

[hput Putnam via rsync]

Robin Wrote:

> So I'm not quite following your Windows rsync problem. 

Specifically, do you mean: 

 

>"I copied files A, B, and C, and there's still 3 files but each file 

is bigger" 

 

> or 

 

> "I copied files A, B, and C to a directory containing files that I 

> thought were identical using the -b flag but now there's twice as 



I'm astounded you seem to think I used the -b flag but you must know that it 
was you who introduced the topic of -b flag.  That suggestion came after all 
rsyncing was done.

I expect the -b flag will play a big role in cleaning things up.



> many files because apparently rsync thinks none of them are 

> identical" 

 

Ok, sorry to be unclear... but really I'm not writing in ancient Sanscrit here.



There was no copying done at all.  Everything was and will be done with rsync.  
I still have not tried the -b option but expect that to be very helpful.



I explained how I ran rsync from an Images/ directory on one linux host 
directly into an Images directory on another linux host.



rsync -vvrptgoD --stats LINHOST1:/Images/  LINHOST2:/Images/



I thought the directories were too large to examine them much by hand but I 
knew how they were built and knew they were at least very similar.

both structurally and content wise.  So that first merge went about how I 
expected there was a small overall size increase of a few hundred MB.



Then I ran "rsync -vvrptgoD winHOST:/Images/   LINHOST2:/Images/"  so from the 
win host to the already once merged LINHOST2:/Images. Both directories showed 
quite similar size according to "du -sh" before rsyncing.



However when the rsyncing was through the LINHOST2:/Images directory had grown 
considerably. I still have not examined by hand but clearly something 
unexpected has happened.  I expect I'll find some significant differences I'd 
forgotten about.



Until I stop and examine things closely; any further discussion will be kind of 
lame.  I was putting off a pains taking examination but I see it must be done.



These thousands of images have built up from 2008 to late 2021 and many Phone 
camera images have been added since then.  So I need to be careful with them.  
With that in mind I have kept backups of all three directories in their 
original state.



Thank you much for your input so far.  I expect to be a good little while 
examining things.  In fact I see now this will be a major undertaking.-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Merging three slightly different directories

[Robin Lee Powell via rsync]

It would help if you gave us an example of what you'd *want* to have
happen in different situations, but what about the -b option?  This
will do nothing with identical files but keep both versions of
non-identical ones.

On Wed, Jun 08, 2022 at 12:24:16AM +, hput via rsync wrote:
> I want to merge 3 slightly different directories of mostly images.
> 
> Not just mostly but the vast majority are images files.
> 
> Each directory has about 285 GB of files.
> 
> At first I thought I would just run a straightish rsync from each directory
> inturn starting with the biggest which is not much bigger ... maybe
> a few MB.
> 
> Like:
> 
> rsync -vvrptgoD --stats /biggest/ /emptydir/
> 
> rsync -vvrptgoD --stats /next-biggest/ /same-dir/
> 
> rsync -vvrptgoD --stats /smallest/ /same-dir
> 
> But after some thought I'm guessing that might be wrong headed way to go.
> 
> All three dir have mostly the same stuff in them and in the same
> places but a close inspection, given the 285 GB would be pretty much a
> non-starter.
> 
> There will be thousands that have matching names maybe newer or older
> bigger etc.  And maybe some of the same stuff but in slightly  different 
> places.
> 
> How can I make rsync do the work for me?  So I don't end up loosing files.
> 
> 
> 
> -- 
> Please use reply-all for most replies to avoid omitting the mailing list.
> To unsubscribe or change options: 
> https://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Merging three slightly different directories

[Robin Lee Powell via rsync]

So I'm not quite following your Windows rsync problem.
Specifically, do you mean:

"I copied files A, B, and C, and there's still 3 files but each file
is bigger"

or

"I copied files A, B, and C to a directory containing files that I
thought were identical using the -b flag but now there's twice as
many files because apparently rsync thinks none of them are
identical"

?

In the former case: Windows and Linux store files on disk
differently; I would expect files to be slightly larger on Windows,
but like 10%, not double.

In the latter case: By default, rsync checks for last-mod-time and size and if 
those
are identical, it treats the file as identical.  This will almost
never be the case across Linux and windows because they store files
differently on disk, so rsync is going to think all the files are
different, so if you're using -b it'll take backups of everything.

Another things to consider: if a directory is named differently on
Windows vs. Linux (i.e. different case on the directory name), rsync
will not merge those directories.

I suggest walking the directory tree to see where the size
differences are, exactly; in Windows I use https://windirstat.net/ ,
there are various equivalents on Linux (including just using du).

On Thu, Jun 09, 2022 at 09:42:27AM -0400, hputn3 wrote:
>  On Thu, 09 Jun 2022 02:20:02 -0400 Robin Lee Powell 
>  wrote 
> 
> 
> 
> > It would help if you gave us an example of what you'd *want* to have 
> 
> > happen in different situations, but what about the -b option?  This 
> 
> > will do nothing with identical files but keep both versions of 
> 
> > non-identical ones. 
> 
> 
> 
> 
> Sorry It seemed obvious to me.  Myopia is a sometime hazard for me.
> 
> 
> 
> 
> 
> I hoped by merging the slightly different directories that have grown up 
> partially independent but largely the same.
> 
> 
> 
> I wanted to pick up what ever changes has occurred in the three different 
> directories over time, into one directory that has everything.
> 
> 
> 
> 
> 
> 
> I have actually done what I was proposing and in the end the size of the 
> Merged Images directory has doubled.
> 
> 
> 
> My original scheme:
> 
> Seemed to work as expected between two Linux HOSTS:  There was a size 
> increase of about 2-300MB
> 
> 
> 
> But something screwy happens when rsyncing from windows to linux or unix.  I 
> doubled the size and I'm certain any actual changes would have been fairly 
> small. However, I suppose if I had decided to move a major internal directory 
> inside Images or rename one or something... that might cause a serious 
> problem for rsync to surmount.
> 
> 
> 
> On Wed, Jun 08, 2022 at 12:24:16AM +, hput via rsync wrote: 
> 
> 
> 
> 
> 
> 
> 
> [...]
> 
> > How can I make rsync do the work for me?  So I don't end up loosing files. 
> 
>  > 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Robin wrote
> 
>  > but what about the -b option?  This 
> 
> > will do nothing with identical files but keep both versions of 
> 
> > non-identical ones.
> 
> 
> 
> After pulling out the man pages and looking at that,  It might be just the 
> thing.
> 
> I still have all original unmolested directories so I can try that and see 
> where it goes.
> 
> 
> 
> But what about the thing with windows...causing a doubling of the size of 
> Images?
> 
> It made me remember having rynced stuff from windows before, long ago, and 
> seeing some kind of nasty behavior
> 
> I might have to hand merge that directory... and egad at 285G that sounds a 
> bit daunting.
> 
> 
> 
> Thank you Robin for your thoughtful and helpful reply

-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html

Re: Merging three slightly different directories

[hputn3 via rsync]

 On Thu, 09 Jun 2022 02:20:02 -0400 Robin Lee Powell 
 wrote 



> It would help if you gave us an example of what you'd *want* to have 

> happen in different situations, but what about the -b option?  This 

> will do nothing with identical files but keep both versions of 

> non-identical ones. 




Sorry It seemed obvious to me.  Myopia is a sometime hazard for me.





I hoped by merging the slightly different directories that have grown up 
partially independent but largely the same.



I wanted to pick up what ever changes has occurred in the three different 
directories over time, into one directory that has everything.






I have actually done what I was proposing and in the end the size of the Merged 
Images directory has doubled.



My original scheme:

Seemed to work as expected between two Linux HOSTS:  There was a size increase 
of about 2-300MB



But something screwy happens when rsyncing from windows to linux or unix.  I 
doubled the size and I'm certain any actual changes would have been fairly 
small. However, I suppose if I had decided to move a major internal directory 
inside Images or rename one or something... that might cause a serious problem 
for rsync to surmount.



On Wed, Jun 08, 2022 at 12:24:16AM +, hput via rsync wrote: 







[...]

> How can I make rsync do the work for me?  So I don't end up loosing files. 

 > 









Robin wrote

 > but what about the -b option?  This 

> will do nothing with identical files but keep both versions of 

> non-identical ones.



After pulling out the man pages and looking at that,  It might be just the 
thing.

I still have all original unmolested directories so I can try that and see 
where it goes.



But what about the thing with windows...causing a doubling of the size of 
Images?

It made me remember having rynced stuff from windows before, long ago, and 
seeing some kind of nasty behavior

I might have to hand merge that directory... and egad at 285G that sounds a bit 
daunting.



Thank you Robin for your thoughtful and helpful reply-- 
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html