Re: what method do you prefer for data transfer between nodes?

[Vincent Lefevre]

On 2023-03-05 16:22:25 +0800, Ken Young wrote:
> 1. scp
> pros: the native tool in the OS
> cons: you will either input password or put key pairs into servers for
> authentication.

I would not see this as a cons.

> 2. rsync
> pros: it can transfer data by increasement
> cons: you need to setup rsyncd server and make the correct authorization.

When I use rsync, I use it always over SSH (for security), and
no need for rsyncd in this case.

> 3. ftp/ftps
> pros: easy to use
> cons: need to setup ftpd server, and the way is not that secure?

I use sftp to transfer files between my Debian laptop and my
Android phones. This is secure, as this is a protocol over SSH.

I also use Subversion (over SSH) for files I want to keep.

And unison for my mail (stored in Maildir format).

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: what method do you prefer for data transfer between nodes?

[Ken Young]

For standalone backup I am using Liteserver's VPS, 512G disk for only 2.4
EUR/m.
Then I setup rsyncd on it, and rsync data from clients via crontab for
backup purpose.


Sincerely,
Ken Young


On Tue, Mar 7, 2023 at 9:21 AM Celejar  wrote:

> On Sun, 5 Mar 2023 14:05:37 +0100
> to...@tuxteam.de wrote:
>
> > On Sun, Mar 05, 2023 at 07:23:49PM +0800, Ken Young wrote:
> > > What provider do you recommend then?
> >
> > Around here (West Europe), for example, mailbox.org [1]. But they
> > are far from the only one (I'm not a customer, nor associated with
> > them, but I do know a few happy customers).
>
> €3 / month for 5 GB, €9 / month for 50 GB - that's rather overpriced
> for cloud storage:
>
> https://mailbox.org/en/services#tariffs
>
> > Whenever they don't cost any money you'll have to ask yourself
> > what their business model is.
>
> The big cloud providers (e.g. Backblaze) - whose business model does
> indeed involve charging money - charge as little as $5 / month per
> *terabyte* (plus egress), in addition to giving a few or few dozen GB
> free:
>
> https://www.backblaze.com/b2/cloud-storage-pricing.html
>
> I'm personally using Scaleway with rclone, which offers 75 GB free (but
> €12 / month per terabyte beyond that).
>
> Other prominent vendors (besides the obvious ones) include Wasabi and
> rsync.net. Pricing is tricky, as they break the fees into different
> categories (hot and cold storage, egress, etc.)
>
> --
> Celejar
>
>

Re: what method do you prefer for data transfer between nodes?

[Celejar]

On Sun, 5 Mar 2023 14:05:37 +0100
to...@tuxteam.de wrote:

> On Sun, Mar 05, 2023 at 07:23:49PM +0800, Ken Young wrote:
> > What provider do you recommend then?
> 
> Around here (West Europe), for example, mailbox.org [1]. But they
> are far from the only one (I'm not a customer, nor associated with
> them, but I do know a few happy customers).

€3 / month for 5 GB, €9 / month for 50 GB - that's rather overpriced
for cloud storage:

https://mailbox.org/en/services#tariffs

> Whenever they don't cost any money you'll have to ask yourself
> what their business model is.

The big cloud providers (e.g. Backblaze) - whose business model does
indeed involve charging money - charge as little as $5 / month per
*terabyte* (plus egress), in addition to giving a few or few dozen GB
free:

https://www.backblaze.com/b2/cloud-storage-pricing.html

I'm personally using Scaleway with rclone, which offers 75 GB free (but
€12 / month per terabyte beyond that).

Other prominent vendors (besides the obvious ones) include Wasabi and
rsync.net. Pricing is tricky, as they break the fees into different
categories (hot and cold storage, egress, etc.)

-- 
Celejar

Re: what method do you prefer for data transfer between nodes?

[tomas]

On Mon, Mar 06, 2023 at 06:41:58PM +0700, Max Nikulin wrote:
> On 05/03/2023 15:22, Ken Young wrote:
> > 
> > 1. scp
> 
> Notice that in bookworm (next release) scp uses SFTP protocol. This
> ssh-based protocol can be used in the current stable release as well. Do not
> confuse it with ftps that is traditional FTP over TLS.

"Classical" scp is going away. It allowed one to play nasty tricks no
one really wants played. For background, see [1].

The "easy cases" many people mentioned in this thread probably won't
notice the fake scp coming now :-)

Cheers

[1] https://lwn.net/Articles/835962/
-- 
t


signature.asc
Description: PGP signature

Re: what method do you prefer for data transfer between nodes?

[Max Nikulin]

On 05/03/2023 15:22, Ken Young wrote:


1. scp


Notice that in bookworm (next release) scp uses SFTP protocol. This 
ssh-based protocol can be used in the current stable release as well. Do 
not confuse it with ftps that is traditional FTP over TLS.

Re: what method do you prefer for data transfer between nodes?

[David Christensen]

On 3/5/23 00:22, Ken Young wrote:

Hello,

The methods I know,

1. scp
pros: the native tool in the OS
cons: you will either input password or put key pairs into servers for
authentication.

2. rsync
pros: it can transfer data by increasement
cons: you need to setup rsyncd server and make the correct authorization.

3. ftp/ftps
pros: easy to use
cons: need to setup ftpd server, and the way is not that secure?

4. rclone
pros:easy to use
cons: hard to setup (you may need a cloud storage for middleware).

For me I most often use scp + rsync. and what's your choice?

Regards,
Ken



Please tell us about your data, your nodes, and the mechanism(s) 
available for transferring data.



David

Re: what method do you prefer for data transfer between nodes?

[Charles Curley]

On Sun, 5 Mar 2023 16:22:25 +0800
Ken Young  wrote:

> Hello,
> 
> The methods I know,
> 
> 1. scp
> pros: the native tool in the OS
> cons: you will either input password or put key pairs into servers for
> authentication.

Not so much of a con. I now automatically add my public keys to a
machine as part of the bring-up process after installation.

> 
> 2. rsync
> pros: it can transfer data by increasement
> cons: you need to setup rsyncd server and make the correct
> authorization.

Nope. rsync over ssh handles that.

> 
> 3. ftp/ftps
> pros: easy to use
> cons: need to setup ftpd server, and the way is not that secure?

I use ftp only for one server where I don't have rsync over ssh
service. One of several reasons I want to abandon that server.

> 
> 4. rclone
> pros:easy to use
> cons: hard to setup (you may need a cloud storage for middleware).

I don't use cloud storage other than my own nextcloud server.

> 
> For me I most often use scp + rsync. and what's your choice?

It depends on what I am doing. I use rsnapshot for backups. scp for
one-offs. I use emacs' tramp mode to edit files on other machines. (It
uses scp by default bu you can specify other transport mechanisms.)

I also use unison, nextcloud and syncthing to sync user data between my
desktop and my laptops. The former is manual for occasional use, the
latter two run automatically as files change. The first two I use when I
am away to back up the laptop; the latter I don't use when away. (Hotels
tend to have crappy networks.)



-- 
Does anybody read signatures any more?

https://charlescurley.com
https://charlescurley.com/blog/

Re: what method do you prefer for data transfer between nodes?

[Ken Young]

That's smart. Thanks

Sincerely,
Ken Young


On Mon, Mar 6, 2023 at 3:43 AM Linux-Fan  wrote:

> Ken Young writes:
>
> > Hello,[1;5B
> >
> >
> > The methods I know,
> >
> > 1. scp
> > pros: the native tool in the OS
> > cons: you will either input password or put key pairs into servers for
> > authentication.
>
> Works for simple cases.
>
> > 2. rsync
> > pros: it can transfer data by increasement
> > cons: you need to setup rsyncd server and make the correct authorization.
>
> Works for simple and complex cases.
>
> > 3. ftp/ftps
> > pros: easy to use
> > cons: need to setup ftpd server, and the way is not that secure?
>
> Whenever possible, I'd prefer 1 or 2 over this.
>
> > 4. rclone
> > pros:easy to use
> > cons: hard to setup (you may need a cloud storage for middleware).
>
> I only use rclone when I want to target a cloud storage.
> A „cloud storage for middleware” does not seem sensible to me when I can
> copy using methods 1 and 2 without using such a middleware.
>
> > For me I most often use scp + rsync. and what's your choice?
>
> These are my standard choices, too. In automated scenarios I often prefer
> rsync over scp due to more flexibility in configuration.
>
> My additional tools for special purposes:
>
> 5. lsyncd
> If you need to keep directories in sync continuously, there is a tool
> called
> `lsyncd` that automates repeated invocation of `rsync` in a smart way.
>
> 6. tar + netcat (or tar + ssh in very rare cases)
> Using tar sacrifices all the flexibility of rsync but may attain a
> significantly higher performance and does not need a lot of flags to do
> the
> right thing by default (i.e. preserve everything when acting as root). I
> prefer this variant when migrating to a new disk or PC because it seems
> to
> be the most efficient variant in a "local trusted network and no speedup
> from incremental copying" scenario.
>
> I documented my approach to this here:
> https://masysma.net/37/data_transfer_netcat_tar.xhtml
>
> HTH and YMMV
> Linux-Fan
>
> öö
>

Re: what method do you prefer for data transfer between nodes?

[Linux-Fan]

Ken Young writes:


Hello,[1;5B


The methods I know,

1. scp
pros: the native tool in the OS
cons: you will either input password or put key pairs into servers for  
authentication.


Works for simple cases.


2. rsync
pros: it can transfer data by increasement 
cons: you need to setup rsyncd server and make the correct authorization.


Works for simple and complex cases.


3. ftp/ftps
pros: easy to use
cons: need to setup ftpd server, and the way is not that secure?


Whenever possible, I'd prefer 1 or 2 over this.


4. rclone
pros:easy to use
cons: hard to setup (you may need a cloud storage for middleware).


I only use rclone when I want to target a cloud storage.
A „cloud storage for middleware” does not seem sensible to me when I can  
copy using methods 1 and 2 without using such a middleware.



For me I most often use scp + rsync. and what's your choice?


These are my standard choices, too. In automated scenarios I often prefer  
rsync over scp due to more flexibility in configuration.


My additional tools for special purposes:

5. lsyncd
If you need to keep directories in sync continuously, there is a tool called  
`lsyncd` that automates repeated invocation of `rsync` in a smart way.


6. tar + netcat (or tar + ssh in very rare cases)
Using tar sacrifices all the flexibility of rsync but may attain a  
significantly higher performance and does not need a lot of flags to do the  
right thing by default (i.e. preserve everything when acting as root). I  
prefer this variant when migrating to a new disk or PC because it seems to  
be the most efficient variant in a "local trusted network and no speedup  
from incremental copying" scenario.


I documented my approach to this here:
https://masysma.net/37/data_transfer_netcat_tar.xhtml

HTH and YMMV
Linux-Fan

öö


pgpX071P0Phsq.pgp
Description: PGP signature

Re: what method do you prefer for data transfer between nodes?

[Rodrigo Cunha]

Hi Ken,
For short, I use scp and rsync too. Work fine with my scripts. Ftp works
fine on old servers. I saw old servers work with this, but with monitoring
in the FTP port for recovery. Another is to mount an NFS directory at
/etc/fstab. Work fine, too. If you have a critical script at the
scheduler(crontab) may be monitoring this with logs.

On Sun, Mar 5, 2023 at 10:48 AM Ken Young  wrote:

> Thanks for the suggestion. I will give it a try on mailbox.org.
>
>
> Sincerely,
> Ken Young
>
>
> On Sun, Mar 5, 2023 at 9:05 PM  wrote:
>
>> On Sun, Mar 05, 2023 at 07:23:49PM +0800, Ken Young wrote:
>> > What provider do you recommend then?
>>
>> Around here (West Europe), for example, mailbox.org [1]. But they
>> are far from the only one (I'm not a customer, nor associated with
>> them, but I do know a few happy customers).
>>
>> Whenever they don't cost any money you'll have to ask yourself
>> what their business model is.
>>
>> Cheers
>>
>> [1] https://mailbox.org/en/
>> --
>> t
>>
>

-- 
Atenciosamente,
Rodrigo da Silva Cunha
São Gonçalo, RJ - Brasil

Re: what method do you prefer for data transfer between nodes?

[Ken Young]

Thanks for the suggestion. I will give it a try on mailbox.org.


Sincerely,
Ken Young


On Sun, Mar 5, 2023 at 9:05 PM  wrote:

> On Sun, Mar 05, 2023 at 07:23:49PM +0800, Ken Young wrote:
> > What provider do you recommend then?
>
> Around here (West Europe), for example, mailbox.org [1]. But they
> are far from the only one (I'm not a customer, nor associated with
> them, but I do know a few happy customers).
>
> Whenever they don't cost any money you'll have to ask yourself
> what their business model is.
>
> Cheers
>
> [1] https://mailbox.org/en/
> --
> t
>

Re: what method do you prefer for data transfer between nodes?

[tomas]

On Sun, Mar 05, 2023 at 07:23:49PM +0800, Ken Young wrote:
> What provider do you recommend then?

Around here (West Europe), for example, mailbox.org [1]. But they
are far from the only one (I'm not a customer, nor associated with
them, but I do know a few happy customers).

Whenever they don't cost any money you'll have to ask yourself
what their business model is.

Cheers

[1] https://mailbox.org/en/
-- 
t


signature.asc
Description: PGP signature

Re: what method do you prefer for data transfer between nodes?

[Timothy M Butterworth]

On Sun, Mar 5, 2023 at 4:00 AM Kamil Jońca  wrote:

> Ken Young  writes:
>
> > Hello,
> >
> > The methods I know,
>
> Short answer: it depends.
>
> I use:
> scp
> rdbms (central server with data + clients)
> git
> samba
>
> FuseSSH with the FISH protocol in Dolphin.



> I cannot say I "prefer" any of them.
>
> KJ
>
> --
> http://stopstopnop.pl/stop_stopnop.pl_o_nas.html
>
>

-- 
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org/
⠈⠳⣄⠀⠀

Re: what method do you prefer for data transfer between nodes?

[Ken Young]

What provider do you recommend then?

Sincerely,
Ken Young


On Sun, Mar 5, 2023 at 6:54 PM  wrote:

> On Sun, Mar 05, 2023 at 11:31:42AM +0100, Michel Verdier wrote:
> > Le 5 mars 2023 tomas a écrit :
> >
> > > The others I'd only use when forced to. Well, rclone, I think,
> > > I'd even refuse when forced to. Eek.
> >
> > Do you have an alternate solution for cloud storage like google one ?
>
> I don't "do" clouds. But around here, I'd recommend a provider who offers
> mail and nextcloud for a small monthly amount (1 EUR, I think).
>
> Cheers
> --
> t
>

Re: what method do you prefer for data transfer between nodes?

[tomas]

On Sun, Mar 05, 2023 at 11:31:42AM +0100, Michel Verdier wrote:
> Le 5 mars 2023 tomas a écrit :
> 
> > The others I'd only use when forced to. Well, rclone, I think,
> > I'd even refuse when forced to. Eek.
> 
> Do you have an alternate solution for cloud storage like google one ?

I don't "do" clouds. But around here, I'd recommend a provider who offers
mail and nextcloud for a small monthly amount (1 EUR, I think).

Cheers
-- 
t


signature.asc
Description: PGP signature

Re: what method do you prefer for data transfer between nodes?

[Michel Verdier]

Le 5 mars 2023 tomas a écrit :

> The others I'd only use when forced to. Well, rclone, I think,
> I'd even refuse when forced to. Eek.

Do you have an alternate solution for cloud storage like google one ?

Re: what method do you prefer for data transfer between nodes?

[Michel Verdier]

Le 5 mars 2023 Ken Young a écrit :

> 2. rsync
> pros: it can transfer data by increasement
> cons: you need to setup rsyncd server and make the correct authorization.

Definitely rsync over ssh : easy and secure. You can further filter
commands and automate with proper ssh keys.

> 4. rclone
> pros:easy to use
> cons: hard to setup (you may need a cloud storage for middleware).

Only for sync with cloud storage

Re: what method do you prefer for data transfer between nodes?

[tomas]

On Sun, Mar 05, 2023 at 04:22:25PM +0800, Ken Young wrote:
> Hello,
> 
> The methods I know,
> 
> 1. scp
> pros: the native tool in the OS
> cons: you will either input password or put key pairs into servers for
> authentication.

Sometimes. From the command line, and if I only have to transfer a
hadnful of files, this one is the most convenient.

> 2. rsync
> pros: it can transfer data by increasement
> cons: you need to setup rsyncd server and make the correct authorization.

This. Hands down.

Pros:
Your list of pros is just too short. A corollary
of your first pro is that you can interrupt the proces at any time
and it'll pick up more or less where it left. Plus you can ask it
what it would do (--dry-run). Plus you you can tell it to make
back-up files at the other end, and to delete non-existing files.
Yadda, yadda.

Cons:
 - you need to setup rsyncd server
   Wrong: you just need one "transport", SSH is fine.
   If you can access the other box via SSH, you're set.

 - and make the correct authorization.
   I don't know what you mean with that: I'd hope any
   method to put files on a box would need proper
   authentificationn/authorization.

The others I'd only use when forced to. Well, rclone, I think,
I'd even refuse when forced to. Eek.

Cheers
-- 
t


signature.asc
Description: PGP signature

Re: what method do you prefer for data transfer between nodes?

[Kamil Jońca]

Ken Young  writes:

> Hello,
>
> The methods I know,

Short answer: it depends.

I use:
scp
rdbms (central server with data + clients)
git
samba

I cannot say I "prefer" any of them.

KJ

-- 
http://stopstopnop.pl/stop_stopnop.pl_o_nas.html

Re: what method do you prefer for data transfer between nodes?

[Jeremy Ardley]

On 5/3/23 16:22, Ken Young wrote:

Hello,

The methods I know,

1. scp
pros: the native tool in the OS
cons: you will either input password or put key pairs into servers for 
authentication.


2. rsync
pros: it can transfer data by increasement
cons: you need to setup rsyncd server and make the correct authorization.

3. ftp/ftps
pros: easy to use
cons: need to setup ftpd server, and the way is not that secure?

4. rclone
pros:easy to use
cons: hard to setup (you may need a cloud storage for middleware).

For me I most often use scp + rsync. and what's your choice?

Regards,
Ken


You can also use git or subversion by way of a third part server.

It involves double handling but I have a vague idea that they both do 
incremental changes faster than rsync. They definitely allow rollbacks.


--
Jeremy
(Lists)