Re: [gentoo-user] replacement for ftp?

[lee]

Alan McKinnon  writes:

> On 03/05/2017 22:04, lee wrote:
>> Alan McKinnon  writes:
>> 
>>> On 30/04/2017 03:11, lee wrote:
 "Poison BL."  writes:

> On Sat, Apr 29, 2017 at 3:24 PM, lee  wrote:
>
>> Mick  writes:
>>
>>> On Tuesday 25 Apr 2017 16:45:37 Alan McKinnon wrote:
 On 25/04/2017 16:29, lee wrote:
> Hi,
>
> since the usage of FTP seems to be declining, what is a replacement
> which is at least as good as FTP?
>
> I'm aware that there's webdav, but that's very awkward to use and
> missing features.

 Why not stick with ftp?
 Or, put another way, why do you feel you need to use something else?

 There's always dropbox
>>>
>>>
>>> Invariably all web hosting ISPs offer ftp(s) for file upload/download.
>> If you
>>> pay a bit more you should be able to get ssh/scp/sftp too.  Indeed, many
>> ISPs
>>> throw in scp/sftp access as part of their basic package.
>>>
>>> Webdav(s) offers the same basic upload/download functionality, so I am
>> not
>>> sure what you find awkward about it, although I'd rather use lftp
>> instead of
>>> cadaver any day. ;-)
>>>
>>> As Alan mentioned, with JavaScript'ed web pages these days there are 
>>> many
>>> webapp'ed ISP offerings like Dropbox and friends.
>>>
>>> What is the use case you have in mind?
>>
>> transferring large amounts of data and automatization in processing at
>> least some of it, without involving a 3rd party
>>
>> "Large amounts" can be "small" like 100MB --- or over 50k files in 12GB,
>> or even more.  The mirror feature of lftp is extremely useful for such
>> things.
>>
>> I wouldn't ever want having to mess around with web pages to figure out
>> how to do this.  Ftp is plain and simple.  So you see why I'm explicitly
>> asking for a replacement which is at least as good as ftp.
>>
>>
>> --
>> "Didn't work" is an error.
>>
>>
> Half petabyte datasets aren't really something I'd personally *ever* trust
> ftp with in the first place.

 Why not?  (12GB are nowhere close to half a petabyte ...)

> That said, it depends entirely on the network
> you're working with. Are you pushing this data in/out of the network your
> machines live in, or are you working primarily internally? If internal,
> what're the network side capabilities you have? Since you're likely 
> already
> using something on the order of CEPH or Gluster to back the datasets where
> they sit, just working with it all across network from that storage would
> be my first instinct.

 The data would come in from suppliers.  There isn't really anything
 going on atm but fetching data once a month which can be like 100MB or
 12GB or more.  That's because ppl don't use ftp ...
>>>
>>> I have the opposite experience.
>>> I have the devil's own time trying to convince people to NOT use ftp for
>>> anything and everything under the sun that even remotely resembles
>>> getting data from A to B...
>> 
>> I guess you're lucky then.
>> 
>>> (especially things that are best done over a
>>> message bus)
>> 
>> Why would anyone try to transfer data over a message bus?  Doesn't that
>> require extra wiring and specialized hardware?
>> 
>>> I'm still not understanding why you are asking your questions. What you
>>> describe looks like the ideal case for ftp:
>> 
>> it is
>> 
>> Still nobody uses it, and apparently ftp usage is generally declining,
>> so I would expect there to be a better alternative.
>> 
>>>
>>> - supplier pushes a file or files somewhere
>>> - you fetch those files later at a suitable time
>>>
>>> it looks like a classic producer/consumer scenario and ftp or any of
>>> it's webby clones like dropbox really it still the best tool overall.
>>> Plus it has the added benefit that no user needs extra software - all
>>> OSes have ftp clients even if it's just a browser
>> 
>> The users don't know about that.
>> 
>> 
>
>
> OK, so here is what you have.
>
> You apparently must use ftp as pretty much nothing else works.

I was merely asking if there is a better solution.

> You als claim that your users are too stupid to use ftp, and can't even
> type ftp:// into a browser.

I never said that.

> I'm sorry, but that's only marginally more believable than claiming
> keyboards are too complicated for your users.

Does it matter what you or I believe?  Some users have difficulties
using a keyboard and/or a mouse.  I've seen that, so no, what you or I
believe does not matter.

> I doubt anyone here can help you - you do not have a technical problem,
> you have a people problem. I recommend the very first action you take
> now is to critically re-examine 

Re: [gentoo-user] replacement for ftp?

[Alan McKinnon]

On 03/05/2017 22:04, lee wrote:
> Alan McKinnon  writes:
> 
>> On 30/04/2017 03:11, lee wrote:
>>> "Poison BL."  writes:
>>>
 On Sat, Apr 29, 2017 at 3:24 PM, lee  wrote:

> Mick  writes:
>
>> On Tuesday 25 Apr 2017 16:45:37 Alan McKinnon wrote:
>>> On 25/04/2017 16:29, lee wrote:
 Hi,

 since the usage of FTP seems to be declining, what is a replacement
 which is at least as good as FTP?

 I'm aware that there's webdav, but that's very awkward to use and
 missing features.
>>>
>>> Why not stick with ftp?
>>> Or, put another way, why do you feel you need to use something else?
>>>
>>> There's always dropbox
>>
>>
>> Invariably all web hosting ISPs offer ftp(s) for file upload/download.
> If you
>> pay a bit more you should be able to get ssh/scp/sftp too.  Indeed, many
> ISPs
>> throw in scp/sftp access as part of their basic package.
>>
>> Webdav(s) offers the same basic upload/download functionality, so I am
> not
>> sure what you find awkward about it, although I'd rather use lftp
> instead of
>> cadaver any day. ;-)
>>
>> As Alan mentioned, with JavaScript'ed web pages these days there are many
>> webapp'ed ISP offerings like Dropbox and friends.
>>
>> What is the use case you have in mind?
>
> transferring large amounts of data and automatization in processing at
> least some of it, without involving a 3rd party
>
> "Large amounts" can be "small" like 100MB --- or over 50k files in 12GB,
> or even more.  The mirror feature of lftp is extremely useful for such
> things.
>
> I wouldn't ever want having to mess around with web pages to figure out
> how to do this.  Ftp is plain and simple.  So you see why I'm explicitly
> asking for a replacement which is at least as good as ftp.
>
>
> --
> "Didn't work" is an error.
>
>
 Half petabyte datasets aren't really something I'd personally *ever* trust
 ftp with in the first place.
>>>
>>> Why not?  (12GB are nowhere close to half a petabyte ...)
>>>
 That said, it depends entirely on the network
 you're working with. Are you pushing this data in/out of the network your
 machines live in, or are you working primarily internally? If internal,
 what're the network side capabilities you have? Since you're likely already
 using something on the order of CEPH or Gluster to back the datasets where
 they sit, just working with it all across network from that storage would
 be my first instinct.
>>>
>>> The data would come in from suppliers.  There isn't really anything
>>> going on atm but fetching data once a month which can be like 100MB or
>>> 12GB or more.  That's because ppl don't use ftp ...
>>
>> I have the opposite experience.
>> I have the devil's own time trying to convince people to NOT use ftp for
>> anything and everything under the sun that even remotely resembles
>> getting data from A to B...
> 
> I guess you're lucky then.
> 
>> (especially things that are best done over a
>> message bus)
> 
> Why would anyone try to transfer data over a message bus?  Doesn't that
> require extra wiring and specialized hardware?
> 
>> I'm still not understanding why you are asking your questions. What you
>> describe looks like the ideal case for ftp:
> 
> it is
> 
> Still nobody uses it, and apparently ftp usage is generally declining,
> so I would expect there to be a better alternative.
> 
>>
>> - supplier pushes a file or files somewhere
>> - you fetch those files later at a suitable time
>>
>> it looks like a classic producer/consumer scenario and ftp or any of
>> it's webby clones like dropbox really it still the best tool overall.
>> Plus it has the added benefit that no user needs extra software - all
>> OSes have ftp clients even if it's just a browser
> 
> The users don't know about that.
> 
> 


OK, so here is what you have.

You apparently must use ftp as pretty much nothing else works.
You als claim that your users are too stupid to use ftp, and can't even
type ftp:// into a browser.

I'm sorry, but that's only marginally more believable than claiming
keyboards are too complicated for your users.

I doubt anyone here can help you - you do not have a technical problem,
you have a people problem. I recommend the very first action you take
now is to critically re-examine why you think of those users in the way
you do and fix/amend what is going on inside your head. Then your way
forward will be clear.


-- 
Alan McKinnon
alan.mckin...@gmail.com

Re: [gentoo-user] replacement for ftp?

[lee]

Alan McKinnon  writes:

> On 30/04/2017 03:11, lee wrote:
>> "Poison BL."  writes:
>> 
>>> On Sat, Apr 29, 2017 at 3:24 PM, lee  wrote:
>>>
 Mick  writes:

> On Tuesday 25 Apr 2017 16:45:37 Alan McKinnon wrote:
>> On 25/04/2017 16:29, lee wrote:
>>> Hi,
>>>
>>> since the usage of FTP seems to be declining, what is a replacement
>>> which is at least as good as FTP?
>>>
>>> I'm aware that there's webdav, but that's very awkward to use and
>>> missing features.
>>
>> Why not stick with ftp?
>> Or, put another way, why do you feel you need to use something else?
>>
>> There's always dropbox
>
>
> Invariably all web hosting ISPs offer ftp(s) for file upload/download.
 If you
> pay a bit more you should be able to get ssh/scp/sftp too.  Indeed, many
 ISPs
> throw in scp/sftp access as part of their basic package.
>
> Webdav(s) offers the same basic upload/download functionality, so I am
 not
> sure what you find awkward about it, although I'd rather use lftp
 instead of
> cadaver any day. ;-)
>
> As Alan mentioned, with JavaScript'ed web pages these days there are many
> webapp'ed ISP offerings like Dropbox and friends.
>
> What is the use case you have in mind?

 transferring large amounts of data and automatization in processing at
 least some of it, without involving a 3rd party

 "Large amounts" can be "small" like 100MB --- or over 50k files in 12GB,
 or even more.  The mirror feature of lftp is extremely useful for such
 things.

 I wouldn't ever want having to mess around with web pages to figure out
 how to do this.  Ftp is plain and simple.  So you see why I'm explicitly
 asking for a replacement which is at least as good as ftp.


 --
 "Didn't work" is an error.


>>> Half petabyte datasets aren't really something I'd personally *ever* trust
>>> ftp with in the first place.
>> 
>> Why not?  (12GB are nowhere close to half a petabyte ...)
>> 
>>> That said, it depends entirely on the network
>>> you're working with. Are you pushing this data in/out of the network your
>>> machines live in, or are you working primarily internally? If internal,
>>> what're the network side capabilities you have? Since you're likely already
>>> using something on the order of CEPH or Gluster to back the datasets where
>>> they sit, just working with it all across network from that storage would
>>> be my first instinct.
>> 
>> The data would come in from suppliers.  There isn't really anything
>> going on atm but fetching data once a month which can be like 100MB or
>> 12GB or more.  That's because ppl don't use ftp ...
>
> I have the opposite experience.
> I have the devil's own time trying to convince people to NOT use ftp for
> anything and everything under the sun that even remotely resembles
> getting data from A to B...

I guess you're lucky then.

> (especially things that are best done over a
> message bus)

Why would anyone try to transfer data over a message bus?  Doesn't that
require extra wiring and specialized hardware?

> I'm still not understanding why you are asking your questions. What you
> describe looks like the ideal case for ftp:

it is

Still nobody uses it, and apparently ftp usage is generally declining,
so I would expect there to be a better alternative.

>
> - supplier pushes a file or files somewhere
> - you fetch those files later at a suitable time
>
> it looks like a classic producer/consumer scenario and ftp or any of
> it's webby clones like dropbox really it still the best tool overall.
> Plus it has the added benefit that no user needs extra software - all
> OSes have ftp clients even if it's just a browser

The users don't know about that.


-- 
"Didn't work" is an error.

Re: [gentoo-user] replacement for ftp?

[lee]

"Poison BL."  writes:

> On Sat, Apr 29, 2017 at 9:11 PM, lee  wrote:
>>
>> "Poison BL."  writes:
>> > Half petabyte datasets aren't really something I'd personally *ever*
> trust
>> > ftp with in the first place.
>>
>> Why not?  (12GB are nowhere close to half a petabyte ...)
>
> Ah... I completely misread that "or over 50k files in 12GB" as 50k files
> *at* 12GB each... which works out to 0.6 PB, incidentally.
>
>> The data would come in from suppliers.  There isn't really anything
>> going on atm but fetching data once a month which can be like 100MB or
>> 12GB or more.  That's because ppl don't use ftp ...
>
> Really, if you're pulling it in from third party suppliers, you tend to be
> tied to what they offer as a method of pulling it from them (or them
> pushing it out to you), unless you're in the unique position to dictate the
> decision for them.

They need to use ftp to deliver the data, we need to use ftp to get the
data.  I don't want that any other way.

The problem is that the ones supposed to deliver data are incompetent
and don't want to use ftp because it's too complicated.  So what's the
better solution?


> [...]
>
>> > How often does it need moved in/out of your facility, and is there no
> way
>> > to break up the processing into smaller chunks than a 0.6PB mass of
> files?
>> > Distribute out the smaller pieces with rsync, scp, or the like, operate
> on
>> > them, and pull back in the results, rather than trying to shift around
> the
>> > entire set. There's a reason Amazon will send a physical truck to a
> site to
>> > import large datasets into glacier... ;)
>>
>> Amazon has trucks?  Perhaps they do in other countries.  Here, amazon is
>> just another web shop.  They might have some delivery vans, but I've
>> never seen one, so I doubt it.  And why would anyone give them their
>> data?  There's no telling what they would do with it.
>
> Amazon's also one of the best known cloud computing suppliers on the planet
> (AWS = Amazon Web Services). They have everything from pure compute
> offerings to cloud storage geared towards *large* data archival. The latter
> offering is named "glacier", and they offer a service for the import of
> data into it (usually the "first pass", incremental changes are generally
> done over the wire) that consists of a shipping truck with a rather nifty
> storage system in the back of it that they hook right into your network.
> You fill it with data, and then they drive it back to one of their data
> centers to load it into place.

They might not have that here.  And who would want to give their data
out of hands?


-- 
"Didn't work" is an error.

Re: [gentoo-user] replacement for ftp?

[lee]

"Walter Dnes"  writes:

>> transferring large amounts of data and automatization in processing at
>> least some of it, without involving a 3rd party
>> 
>> "Large amounts" can be "small" like 100MB --- or over 50k files in 12GB,
>> or even more.  The mirror feature of lftp is extremely useful for such
>> things.
>> 
>> I wouldn't ever want having to mess around with web pages to figure out
>> how to do this.  Ftp is plain and simple.  So you see why I'm explicitly
>> asking for a replacement which is at least as good as ftp.
>
>   How about "wget"?  It can handle ftp and http, and it can be scripted.

Explain to someone unable to use Filezilla how to do that ...

I'd have to look up if wget can do ftp with TLS.

> And it can discriminate on timestamps, i.e. only download a file if it
> has been changed since the latest download at your site.
>
>   Then there's always "sneakernet".  To quote Andrew Tanenbaum from 1981
>
>> Never underestimate the bandwidth of a station wagon full of tapes
>> hurtling down the highway.

Hm, I'll suggest that, thanks.  Ppl might be more likely to think they
should be able to burn DVDs and send them in the mail than they are to
think they could use something much simpler, easier, faster and more
secure, like ftp ...


-- 
"Didn't work" is an error.

Re: [gentoo-user] replacement for ftp?

[lee]

Nils Freydank  writes:

> On Sat, 30 Apr 2017 19:04:06 +0200 Andrew Savchenko wrote:
>> [...]
>> I fail to see why FTP needs to be replaced: it works, it is
>> supported, it is secure when used with care, it is damn fast.
>
> I’ll just drop the somewhat popular rant “FTP must die“[1] and a follow-up
> discussion about it[2]. IMHO the main reasons are missing data integrity and
> authentication security issues. The latter one can be solved with FTPS[3] - 
> but honestly
> I never saw FTPS somewhere actually used in the wild.

I've done it that way because I didn't want unencrypted transfers and
sftp appears to be deprecated.  It's working fine, and I don't want to
replace it unless there were a better solution.


>
>
> [1] http://mywiki.wooledge.org/FtpMustDie
> [2] https://news.ycombinator.com/item?id=11251907
> [3] i.e. FTP over SSL/TLS (not to mix up with SFTP, which comes from the SSH 
> family)
>
> Greetings,
> Nils

-- 
"Didn't work" is an error.

Re: [gentoo-user] replacement for ftp?

[Mick]

On Monday 01 May 2017 22:36:00 Nils Freydank wrote:
> On Sat, 30 Apr 2017 19:04:06 +0200 Andrew Savchenko wrote:
> > [...]
> > I fail to see why FTP needs to be replaced: it works, it is
> > supported, it is secure when used with care, it is damn fast.
> 
> I’ll just drop the somewhat popular rant “FTP must die“[1] and a follow-up
> discussion about it[2]. IMHO the main reasons are missing data integrity and
> authentication security issues. The latter one can be solved with FTPS[3] -
> but honestly I never saw FTPS somewhere actually used in the wild.

I'm not sure what you mean "used in the wild".  I use lftp to connect via ftps 
with a number of webservers for updates and backups on a daily basis.  Some of 
the connections are scripted.


> [1] http://mywiki.wooledge.org/FtpMustDie
> [2] https://news.ycombinator.com/item?id=11251907
> [3] i.e. FTP over SSL/TLS (not to mix up with SFTP, which comes from the SSH
> family)
> 
> Greetings,
> Nils

-- 
Regards,
Mick

signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-user] replacement for ftp?

[Nils Freydank]

On Sat, 30 Apr 2017 19:04:06 +0200 Andrew Savchenko wrote:
> [...]
> I fail to see why FTP needs to be replaced: it works, it is
> supported, it is secure when used with care, it is damn fast.

I’ll just drop the somewhat popular rant “FTP must die“[1] and a follow-up
discussion about it[2]. IMHO the main reasons are missing data integrity and
authentication security issues. The latter one can be solved with FTPS[3] - but 
honestly
I never saw FTPS somewhere actually used in the wild.


[1] http://mywiki.wooledge.org/FtpMustDie
[2] https://news.ycombinator.com/item?id=11251907
[3] i.e. FTP over SSL/TLS (not to mix up with SFTP, which comes from the SSH 
family)

Greetings,
Nils

-- 
GPG fingerprint: '00EF D31F 1B60 D5DB ADB8  31C1 C0EC E696 0E54 475B'
Nils Freydank

signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-user] replacement for ftp?

[Andrew Savchenko]

On Tue, 25 Apr 2017 15:29:18 +0100 lee wrote:
> 
> Hi,
> 
> since the usage of FTP seems to be declining, what is a replacement
> which is at least as good as FTP?

I fail to see why FTP needs to be replaced: it works, it is
supported, it is secure when used with care, it is damn fast.

Best regards,
Andrew Savchenko


pgpH4meGDETra.pgp
Description: PGP signature

Re: [gentoo-user] replacement for ftp?

[Poison BL.]

On Sat, Apr 29, 2017 at 9:11 PM, lee  wrote:
>
> "Poison BL."  writes:
> > Half petabyte datasets aren't really something I'd personally *ever*
trust
> > ftp with in the first place.
>
> Why not?  (12GB are nowhere close to half a petabyte ...)

Ah... I completely misread that "or over 50k files in 12GB" as 50k files
*at* 12GB each... which works out to 0.6 PB, incidentally.

> The data would come in from suppliers.  There isn't really anything
> going on atm but fetching data once a month which can be like 100MB or
> 12GB or more.  That's because ppl don't use ftp ...

Really, if you're pulling it in from third party suppliers, you tend to be
tied to what they offer as a method of pulling it from them (or them
pushing it out to you), unless you're in the unique position to dictate the
decision for them. From there, assuming you can push your choice of product
on them, it becomes a question of how often the same dataset will need
updated from the same sources, how much it changes between updates, how
secure it needs to be in transit, how much you need to be able to trust
that the source is still legitimately who you think it is, and how much
verification that there wasn't any corruption during the transfer. Generic
FTP has been losing favor over time because it was built in a time that
many of those questions weren't really at the top of the list for concerns.

SFTP (or SCP) (as long as keys are handled properly) allows for pretty
solid certainty that a) both ends of the connection are who they say they
are, b) those two ends are the only ones reading the data in transit, and
c) the data that was sent is the same that was received (simply as a side
benefit of the encryption/decryption). Rsync over SSH gives the same set of
benefits, reduces the bandwidth used for updating the dataset (when it's
the same dataset, at least), and will also verify the data on both ends (as
it exists on disk) matches. If you're particularly lucky, the data might
even hit just the right mark that benefits from the in-line compression you
can turn on with SSH, too, cutting down the actual amount of bandwidth you
burn through for each transfer.

If your suppliers all have *nix based systems available, those are also
standard tools that they'll have on hand. If they're strictly Windows
shops, SCP/SFTP are still readily available, though they aren't built into
the OS... rsync gets a bit trickier.

> > How often does it need moved in/out of your facility, and is there no
way
> > to break up the processing into smaller chunks than a 0.6PB mass of
files?
> > Distribute out the smaller pieces with rsync, scp, or the like, operate
on
> > them, and pull back in the results, rather than trying to shift around
the
> > entire set. There's a reason Amazon will send a physical truck to a
site to
> > import large datasets into glacier... ;)
>
> Amazon has trucks?  Perhaps they do in other countries.  Here, amazon is
> just another web shop.  They might have some delivery vans, but I've
> never seen one, so I doubt it.  And why would anyone give them their
> data?  There's no telling what they would do with it.

Amazon's also one of the best known cloud computing suppliers on the planet
(AWS = Amazon Web Services). They have everything from pure compute
offerings to cloud storage geared towards *large* data archival. The latter
offering is named "glacier", and they offer a service for the import of
data into it (usually the "first pass", incremental changes are generally
done over the wire) that consists of a shipping truck with a rather nifty
storage system in the back of it that they hook right into your network.
You fill it with data, and then they drive it back to one of their data
centers to load it into place.

--
Poison [BLX]
Joshua M. Murphy

Re: [gentoo-user] replacement for ftp?

[Alan McKinnon]

On 30/04/2017 03:11, lee wrote:
> "Poison BL."  writes:
> 
>> On Sat, Apr 29, 2017 at 3:24 PM, lee  wrote:
>>
>>> Mick  writes:
>>>
 On Tuesday 25 Apr 2017 16:45:37 Alan McKinnon wrote:
> On 25/04/2017 16:29, lee wrote:
>> Hi,
>>
>> since the usage of FTP seems to be declining, what is a replacement
>> which is at least as good as FTP?
>>
>> I'm aware that there's webdav, but that's very awkward to use and
>> missing features.
>
> Why not stick with ftp?
> Or, put another way, why do you feel you need to use something else?
>
> There's always dropbox


 Invariably all web hosting ISPs offer ftp(s) for file upload/download.
>>> If you
 pay a bit more you should be able to get ssh/scp/sftp too.  Indeed, many
>>> ISPs
 throw in scp/sftp access as part of their basic package.

 Webdav(s) offers the same basic upload/download functionality, so I am
>>> not
 sure what you find awkward about it, although I'd rather use lftp
>>> instead of
 cadaver any day. ;-)

 As Alan mentioned, with JavaScript'ed web pages these days there are many
 webapp'ed ISP offerings like Dropbox and friends.

 What is the use case you have in mind?
>>>
>>> transferring large amounts of data and automatization in processing at
>>> least some of it, without involving a 3rd party
>>>
>>> "Large amounts" can be "small" like 100MB --- or over 50k files in 12GB,
>>> or even more.  The mirror feature of lftp is extremely useful for such
>>> things.
>>>
>>> I wouldn't ever want having to mess around with web pages to figure out
>>> how to do this.  Ftp is plain and simple.  So you see why I'm explicitly
>>> asking for a replacement which is at least as good as ftp.
>>>
>>>
>>> --
>>> "Didn't work" is an error.
>>>
>>>
>> Half petabyte datasets aren't really something I'd personally *ever* trust
>> ftp with in the first place.
> 
> Why not?  (12GB are nowhere close to half a petabyte ...)
> 
>> That said, it depends entirely on the network
>> you're working with. Are you pushing this data in/out of the network your
>> machines live in, or are you working primarily internally? If internal,
>> what're the network side capabilities you have? Since you're likely already
>> using something on the order of CEPH or Gluster to back the datasets where
>> they sit, just working with it all across network from that storage would
>> be my first instinct.
> 
> The data would come in from suppliers.  There isn't really anything
> going on atm but fetching data once a month which can be like 100MB or
> 12GB or more.  That's because ppl don't use ftp ...

I have the opposite experience.
I have the devil's own time trying to convince people to NOT use ftp for
anything and everything under the sun that even remotely resembles
getting data from A to B... (especially things that are best done over a
message bus)

I'm still not understanding why you are asking your questions. What you
describe looks like the ideal case for ftp:

- supplier pushes a file or files somewhere
- you fetch those files later at a suitable time

it looks like a classic producer/consumer scenario and ftp or any of
it's webby clones like dropbox really it still the best tool overall.
Plus it has the added benefit that no user needs extra software - all
OSes have ftp clients even if it's just a browser

-- 
Alan McKinnon
alan.mckin...@gmail.com

Re: [gentoo-user] replacement for ftp?

[Walter Dnes]

> transferring large amounts of data and automatization in processing at
> least some of it, without involving a 3rd party
> 
> "Large amounts" can be "small" like 100MB --- or over 50k files in 12GB,
> or even more.  The mirror feature of lftp is extremely useful for such
> things.
> 
> I wouldn't ever want having to mess around with web pages to figure out
> how to do this.  Ftp is plain and simple.  So you see why I'm explicitly
> asking for a replacement which is at least as good as ftp.

  How about "wget"?  It can handle ftp and http, and it can be scripted.
And it can discriminate on timestamps, i.e. only download a file if it
has been changed since the latest download at your site.

  Then there's always "sneakernet".  To quote Andrew Tanenbaum from 1981

> Never underestimate the bandwidth of a station wagon full of tapes
> hurtling down the highway.

-- 
Walter Dnes 
I don't run "desktop environments"; I run useful applications

Re: [gentoo-user] replacement for ftp?

[lee]

"Poison BL."  writes:

> On Sat, Apr 29, 2017 at 3:24 PM, lee  wrote:
>
>> Mick  writes:
>>
>> > On Tuesday 25 Apr 2017 16:45:37 Alan McKinnon wrote:
>> >> On 25/04/2017 16:29, lee wrote:
>> >> > Hi,
>> >> >
>> >> > since the usage of FTP seems to be declining, what is a replacement
>> >> > which is at least as good as FTP?
>> >> >
>> >> > I'm aware that there's webdav, but that's very awkward to use and
>> >> > missing features.
>> >>
>> >> Why not stick with ftp?
>> >> Or, put another way, why do you feel you need to use something else?
>> >>
>> >> There's always dropbox
>> >
>> >
>> > Invariably all web hosting ISPs offer ftp(s) for file upload/download.
>> If you
>> > pay a bit more you should be able to get ssh/scp/sftp too.  Indeed, many
>> ISPs
>> > throw in scp/sftp access as part of their basic package.
>> >
>> > Webdav(s) offers the same basic upload/download functionality, so I am
>> not
>> > sure what you find awkward about it, although I'd rather use lftp
>> instead of
>> > cadaver any day. ;-)
>> >
>> > As Alan mentioned, with JavaScript'ed web pages these days there are many
>> > webapp'ed ISP offerings like Dropbox and friends.
>> >
>> > What is the use case you have in mind?
>>
>> transferring large amounts of data and automatization in processing at
>> least some of it, without involving a 3rd party
>>
>> "Large amounts" can be "small" like 100MB --- or over 50k files in 12GB,
>> or even more.  The mirror feature of lftp is extremely useful for such
>> things.
>>
>> I wouldn't ever want having to mess around with web pages to figure out
>> how to do this.  Ftp is plain and simple.  So you see why I'm explicitly
>> asking for a replacement which is at least as good as ftp.
>>
>>
>> --
>> "Didn't work" is an error.
>>
>>
> Half petabyte datasets aren't really something I'd personally *ever* trust
> ftp with in the first place.

Why not?  (12GB are nowhere close to half a petabyte ...)

> That said, it depends entirely on the network
> you're working with. Are you pushing this data in/out of the network your
> machines live in, or are you working primarily internally? If internal,
> what're the network side capabilities you have? Since you're likely already
> using something on the order of CEPH or Gluster to back the datasets where
> they sit, just working with it all across network from that storage would
> be my first instinct.

The data would come in from suppliers.  There isn't really anything
going on atm but fetching data once a month which can be like 100MB or
12GB or more.  That's because ppl don't use ftp ...

> How often does it need moved in/out of your facility, and is there no way
> to break up the processing into smaller chunks than a 0.6PB mass of files?
> Distribute out the smaller pieces with rsync, scp, or the like, operate on
> them, and pull back in the results, rather than trying to shift around the
> entire set. There's a reason Amazon will send a physical truck to a site to
> import large datasets into glacier... ;)

Amazon has trucks?  Perhaps they do in other countries.  Here, amazon is
just another web shop.  They might have some delivery vans, but I've
never seen one, so I doubt it.  And why would anyone give them their
data?  There's no telling what they would do with it.


-- 
"Didn't work" is an error.

Re: [gentoo-user] replacement for ftp?

[Poison BL.]

On Sat, Apr 29, 2017 at 3:24 PM, lee  wrote:

> Mick  writes:
>
> > On Tuesday 25 Apr 2017 16:45:37 Alan McKinnon wrote:
> >> On 25/04/2017 16:29, lee wrote:
> >> > Hi,
> >> >
> >> > since the usage of FTP seems to be declining, what is a replacement
> >> > which is at least as good as FTP?
> >> >
> >> > I'm aware that there's webdav, but that's very awkward to use and
> >> > missing features.
> >>
> >> Why not stick with ftp?
> >> Or, put another way, why do you feel you need to use something else?
> >>
> >> There's always dropbox
> >
> >
> > Invariably all web hosting ISPs offer ftp(s) for file upload/download.
> If you
> > pay a bit more you should be able to get ssh/scp/sftp too.  Indeed, many
> ISPs
> > throw in scp/sftp access as part of their basic package.
> >
> > Webdav(s) offers the same basic upload/download functionality, so I am
> not
> > sure what you find awkward about it, although I'd rather use lftp
> instead of
> > cadaver any day. ;-)
> >
> > As Alan mentioned, with JavaScript'ed web pages these days there are many
> > webapp'ed ISP offerings like Dropbox and friends.
> >
> > What is the use case you have in mind?
>
> transferring large amounts of data and automatization in processing at
> least some of it, without involving a 3rd party
>
> "Large amounts" can be "small" like 100MB --- or over 50k files in 12GB,
> or even more.  The mirror feature of lftp is extremely useful for such
> things.
>
> I wouldn't ever want having to mess around with web pages to figure out
> how to do this.  Ftp is plain and simple.  So you see why I'm explicitly
> asking for a replacement which is at least as good as ftp.
>
>
> --
> "Didn't work" is an error.
>
>
Half petabyte datasets aren't really something I'd personally *ever* trust
ftp with in the first place. That said, it depends entirely on the network
you're working with. Are you pushing this data in/out of the network your
machines live in, or are you working primarily internally? If internal,
what're the network side capabilities you have? Since you're likely already
using something on the order of CEPH or Gluster to back the datasets where
they sit, just working with it all across network from that storage would
be my first instinct.

How often does it need moved in/out of your facility, and is there no way
to break up the processing into smaller chunks than a 0.6PB mass of files?
Distribute out the smaller pieces with rsync, scp, or the like, operate on
them, and pull back in the results, rather than trying to shift around the
entire set. There's a reason Amazon will send a physical truck to a site to
import large datasets into glacier... ;)

-- 
Poison [BLX]
Joshua M. Murphy

Re: [gentoo-user] replacement for ftp?

[lee]

"Poison BL."  writes:

> On Tue, Apr 25, 2017 at 10:29 AM, lee  wrote:
>
>>
>> Hi,
>>
>> since the usage of FTP seems to be declining, what is a replacement
>> which is at least as good as FTP?
>>
>> I'm aware that there's webdav, but that's very awkward to use and
>> missing features.
>>
>>
>> --
>> "Didn't work" is an error.
>>
>>
> The one issue I have with all the answers I've seen is that they all lack
> the most important question. You're asking for alternatives for an old tool
> that was used for many use cases that, these days, have evolved to have
> very different requirements for security, integration of access methods,
> and general workflows for use. FTP used to be the go-to for long distance
> file sharing for *all* use cases, one to one (user managing a website's
> content), many to one (upload site), one to many (download site), etc.
> What's your use case?

all of them, with encrypted transfers and users needing a password for
access

I don't know anything better than ftp for this.  Alternatively, there
would need to be several different services for each group of users
accommodating their particular use case, and being a nightmare to deploy,
to maintain and to use.


-- 
"Didn't work" is an error.

Re: [gentoo-user] replacement for ftp?

[lee]

Alan McKinnon  writes:

> On 25/04/2017 16:29, lee wrote:
>> 
>> Hi,
>> 
>> since the usage of FTP seems to be declining, what is a replacement
>> which is at least as good as FTP?
>> 
>> I'm aware that there's webdav, but that's very awkward to use and
>> missing features.
>> 
>> 
>
> Why not stick with ftp?

The intended users are incompetent, hence it is too difficult to use ...

> Or, put another way, why do you feel you need to use something else?

I don't want to use anything else.

Yet even Debian has announced that they will shut down their ftp
services in November, one of the reasons being that almost no one uses
them.  Of course, their application is different from what I'm looking
for because they only have downloads and no uploads.

However, another reason given was that ftp isn't exactly friendly to
firewalls and requires "awkward kludges" when load balancing is used.
That is a pretty good reason.

Anyway, when pretty much nobody uses a particular software anymore, it
won't be very feasible to use that software.

> There's always dropbox

Well, dropbox sucks.  I got a dropbox link and it didn't work at all,
and handing out the data to some 3rd party is a very bad idea.  It's
also difficult to automate things with that.


-- 
"Didn't work" is an error.

Re: [gentoo-user] replacement for ftp?

[lee]

Danny YUE  writes:

> On 2017-04-25 14:29, lee  wrote:
>> Hi,
>>
>> since the usage of FTP seems to be declining, what is a replacement
>> which is at least as good as FTP?
>>
>> I'm aware that there's webdav, but that's very awkward to use and
>> missing features.
>
> What about sshfs? It allows you to mount a location that can be accessed
> via ssh to your local file system, as if you are using ssh.

Doesn't that require ssh access?  And how do you explain that to ppl
finding it too difficult to use Filezilla?  Is it available for Windoze?

> Also samba can be a replacement. I have a samba server on my OpenWRT
> router and use mount.cifs to mount it...

Does that work well, reliably and securely over internet connections?


-- 
"Didn't work" is an error.

Re: [gentoo-user] replacement for ftp?

[lee]

Mick  writes:

> On Tuesday 25 Apr 2017 16:45:37 Alan McKinnon wrote:
>> On 25/04/2017 16:29, lee wrote:
>> > Hi,
>> > 
>> > since the usage of FTP seems to be declining, what is a replacement
>> > which is at least as good as FTP?
>> > 
>> > I'm aware that there's webdav, but that's very awkward to use and
>> > missing features.
>> 
>> Why not stick with ftp?
>> Or, put another way, why do you feel you need to use something else?
>> 
>> There's always dropbox
>
>
> Invariably all web hosting ISPs offer ftp(s) for file upload/download.  If 
> you 
> pay a bit more you should be able to get ssh/scp/sftp too.  Indeed, many ISPs 
> throw in scp/sftp access as part of their basic package.
>
> Webdav(s) offers the same basic upload/download functionality, so I am not 
> sure what you find awkward about it, although I'd rather use lftp instead of 
> cadaver any day. ;-)
>
> As Alan mentioned, with JavaScript'ed web pages these days there are many 
> webapp'ed ISP offerings like Dropbox and friends.
>
> What is the use case you have in mind?

transferring large amounts of data and automatization in processing at
least some of it, without involving a 3rd party

"Large amounts" can be "small" like 100MB --- or over 50k files in 12GB,
or even more.  The mirror feature of lftp is extremely useful for such
things.

I wouldn't ever want having to mess around with web pages to figure out
how to do this.  Ftp is plain and simple.  So you see why I'm explicitly
asking for a replacement which is at least as good as ftp.


-- 
"Didn't work" is an error.

Re: [gentoo-user] replacement for ftp?

[Danny YUE]

On 2017-04-26 20:25, R0b0t1  wrote:
> On Tue, Apr 25, 2017 at 8:26 PM, Danny YUE  wrote:
>>
>> On 2017-04-25 19:59, R0b0t1  wrote:
>>> On Tue, Apr 25, 2017 at 10:05 AM, Danny YUE  wrote:

 On 2017-04-25 14:29, lee  wrote:
> Hi,
>
> since the usage of FTP seems to be declining, what is a replacement
> which is at least as good as FTP?
>
> I'm aware that there's webdav, but that's very awkward to use and
> missing features.

 What about sshfs? It allows you to mount a location that can be accessed
 via ssh to your local file system, as if you are using ssh.

>>>
>>> In a similar vein, scp.
>>
>> And considering something still robust but a little more smart, rsync.
>>
>
> I was actually going to come back and suggest rscync over ssh. I
> didn't originally mention it because I typically associate rsync with
> backups.

I recently found that rsnapshot (based on rsync) is a good and solid
tool for backup...You may try it out ;-)

Re: [gentoo-user] replacement for ftp?

[Poison BL.]

On Tue, Apr 25, 2017 at 10:29 AM, lee  wrote:

>
> Hi,
>
> since the usage of FTP seems to be declining, what is a replacement
> which is at least as good as FTP?
>
> I'm aware that there's webdav, but that's very awkward to use and
> missing features.
>
>
> --
> "Didn't work" is an error.
>
>
The one issue I have with all the answers I've seen is that they all lack
the most important question. You're asking for alternatives for an old tool
that was used for many use cases that, these days, have evolved to have
very different requirements for security, integration of access methods,
and general workflows for use. FTP used to be the go-to for long distance
file sharing for *all* use cases, one to one (user managing a website's
content), many to one (upload site), one to many (download site), etc.
What's your use case?

-- 
Poison [BLX]
Joshua M. Murphy

Re: [gentoo-user] replacement for ftp?

[R0b0t1]

On Tue, Apr 25, 2017 at 8:26 PM, Danny YUE  wrote:
>
> On 2017-04-25 19:59, R0b0t1  wrote:
>> On Tue, Apr 25, 2017 at 10:05 AM, Danny YUE  wrote:
>>>
>>> On 2017-04-25 14:29, lee  wrote:
 Hi,

 since the usage of FTP seems to be declining, what is a replacement
 which is at least as good as FTP?

 I'm aware that there's webdav, but that's very awkward to use and
 missing features.
>>>
>>> What about sshfs? It allows you to mount a location that can be accessed
>>> via ssh to your local file system, as if you are using ssh.
>>>
>>
>> In a similar vein, scp.
>
> And considering something still robust but a little more smart, rsync.
>

I was actually going to come back and suggest rscync over ssh. I
didn't originally mention it because I typically associate rsync with
backups.

Re: [gentoo-user] replacement for ftp?

[Danny YUE]

On 2017-04-25 19:59, R0b0t1  wrote:
> On Tue, Apr 25, 2017 at 10:05 AM, Danny YUE  wrote:
>>
>> On 2017-04-25 14:29, lee  wrote:
>>> Hi,
>>>
>>> since the usage of FTP seems to be declining, what is a replacement
>>> which is at least as good as FTP?
>>>
>>> I'm aware that there's webdav, but that's very awkward to use and
>>> missing features.
>>
>> What about sshfs? It allows you to mount a location that can be accessed
>> via ssh to your local file system, as if you are using ssh.
>>
>
> In a similar vein, scp.

And considering something still robust but a little more smart, rsync.

Re: [gentoo-user] replacement for ftp?

[R0b0t1]

On Tue, Apr 25, 2017 at 10:05 AM, Danny YUE  wrote:
>
> On 2017-04-25 14:29, lee  wrote:
>> Hi,
>>
>> since the usage of FTP seems to be declining, what is a replacement
>> which is at least as good as FTP?
>>
>> I'm aware that there's webdav, but that's very awkward to use and
>> missing features.
>
> What about sshfs? It allows you to mount a location that can be accessed
> via ssh to your local file system, as if you are using ssh.
>

In a similar vein, scp.

Re: [gentoo-user] replacement for ftp?

[Danny YUE]

On 2017-04-25 14:29, lee  wrote:
> Hi,
>
> since the usage of FTP seems to be declining, what is a replacement
> which is at least as good as FTP?
>
> I'm aware that there's webdav, but that's very awkward to use and
> missing features.

What about sshfs? It allows you to mount a location that can be accessed
via ssh to your local file system, as if you are using ssh.

Also samba can be a replacement. I have a samba server on my OpenWRT
router and use mount.cifs to mount it...

May these be helpful.

Danny

Re: [gentoo-user] replacement for ftp?

[Mick]

On Tuesday 25 Apr 2017 16:45:37 Alan McKinnon wrote:
> On 25/04/2017 16:29, lee wrote:
> > Hi,
> > 
> > since the usage of FTP seems to be declining, what is a replacement
> > which is at least as good as FTP?
> > 
> > I'm aware that there's webdav, but that's very awkward to use and
> > missing features.
> 
> Why not stick with ftp?
> Or, put another way, why do you feel you need to use something else?
> 
> There's always dropbox


Invariably all web hosting ISPs offer ftp(s) for file upload/download.  If you 
pay a bit more you should be able to get ssh/scp/sftp too.  Indeed, many ISPs 
throw in scp/sftp access as part of their basic package.

Webdav(s) offers the same basic upload/download functionality, so I am not 
sure what you find awkward about it, although I'd rather use lftp instead of 
cadaver any day. ;-)

As Alan mentioned, with JavaScript'ed web pages these days there are many 
webapp'ed ISP offerings like Dropbox and friends.

What is the use case you have in mind?
-- 
Regards,
Mick

signature.asc
Description: This is a digitally signed message part.

Re: [gentoo-user] replacement for ftp?

[Alan McKinnon]

On 25/04/2017 16:29, lee wrote:
> 
> Hi,
> 
> since the usage of FTP seems to be declining, what is a replacement
> which is at least as good as FTP?
> 
> I'm aware that there's webdav, but that's very awkward to use and
> missing features.
> 
> 

Why not stick with ftp?
Or, put another way, why do you feel you need to use something else?

There's always dropbox

-- 
Alan McKinnon
alan.mckin...@gmail.com

[gentoo-user] replacement for ftp?

[lee]

Hi,

since the usage of FTP seems to be declining, what is a replacement
which is at least as good as FTP?

I'm aware that there's webdav, but that's very awkward to use and
missing features.


-- 
"Didn't work" is an error.