about technical documentation and tutorials - Re: Cannot connect via FTPES; lftp hangs
On Tuesday 07 June 2011 07:32:16 pm you wrote:
> On Jun 7, 2011, at 09:59, augustin wrote:
> >> set ftp:use-feat/example.com off
> >> set ftp:ssl-force/example.com on
> >
> > This feature is s cool and s undocumented anywhere I looked (man
> > page as well as online lftp tutorials).
>
> It's there in the man page, though perhaps not easy to understand what it's
> talking about on first read:
>
> ”You can set one variable several times for different closures, and thus
> you can get a particular settings for particular state. The closure is to
> be specified after variable name separated with slash `/'.
>
> The closure for `dns:', `net:', `ftp:', `http:', `hftp:' domain variables
> is currently just the host name as you specify it in the `open' command
> (with some exceptions where closure is meaningless, e.g. dns:cache-size).
> For some `cmd:' domain variables the closure is current URL without path.
> For other variables it is not currently used. See examples in the sample
> lftp.conf.“
>
> > Also the exact syntax to use on configuration files is not clearly
> > mentioned anywhere. Luckily, I found some forum posts where people had
> > posted their config files, which was helpful.
>
> As it said above, that sample system-wide lftp.conf file which is normally
> installed as {prefix}/etc/lftp.conf is a good place to start.
>
1) Ooops! Thank you very much for correcting me. I have made the necessary
edits to the wiki accordingly.
2) Since I was at it, I added a footnote with a reference to your email, which
I largely quote. I know you told me privately that no credit was needed, but I
still prefer crediting you :)
3) The quote from the man page is both obtuse, and also well hidden. Over the
last couple of weeks, I went many times over the whole man page, scanning for
relevant information to my problem. I think I saw the paragraph early on, but
I certainly didn't understand it and didn't realize how it was relevant to my
problem.
The thing is line 1215 (!!) or so of the man page! There is no heading or
anything, thus it is easily missed. It should be way up, at the top of the man
page, with a bold heading, to make it more visible.
I saw the reference to /etc/lftp.conf in the FILES section near the bottom,
although I almost had a look once, it never occurred to me that it was not an
empty file. I stand corrected. Thanks.
4) I hope nobody gets me wrong. In order to prevent misunderstanding, allow me
to make the following clear:
Nothing I have written on this mailing list was meant as a criticism of the
lftp developer nor of those who wrote the man page. On the contrary. I am very
grateful for their efforts. I am grateful for a nice software, to those who
developed it and to those who provided patches both to the software and the
man page. They did a great job.
And this being the world of Free (Libre) software, I contribute by putting a
tiny cherry on top, in the form of the wiki tutorial.
5) Generally speaking I find man pages hard to understand, at least at first.
They often read as a technical specification document. Tutorials (and wikis)
are completely different beasts. The two complement each other. When I don't
know at all how to use a software, I find tutorial-like documents (wikis) much
easier to understand. But once I understand the basics of the software, a
technical specification-like documents (like man pages) are helpful because of
their comprehensiveness.
6) Ultimately, it's the symbiosis between the different approaches and the
varying contributions that make the Libre Software world great. It ties right
back into the "The best combination" theme that was the main object of my blog
entry I referenced earlier.
Blessings,
Augustin.
--
Friends: http://www.reuniting.info/
My projects:
http://astralcity.org/ http://3enjeux.overshoot.tv/ http://linux.overshoot.tv/
http://overshoot.tv/ http://charityware.info/ http://masquilier.org/
http://openteacher.info/ http://minguo.info/
http://www.wechange.org/ http://searching911.info/
.
Re: How to ascertain that the connection is secure? Re: Cannot connect via FTPES; lftp hangs
On Jun 7, 2011, at 09:59, augustin wrote:
>> set ftp:use-feat/example.com off
>> set ftp:ssl-force/example.com on
> This feature is s cool and s undocumented anywhere I looked (man page
> as well as online lftp tutorials).
It's there in the man page, though perhaps not easy to understand what it's
talking about on first read:
”You can set one variable several times for different closures, and thus you
can get a particular settings for particular state. The closure is to be
specified after variable name separated with slash `/'.
The closure for `dns:', `net:', `ftp:', `http:', `hftp:' domain variables is
currently just the host name as you specify it in the `open' command (with some
exceptions where closure is meaningless, e.g. dns:cache-size). For some `cmd:'
domain variables the closure is current URL without path. For other variables
it is not currently used.
See examples in the sample lftp.conf.“
> Also the exact syntax to use on configuration files is not clearly mentioned
> anywhere. Luckily, I found some forum posts where people had posted their
> config files, which was helpful.
As it said above, that sample system-wide lftp.conf file which is normally
installed as {prefix}/etc/lftp.conf is a good place to start.
Re: How to ascertain that the connection is secure? Re: Cannot connect via FTPES; lftp hangs
Hello Daniel and all, On Thursday 02 June 2011 12:06:19 am you wrote: > You can conveniently set settings to apply only to specific servers, for > example set ftp:ssl-force/ftp.example.com on > set ftp:use-feat/example.com off > set ftp:ssl-force/example.com on This feature is s cool and s undocumented anywhere I looked (man page as well as online lftp tutorials). Also the exact syntax to use on configuration files is not clearly mentioned anywhere. Luckily, I found some forum posts where people had posted their config files, which was helpful. I have added this in the new wiki page: http://linux.overshoot.tv/wiki/networking/lftp#Configuration_files Let me know if I have anything wrong or if I missed something potentially useful. > Looks like a misbehaving server. > > A friendlier server would advertise AUTH TLS in the FEAT reply so that > clients connecting know it's supported. That was useful. Apparently, in my case, this was the source of the difficulty. I documented this possibility in the wiki, using the examples from this thread. Earlier in the thread, you wrote: > don't even use > "ftps://" with lftp since that is for implicit ftps, [...] > For explicit TLS just open it like "ftp://"; or you don't even need to > specify a protocol since ftp is the default. > Using an encrypted control connection when available is also turned on by > default in lftp (set ftp:use-feat yes, set ftp:ssl-allow yes). This was very helpful as I was trying to complete the table with the various protocols and their URI schemes. Let me know if you see and glaring mistakes: http://linux.overshoot.tv/wiki/networking/lftp#A_multitude_of_protocols I hope that this table alone will make it easier for future users to understand when to use what. In my case: the server uses FTPES which is really FTPS (explicit) except that I cannot use the FTPS:// URI scheme (used for FTPS implicit) but rather the FTP:// scheme, except that it won't connect securely because the sever won't acknowledge using FTPES in the first place even though it really does! I think that's a nice summary of the whole thread!! LOL! No wonder I was confused. :) I hope to be the last one to be confused on this specific issue. > Keep in mind I'm just a fellow user hanging around on this mailing list, my > only qualification being a long-time satisified user of lftp. :) That's already a lot. See my blog on the whole experience: http://linux.overshoot.tv/blogs/augustin/best_combination_linux_users_lftp_example > There are two additional things to note in regards to using TLS with ftp. > > First is certificate verification, same as when you'd visit an https web > site. It's of little comfort that your password was sent with strong > encryption if you sent it to the wrong guy. TLS uses certificates to help > ensure you are connected to who you intented to. > > A basic setup is to make sure certificate verification is turned on (these > too are on by default in the current version): set ssl:check-hostname yes > set ssl:verify-certificate yes > set ssl:ca-file "path to your a certificate bundle file, containing the > certificate authorities you choose to trust" > > An easy answer to what bundle of certificates authorities to trust is to > just take what your browser vendor (eg. Mozilla) or operating system > vendor supplies. Then you'll be generally as safe as you'd be accessing > https web site in your browser. More paranoid users might hand-pick what > certificate authorities to trust on their own. > > The second important part for ftp with TLS is unique for ftp's peculiarity > of using multiple connections, one as a control channel and a separate one > for transferring data. > > lftp by default is set to encrypt only the control channel and leave the > data channel in the clear. I find computers and Internet connections > plenty fast enough nowadays to afford encrypting everything, so just turn > it all on: > > set ftp:ssl-protect-data yes > set ftp:ssl-protect-list yes > > As you can see from all this, everything is a whole lot simpler if you just > connect with sftp to an ssh2 server instead. Everything is always > encrypted no matter what, no separate control and data channels to worry > about, no certificate authorities to trust (a host fingerprint is verified > instead). Thank you for this explanation. It is, at very long last, starting to make sense to me. I couldn't have said it better, so, as per your authorization, I have added it almost verbatim to the wiki. Many, many, many, thanks for your tremendous help. :) Blessings, Augustin. -- Friends: http://www.reuniting.info/ My projects: http://astralcity.org/ http://3enjeux.overshoot.tv/ http://linux.overshoot.tv/ http://overshoot.tv/ http://charityware.info/ http://masquilier.org/ http://openteacher.info/ http://minguo.info/ http://www.wechange.org/ http://searching911.info/ .
Re: How to ascertain that the connection is secure? Re: Cannot connect via FTPES; lftp hangs
On Thursday 02 June 2011 12:04:07 am you wrote: > Good, so it looks like your server supports AUTH TLS despite not admitting > to it in the FEAT reply. Just turn use-feat off for that server then. > > set ftp:use-feat/example.com off > set ftp:ssl-force/example.com on Great. That seems to do the trick. :) > > how do I know I have a secure connection? ---> AUTH TLS <--- 234 AUTH TLS OK. Is that it? > Keep in mind I'm just a fellow user hanging around on this mailing list, my > only qualification being a long-time satisified user of lftp. :) I repeat what i just told you privately: I appreciate all the more all the time you spend trying to help me. I am very grateful. http://linux.overshoot.tv/ticket/176#comment-225 I need to go to bed (I'll sleep better tonight ;)). I'll try to digest all the information tomorrow and summarize the important bits in the docs. Thank you Daniel again for all your help so far :) Blessings, Augustin. -- Friends: http://www.reuniting.info/ My projects: http://astralcity.org/ http://3enjeux.overshoot.tv/ http://linux.overshoot.tv/ http://overshoot.tv/ http://charityware.info/ http://masquilier.org/ http://openteacher.info/ http://minguo.info/ http://www.wechange.org/ http://searching911.info/ .
Re: How to ascertain that the connection is secure? Re: Cannot connect via FTPES; lftp hangs
On Jun 1, 2011, at 16:44, augustin wrote: > With the settings you suggest, I did not really suggest changing any settings, I just noted that those are already the defaults so you shouldn't need to do anything and lftp will automatically use the secure connection if the server tells it that it's supported. > how do I know I have a secure connection? You are already on the right track, just turn on ssl-force for at least that server and then lftp will refuse to proceed without encryption. You can conveniently set settings to apply only to specific servers, for example set ftp:ssl-force/ftp.example.com on Then you can just leave that forever in your ~/.lftp/rc file and not have to think about it ever again. > Again, the man page does not differentiate between the various protocols, so > I am double-plus unclear as to what options are available for which protocol. Actually the settings are all nicely prefixed by the protocol they apply to, or a different prefix to note that it applies in a more generic way to all protocols. It's "ftp:ssl-force" so it applies to ftp. "http:user-agent" is for http, and so on. > I tried: > set ftp:ssl-force true > but I get: > "Login failed: ftp:ssl-force is set and server does not support or allow SSL" Looks like a misbehaving server. > What's odd is the ftp:use-feat option. > The following lines are only present with: > set ftp:use-feat true > ---> FEAT > <--- 211-Extensions supported: > <--- EPRT > <--- IDLE > <--- MDTM > <--- SIZE > <--- REST STREAM > <--- MLST type*;size*;sizd*;modify*;UNIX.mode*;UNIX.uid*;UNIX.gid*;unique*; > <--- MLSD > <--- ESTP > <--- PASV > <--- EPSV > <--- SPSV > <--- 211 End. > ---> OPTS MLST type;size;modify;UNIX.mode;UNIX.uid;UNIX.gid; > <--- 200 MLST OPTS type;size;sizd;modify;UNIX.mode;UNIX.uid;UNIX.gid;unique A friendlier server would advertise AUTH TLS in the FEAT reply so that clients connecting know it's supported. Something like this: ---> FEAT <--- 211-Features: <--- AUTH SSL <--- AUTH TLS <--- EPRT <--- EPSV <--- MDTM <--- PASV <--- PBSZ <--- PROT <--- REST STREAM <--- SIZE <--- TVFS <--- UTF8 <--- 211 End > The following lines are only present with: > set ftp:use-feat false > ---> AUTH TLS > <--- 234 AUTH TLS OK. Good, so it looks like your server supports AUTH TLS despite not admitting to it in the FEAT reply. Just turn use-feat off for that server then. set ftp:use-feat/example.com off set ftp:ssl-force/example.com on > When searching the web earlier, I found other people asking very similar > questions. There is a lot of confusion surrounding these topics; that's why I > am offering, with your help, to clearly document all of this. Keep in mind I'm just a fellow user hanging around on this mailing list, my only qualification being a long-time satisified user of lftp. :) There are two additional things to note in regards to using TLS with ftp. First is certificate verification, same as when you'd visit an https web site. It's of little comfort that your password was sent with strong encryption if you sent it to the wrong guy. TLS uses certificates to help ensure you are connected to who you intented to. A basic setup is to make sure certificate verification is turned on (these too are on by default in the current version): set ssl:check-hostname yes set ssl:verify-certificate yes set ssl:ca-file "path to your a certificate bundle file, containing the certificate authorities you choose to trust" An easy answer to what bundle of certificates authorities to trust is to just take what your browser vendor (eg. Mozilla) or operating system vendor supplies. Then you'll be generally as safe as you'd be accessing https web site in your browser. More paranoid users might hand-pick what certificate authorities to trust on their own. The second important part for ftp with TLS is unique for ftp's peculiarity of using multiple connections, one as a control channel and a separate one for transferring data. lftp by default is set to encrypt only the control channel and leave the data channel in the clear. I find computers and Internet connections plenty fast enough nowadays to afford encrypting everything, so just turn it all on: set ftp:ssl-protect-data yes set ftp:ssl-protect-list yes As you can see from all this, everything is a whole lot simpler if you just connect with sftp to an ssh2 server instead. Everything is always encrypted no matter what, no separate control and data channels to worry about, no certificate authorities to trust (a host fingerprint is verified instead).
How to ascertain that the connection is secure? Re: Cannot connect via FTPES; lftp hangs
On Tuesday 31 May 2011 11:47:58 pm Daniel Fazekas wrote: > I too got confused by that name though last time, so don't even use > "ftps://" with lftp since that is for implicit ftps, sorry about the bad > advice. For explicit TLS just open it like "ftp://"; or you don't even need > to specify a protocol since ftp is the default. Using an encrypted control > connection when available is also turned on by default in lftp (set > ftp:use-feat yes, set ftp:ssl-allow yes). > > So you really don't have to do anything but open it normally like > $ lftp jack.masquilier@ftp.ocsa-data.net > > and you should be good to go. Thanks. It works indeed, but I am back to my own starting point. I was trying the ftps:// prefix to force a secure connection (maybe I was too clever for my own good). With the settings you suggest, how do I know I have a secure connection? What tell-tale sign can I look to to ascertain that my connection is secure and that the credentials (username, password) have not been sent in clear over the network? Again, the man page does not differentiate between the various protocols, so I am double-plus unclear as to what options are available for which protocol. You advise: set ftp:ssl-allow true I tried: set ftp:ssl-force true but I get: "Login failed: ftp:ssl-force is set and server does not support or allow SSL" What's odd is the ftp:use-feat option. I tried to set it to on then to off, copying the debug output into 2 text files, and then using diff to spot the differences. The following lines are only present with: set ftp:use-feat true ---> FEAT <--- 211-Extensions supported: <--- EPRT <--- IDLE <--- MDTM <--- SIZE <--- REST STREAM <--- MLST type*;size*;sizd*;modify*;UNIX.mode*;UNIX.uid*;UNIX.gid*;unique*; <--- MLSD <--- ESTP <--- PASV <--- EPSV <--- SPSV <--- 211 End. ---> OPTS MLST type;size;modify;UNIX.mode;UNIX.uid;UNIX.gid; <--- 200 MLST OPTS type;size;sizd;modify;UNIX.mode;UNIX.uid;UNIX.gid;unique The following lines are only present with: set ftp:use-feat false ---> AUTH TLS <--- 234 AUTH TLS OK. Certificate depth: 1; subject: /O=Root CA/OU=http://www.cacert.org/CN=CA Cert Signing Authority/emailAddress=supportatcacert.org; issuer: /O=Root CA/OU=http://www.cacert.org/CN=CA Cert Signing Authority/emailAddress=supportatcacert.org So, what's weird is that AUTH TLS and the certificate details are only present when use-feat is set to FALSE! So, how do I know that the connection is secure? How to prevent unsecure connections with FTPS (explicit) knowing that ftp:ssl-force true does not work (see above)? When searching the web earlier, I found other people asking very similar questions. There is a lot of confusion surrounding these topics; that's why I am offering, with your help, to clearly document all of this. Thanks again, Augustin. -- Friends: http://www.reuniting.info/ My projects: http://astralcity.org/ http://3enjeux.overshoot.tv/ http://linux.overshoot.tv/ http://overshoot.tv/ http://charityware.info/ http://masquilier.org/ http://openteacher.info/ http://minguo.info/ http://www.wechange.org/ http://searching911.info/ .
Re: Cannot connect via FTPES; lftp hangs
On May 31, 2011, at 17:03, augustin wrote: > Actually, I am very confused by the profusion of protocols. [...] > specifically mention using "FTPES - FTP over explicit (TLS/SSL)". It's the first time I heard it referred to as "FTPES," that's probably just a made up name by the FileZilla developers that pretty much nobody else uses. I too got confused by that name though last time, so don't even use "ftps://" with lftp since that is for implicit ftps, sorry about the bad advice. For explicit TLS just open it like "ftp://"; or you don't even need to specify a protocol since ftp is the default. Using an encrypted control connection when available is also turned on by default in lftp (set ftp:use-feat yes, set ftp:ssl-allow yes). So you really don't have to do anything but open it normally like $ lftp jack.masquilier@ftp.ocsa-data.net and you should be good to go.
Re: Cannot connect via FTPES; lftp hangs
On Tuesday 31 May 2011 07:09:10 pm Daniel Fazekas wrote: > On May 31, 2011, at 09:46, augustin wrote: > > I am trying to securely connect to a server running FTPES: > > http://ouvaton.coop/spip.php?article376 > > I tried with filezilla and I can connect. > > But with lftp, the connection hangs at ls: > > lftp -d sftp://jack.masquilier@ftp.ocsa-data.net:21 > > From the screenshots at your link this doesn't look like an ssh2 sftp > server. Using port 21 for sftp would be unusual too. You should try > connecting as ftps://… Thank you Daniel. Actually, I am very confused by the profusion of protocols. Using wikipedia, I have started to summarize the different secure protocols supported by lftp. See the table at the bottom of the article: http://linux.overshoot.tv/wiki/networking/lftp The official instructions from my host (in French here: http://ouvaton.coop/spip.php?article376 specifically mention using "FTPES - FTP over explicit (TLS/SSL)". As noted earlier, when I use filezilla with "FTPES - FTP over explicit (TLS/SSL)", it works, so I think we can rule out a problem on the server side. Q1) Does lftp support "FTPES - FTP over explicit (TLS/SSL)"? My understanding is that it does although FTPES is not listed (FTPS is). Q2) How to tell lftp to specifically use FTPES? I tried: 1) ftpes:// $ lftp -d ftpes://jack.masquilier@ftp.ocsa-data.net lftp: ftpes - not supported protocol 2) ftps:// Daniel does well to mention the port 21. One of the two screenshots from my host (link above) has the port empty, the other shows the port 21. With port 21 ==> Unknown protocol ftp -d ftps://jack.masquilier@ftp.ocsa-data.net:21 Password: lftptest Resolving host address... 1 address found: 194.36.166.14 lftp jack.masquilier@ftp.ocsa-data.net:~> ls Connecting to ftp.ocsa-data.net (194.36.166.14) port 21 SSL_connect: unknown protocol Closing control socket ls: Fatal error: SSL_connect: unknown protocol lftp jack.masquilier@ftp.ocsa-data.net:~> exit Without any port > Connection refused. lftp -d ftps://jack.masquilier@ftp.ocsa-data.net Password: lftptest Resolving host address... 1 address found: 194.36.166.14 lftp jack.masquilier@ftp.ocsa-data.net:~> ls Connecting to ftp.ocsa-data.net (194.36.166.14) port 990 Socket error (Connection refused) - reconnecting Closing control socket `ls' at 0 [Delaying before reconnect: 21] 3) sftp:// ==> hangs at 'ls' [connecting] $ lftp -d sftp://jack.masquilier@ftp.ocsa-data.net Password: lftp jack.masquilier@ftp.ocsa-data.net:~> ls Running connect program (ssh -a -x -s -l jack.masquilier.org ftp.ocsa- data.net sftp) ---> sending a packet, length=5, type=1(INIT), id=0 `ls' at 0 [Connecting...] Q3) Given the credentials and password given above, can you try to upload a .txt file via FTPES? I'd like to rule out a router setting (no firewall is set, but I don't know about port forwarding, if anything needs to be set.) Q4) Generally speaking, I have been asking around and searching the web, and I found many, many posts by people asking similar questions, but mostly without any useful answer. My own post at ubuntuforums has remained unanswered... because nobody knows. As noted before, I am documenting things as I go along in the wiki here: http://linux.overshoot.tv/wiki/networking/lftp Any hint is welcome about what ought to be documented (especially with regard to secure connections). Yes, I read the man page. It didn't help to clarify any of the confusions I had about all the various protocols. I'd like to create a more easy understandable tutorial that could be complementary to the man page. Thanks, Augustin. -- Friends: http://www.reuniting.info/ My projects: http://astralcity.org/ http://3enjeux.overshoot.tv/ http://linux.overshoot.tv/ http://overshoot.tv/ http://charityware.info/ http://masquilier.org/ http://openteacher.info/ http://minguo.info/ http://www.wechange.org/ http://searching911.info/ .
Re: Cannot connect via FTPES; lftp hangs
On May 31, 2011, at 09:46, augustin wrote: > I am trying to securely connect to a server running FTPES: > http://ouvaton.coop/spip.php?article376 > I tried with filezilla and I can connect. > But with lftp, the connection hangs at ls: > lftp -d sftp://jack.masquilier@ftp.ocsa-data.net:21 >From the screenshots at your link this doesn't look like an ssh2 sftp server. >Using port 21 for sftp would be unusual too. You should try connecting as ftps://…
Cannot connect via FTPES; lftp hangs
Hello, As noted here: http://linux.overshoot.tv/ticket/176 lftp hangs at ls: `ls' at 0 [Connecting...] I am trying to securely connect to a server running FTPES: http://ouvaton.coop/spip.php?article376 I tried with filezilla and I can connect. But with lftp, the connection hangs at ls: lftp -d sftp://jack.masquilier@ftp.ocsa-data.net:21 Password: lftp jack.masquilier@ftp.ocsa-data.net:~> ls Running connect program (ssh -a -x -s -l jack.masquilier.org -p 21 ftp.ocsa-data.net sftp) ---> sending a packet, length=5, type=1(INIT), id=0 Interrupt Disconnecting lftp jack.masquilier@ftp.ocsa-data.net:~> ls Running connect program (ssh -a -x -s -l jack.masquilier.org -p 21 ftp.ocsa-data.net sftp) ---> sending a packet, length=5, type=1(INIT), id=0 `ls' at 0 [Connecting...] Can you test for me? The password is lftptest and the user name and server are those indicated above. Additional information can be found here: http://ubuntuforums.org/showthread.php?t=1768640 With your help, I'd like to properly document the solution and also the way to debug a faulty connection. Thanks, Augustin. -- Friends: http://www.reuniting.info/ My projects: http://astralcity.org/ http://3enjeux.overshoot.tv/ http://linux.overshoot.tv/ http://overshoot.tv/ http://charityware.info/ http://masquilier.org/ http://openteacher.info/ http://minguo.info/ http://www.wechange.org/ http://searching911.info/ .
