Re: How to ascertain that the connection is secure? Re: Cannot connect via FTPES; lftp hangs
Hello Daniel and all, On Thursday 02 June 2011 12:06:19 am you wrote: You can conveniently set settings to apply only to specific servers, for example set ftp:ssl-force/ftp.example.com on set ftp:use-feat/example.com off set ftp:ssl-force/example.com on This feature is s cool and s undocumented anywhere I looked (man page as well as online lftp tutorials). Also the exact syntax to use on configuration files is not clearly mentioned anywhere. Luckily, I found some forum posts where people had posted their config files, which was helpful. I have added this in the new wiki page: http://linux.overshoot.tv/wiki/networking/lftp#Configuration_files Let me know if I have anything wrong or if I missed something potentially useful. Looks like a misbehaving server. A friendlier server would advertise AUTH TLS in the FEAT reply so that clients connecting know it's supported. That was useful. Apparently, in my case, this was the source of the difficulty. I documented this possibility in the wiki, using the examples from this thread. Earlier in the thread, you wrote: don't even use ftps:// with lftp since that is for implicit ftps, [...] For explicit TLS just open it like ftp://; or you don't even need to specify a protocol since ftp is the default. Using an encrypted control connection when available is also turned on by default in lftp (set ftp:use-feat yes, set ftp:ssl-allow yes). This was very helpful as I was trying to complete the table with the various protocols and their URI schemes. Let me know if you see and glaring mistakes: http://linux.overshoot.tv/wiki/networking/lftp#A_multitude_of_protocols I hope that this table alone will make it easier for future users to understand when to use what. In my case: the server uses FTPES which is really FTPS (explicit) except that I cannot use the FTPS:// URI scheme (used for FTPS implicit) but rather the FTP:// scheme, except that it won't connect securely because the sever won't acknowledge using FTPES in the first place even though it really does! I think that's a nice summary of the whole thread!! LOL! No wonder I was confused. :) I hope to be the last one to be confused on this specific issue. Keep in mind I'm just a fellow user hanging around on this mailing list, my only qualification being a long-time satisified user of lftp. :) That's already a lot. See my blog on the whole experience: http://linux.overshoot.tv/blogs/augustin/best_combination_linux_users_lftp_example There are two additional things to note in regards to using TLS with ftp. First is certificate verification, same as when you'd visit an https web site. It's of little comfort that your password was sent with strong encryption if you sent it to the wrong guy. TLS uses certificates to help ensure you are connected to who you intented to. A basic setup is to make sure certificate verification is turned on (these too are on by default in the current version): set ssl:check-hostname yes set ssl:verify-certificate yes set ssl:ca-file path to your a certificate bundle file, containing the certificate authorities you choose to trust An easy answer to what bundle of certificates authorities to trust is to just take what your browser vendor (eg. Mozilla) or operating system vendor supplies. Then you'll be generally as safe as you'd be accessing https web site in your browser. More paranoid users might hand-pick what certificate authorities to trust on their own. The second important part for ftp with TLS is unique for ftp's peculiarity of using multiple connections, one as a control channel and a separate one for transferring data. lftp by default is set to encrypt only the control channel and leave the data channel in the clear. I find computers and Internet connections plenty fast enough nowadays to afford encrypting everything, so just turn it all on: set ftp:ssl-protect-data yes set ftp:ssl-protect-list yes As you can see from all this, everything is a whole lot simpler if you just connect with sftp to an ssh2 server instead. Everything is always encrypted no matter what, no separate control and data channels to worry about, no certificate authorities to trust (a host fingerprint is verified instead). Thank you for this explanation. It is, at very long last, starting to make sense to me. I couldn't have said it better, so, as per your authorization, I have added it almost verbatim to the wiki. Many, many, many, thanks for your tremendous help. :) Blessings, Augustin. -- Friends: http://www.reuniting.info/ My projects: http://astralcity.org/ http://3enjeux.overshoot.tv/ http://linux.overshoot.tv/ http://overshoot.tv/ http://charityware.info/ http://masquilier.org/ http://openteacher.info/ http://minguo.info/ http://www.wechange.org/ http://searching911.info/ .
Re: How to ascertain that the connection is secure? Re: Cannot connect via FTPES; lftp hangs
On Jun 7, 2011, at 09:59, augustin wrote:
set ftp:use-feat/example.com off
set ftp:ssl-force/example.com on
This feature is s cool and s undocumented anywhere I looked (man page
as well as online lftp tutorials).
It's there in the man page, though perhaps not easy to understand what it's
talking about on first read:
”You can set one variable several times for different closures, and thus you
can get a particular settings for particular state. The closure is to be
specified after variable name separated with slash `/'.
The closure for `dns:', `net:', `ftp:', `http:', `hftp:' domain variables is
currently just the host name as you specify it in the `open' command (with some
exceptions where closure is meaningless, e.g. dns:cache-size). For some `cmd:'
domain variables the closure is current URL without path. For other variables
it is not currently used.
See examples in the sample lftp.conf.“
Also the exact syntax to use on configuration files is not clearly mentioned
anywhere. Luckily, I found some forum posts where people had posted their
config files, which was helpful.
As it said above, that sample system-wide lftp.conf file which is normally
installed as {prefix}/etc/lftp.conf is a good place to start.
Re: How to ascertain that the connection is secure? Re: Cannot connect via FTPES; lftp hangs
On Jun 1, 2011, at 16:44, augustin wrote: With the settings you suggest, I did not really suggest changing any settings, I just noted that those are already the defaults so you shouldn't need to do anything and lftp will automatically use the secure connection if the server tells it that it's supported. how do I know I have a secure connection? You are already on the right track, just turn on ssl-force for at least that server and then lftp will refuse to proceed without encryption. You can conveniently set settings to apply only to specific servers, for example set ftp:ssl-force/ftp.example.com on Then you can just leave that forever in your ~/.lftp/rc file and not have to think about it ever again. Again, the man page does not differentiate between the various protocols, so I am double-plus unclear as to what options are available for which protocol. Actually the settings are all nicely prefixed by the protocol they apply to, or a different prefix to note that it applies in a more generic way to all protocols. It's ftp:ssl-force so it applies to ftp. http:user-agent is for http, and so on. I tried: set ftp:ssl-force true but I get: Login failed: ftp:ssl-force is set and server does not support or allow SSL Looks like a misbehaving server. What's odd is the ftp:use-feat option. The following lines are only present with: set ftp:use-feat true --- FEAT --- 211-Extensions supported: --- EPRT --- IDLE --- MDTM --- SIZE --- REST STREAM --- MLST type*;size*;sizd*;modify*;UNIX.mode*;UNIX.uid*;UNIX.gid*;unique*; --- MLSD --- ESTP --- PASV --- EPSV --- SPSV --- 211 End. --- OPTS MLST type;size;modify;UNIX.mode;UNIX.uid;UNIX.gid; --- 200 MLST OPTS type;size;sizd;modify;UNIX.mode;UNIX.uid;UNIX.gid;unique A friendlier server would advertise AUTH TLS in the FEAT reply so that clients connecting know it's supported. Something like this: --- FEAT --- 211-Features: --- AUTH SSL --- AUTH TLS --- EPRT --- EPSV --- MDTM --- PASV --- PBSZ --- PROT --- REST STREAM --- SIZE --- TVFS --- UTF8 --- 211 End The following lines are only present with: set ftp:use-feat false --- AUTH TLS --- 234 AUTH TLS OK. Good, so it looks like your server supports AUTH TLS despite not admitting to it in the FEAT reply. Just turn use-feat off for that server then. set ftp:use-feat/example.com off set ftp:ssl-force/example.com on When searching the web earlier, I found other people asking very similar questions. There is a lot of confusion surrounding these topics; that's why I am offering, with your help, to clearly document all of this. Keep in mind I'm just a fellow user hanging around on this mailing list, my only qualification being a long-time satisified user of lftp. :) There are two additional things to note in regards to using TLS with ftp. First is certificate verification, same as when you'd visit an https web site. It's of little comfort that your password was sent with strong encryption if you sent it to the wrong guy. TLS uses certificates to help ensure you are connected to who you intented to. A basic setup is to make sure certificate verification is turned on (these too are on by default in the current version): set ssl:check-hostname yes set ssl:verify-certificate yes set ssl:ca-file path to your a certificate bundle file, containing the certificate authorities you choose to trust An easy answer to what bundle of certificates authorities to trust is to just take what your browser vendor (eg. Mozilla) or operating system vendor supplies. Then you'll be generally as safe as you'd be accessing https web site in your browser. More paranoid users might hand-pick what certificate authorities to trust on their own. The second important part for ftp with TLS is unique for ftp's peculiarity of using multiple connections, one as a control channel and a separate one for transferring data. lftp by default is set to encrypt only the control channel and leave the data channel in the clear. I find computers and Internet connections plenty fast enough nowadays to afford encrypting everything, so just turn it all on: set ftp:ssl-protect-data yes set ftp:ssl-protect-list yes As you can see from all this, everything is a whole lot simpler if you just connect with sftp to an ssh2 server instead. Everything is always encrypted no matter what, no separate control and data channels to worry about, no certificate authorities to trust (a host fingerprint is verified instead).
Re: How to ascertain that the connection is secure? Re: Cannot connect via FTPES; lftp hangs
On Thursday 02 June 2011 12:04:07 am you wrote: Good, so it looks like your server supports AUTH TLS despite not admitting to it in the FEAT reply. Just turn use-feat off for that server then. set ftp:use-feat/example.com off set ftp:ssl-force/example.com on Great. That seems to do the trick. :) how do I know I have a secure connection? --- AUTH TLS --- 234 AUTH TLS OK. Is that it? Keep in mind I'm just a fellow user hanging around on this mailing list, my only qualification being a long-time satisified user of lftp. :) I repeat what i just told you privately: I appreciate all the more all the time you spend trying to help me. I am very grateful. http://linux.overshoot.tv/ticket/176#comment-225 I need to go to bed (I'll sleep better tonight ;)). I'll try to digest all the information tomorrow and summarize the important bits in the docs. Thank you Daniel again for all your help so far :) Blessings, Augustin. -- Friends: http://www.reuniting.info/ My projects: http://astralcity.org/ http://3enjeux.overshoot.tv/ http://linux.overshoot.tv/ http://overshoot.tv/ http://charityware.info/ http://masquilier.org/ http://openteacher.info/ http://minguo.info/ http://www.wechange.org/ http://searching911.info/ .
Cannot connect via FTPES; lftp hangs
Hello, As noted here: http://linux.overshoot.tv/ticket/176 lftp hangs at ls: `ls' at 0 [Connecting...] I am trying to securely connect to a server running FTPES: http://ouvaton.coop/spip.php?article376 I tried with filezilla and I can connect. But with lftp, the connection hangs at ls: lftp -d sftp://jack.masquilier@ftp.ocsa-data.net:21 Password: lftp jack.masquilier@ftp.ocsa-data.net:~ ls Running connect program (ssh -a -x -s -l jack.masquilier.org -p 21 ftp.ocsa-data.net sftp) --- sending a packet, length=5, type=1(INIT), id=0 Interrupt Disconnecting lftp jack.masquilier@ftp.ocsa-data.net:~ ls Running connect program (ssh -a -x -s -l jack.masquilier.org -p 21 ftp.ocsa-data.net sftp) --- sending a packet, length=5, type=1(INIT), id=0 `ls' at 0 [Connecting...] Can you test for me? The password is lftptest and the user name and server are those indicated above. Additional information can be found here: http://ubuntuforums.org/showthread.php?t=1768640 With your help, I'd like to properly document the solution and also the way to debug a faulty connection. Thanks, Augustin. -- Friends: http://www.reuniting.info/ My projects: http://astralcity.org/ http://3enjeux.overshoot.tv/ http://linux.overshoot.tv/ http://overshoot.tv/ http://charityware.info/ http://masquilier.org/ http://openteacher.info/ http://minguo.info/ http://www.wechange.org/ http://searching911.info/ .
Re: Cannot connect via FTPES; lftp hangs
On May 31, 2011, at 09:46, augustin wrote: I am trying to securely connect to a server running FTPES: http://ouvaton.coop/spip.php?article376 I tried with filezilla and I can connect. But with lftp, the connection hangs at ls: lftp -d sftp://jack.masquilier@ftp.ocsa-data.net:21 From the screenshots at your link this doesn't look like an ssh2 sftp server. Using port 21 for sftp would be unusual too. You should try connecting as ftps://…
Re: Cannot connect via FTPES; lftp hangs
On Tuesday 31 May 2011 07:09:10 pm Daniel Fazekas wrote: On May 31, 2011, at 09:46, augustin wrote: I am trying to securely connect to a server running FTPES: http://ouvaton.coop/spip.php?article376 I tried with filezilla and I can connect. But with lftp, the connection hangs at ls: lftp -d sftp://jack.masquilier@ftp.ocsa-data.net:21 From the screenshots at your link this doesn't look like an ssh2 sftp server. Using port 21 for sftp would be unusual too. You should try connecting as ftps://… Thank you Daniel. Actually, I am very confused by the profusion of protocols. Using wikipedia, I have started to summarize the different secure protocols supported by lftp. See the table at the bottom of the article: http://linux.overshoot.tv/wiki/networking/lftp The official instructions from my host (in French here: http://ouvaton.coop/spip.php?article376 specifically mention using FTPES - FTP over explicit (TLS/SSL). As noted earlier, when I use filezilla with FTPES - FTP over explicit (TLS/SSL), it works, so I think we can rule out a problem on the server side. Q1) Does lftp support FTPES - FTP over explicit (TLS/SSL)? My understanding is that it does although FTPES is not listed (FTPS is). Q2) How to tell lftp to specifically use FTPES? I tried: 1) ftpes:// $ lftp -d ftpes://jack.masquilier@ftp.ocsa-data.net lftp: ftpes - not supported protocol 2) ftps:// Daniel does well to mention the port 21. One of the two screenshots from my host (link above) has the port empty, the other shows the port 21. With port 21 == Unknown protocol ftp -d ftps://jack.masquilier@ftp.ocsa-data.net:21 Password: lftptest Resolving host address... 1 address found: 194.36.166.14 lftp jack.masquilier@ftp.ocsa-data.net:~ ls Connecting to ftp.ocsa-data.net (194.36.166.14) port 21 SSL_connect: unknown protocol Closing control socket ls: Fatal error: SSL_connect: unknown protocol lftp jack.masquilier@ftp.ocsa-data.net:~ exit Without any port Connection refused. lftp -d ftps://jack.masquilier@ftp.ocsa-data.net Password: lftptest Resolving host address... 1 address found: 194.36.166.14 lftp jack.masquilier@ftp.ocsa-data.net:~ ls Connecting to ftp.ocsa-data.net (194.36.166.14) port 990 Socket error (Connection refused) - reconnecting Closing control socket `ls' at 0 [Delaying before reconnect: 21] 3) sftp:// == hangs at 'ls' [connecting] $ lftp -d sftp://jack.masquilier@ftp.ocsa-data.net Password: lftp jack.masquilier@ftp.ocsa-data.net:~ ls Running connect program (ssh -a -x -s -l jack.masquilier.org ftp.ocsa- data.net sftp) --- sending a packet, length=5, type=1(INIT), id=0 `ls' at 0 [Connecting...] Q3) Given the credentials and password given above, can you try to upload a .txt file via FTPES? I'd like to rule out a router setting (no firewall is set, but I don't know about port forwarding, if anything needs to be set.) Q4) Generally speaking, I have been asking around and searching the web, and I found many, many posts by people asking similar questions, but mostly without any useful answer. My own post at ubuntuforums has remained unanswered... because nobody knows. As noted before, I am documenting things as I go along in the wiki here: http://linux.overshoot.tv/wiki/networking/lftp Any hint is welcome about what ought to be documented (especially with regard to secure connections). Yes, I read the man page. It didn't help to clarify any of the confusions I had about all the various protocols. I'd like to create a more easy understandable tutorial that could be complementary to the man page. Thanks, Augustin. -- Friends: http://www.reuniting.info/ My projects: http://astralcity.org/ http://3enjeux.overshoot.tv/ http://linux.overshoot.tv/ http://overshoot.tv/ http://charityware.info/ http://masquilier.org/ http://openteacher.info/ http://minguo.info/ http://www.wechange.org/ http://searching911.info/ .
