The part about how long we've been debugging this is based on my over-simplification. That's the problem with communicating with email. It’s actually a much longer conversation. Not to go off topic but, what we are doing is trying to take Apache FTP Server and give it monetary value. The only way you can do that is to combine it with much more sophisticated features, like a secured virtual file system. Our product is quite large and complex.
But what we've been discussing here is our proof-of-concept server and proxy, which is testing Apache FTP Server and MINA in isolation, as well as Apache VFS. We've been debugging as we build it up. But, we have FINALLY gotten it working. There were many problems. The primary issue was that we had to remove ALL buffering in the socket stream handlers on the proxy. We tried every Java IO handler (readers, writers, buffered input/output streams, Scanner, PrintWriter, Object streams, Data streams), none worked. Direct reads/writes to/from a byte array was the only thing that worked. I think the problem is that buffered IO interferes with reading/writing whole messages, and FTP requires whole messages. FTP does NOT have message boundary markers, only return / newline chars, which is not necessarily a message boundary (see MLSD). Using Wireshark we also found out that TCP was breaking up the MLSD response from the server into multiple messages for reasons beyond our understanding. Buffer size were plenty big so there was no reason for this. We had to alter the IO to compensate. There was some incorrect handling of FTP commands in our Ftplet as well, resulting in the 226 message happening out of order. Consulting the RFCs cleared this up. 226 must be intercepted and tell the proxy to close the data channel. What Apache FTP Server and/or MINA needs is perhaps some overloaded command classes (e.g. PASV, MLSD) to take into account the use of a proxy. That would be the ultimate solution. We may end up creating these. Thanks for your help. Ken -----Original Message----- From: David Latorre <dvl...@gmail.com> Sent: Thursday, August 22, 2019 6:39 PM To: ftpserver-users@mina.apache.org Subject: Re: FTP Data Channels Through Reverse Proxy Hi Kenneth, Of course there are FTP-capable (reverse) proxies out there. Maybe there are no free ones; that´s different... Regarding when to close the data channel, that should be easy: if the FTP server closes the data channel, the proxy server should close it as well... Is there a reason why you can´t do that? It´s been a while since I developed our small proxy but I´m quite sure it was enough for us. You mention you´ve been debugging the issue for 6 weeks but only recently (the last day?) realised that the data channel was not closed by the proxy. I think you need some way to get a better insight on what the proxy is doing. My recommendation is a tcpdump network trace from the Proxy server itself so you can capture the traffic between the FTP server and the proxy, as well as the traffic between the proxy and the client. If I have understood your problem correctly, you think the "unknown internal server error" only happens when there´s a proxy between client and server (although that should be transparent for the server). It´s of course a potential explanation, but in all honesty, I haven´t even seen any evidence that the FTPServer is the one closing the control channel. If the network traces say that it´s definitely the FTPServer deciding to close the connection and the SFTP server logs don´t show anything, I believe you´ll have to debug the FTPServer to see what´s going on. I´m sorry we cannot be more helpful. It was with a previous version of FTPServer, but we have used a proxy to forward FTPS traffic to Mina Ftpserver so I´m hoping you´ll finally get to the bottom of this. Best regards, David El jue., 22 ago. 2019 a las 21:15, Kenneth Taylor (< kenneth.tay...@dataexpress.com>) escribió: > David, > > When you say "a proxy that understands FTP", there is no such proxy. > Remember we are doing reverse proxy, not proxy. And the proxy we are > using is one we wrote. And we do have to tell it when to close the > data channel. There is nothing to magically do that. We have now > changed our server Ftplet.afterCommand() to look for the 226 reply and > to call the proxy to close the data chnnel. So that part is fixed. > > But the client still fails on MLSD response. It does not understand > what is coming from the proxy I guess, but its identical to what's > coming from the server. So the only thing that could be causing the > problem is that Apache is closing the command channel, which we see it > doing, and it should not be doing that. We set all the timeouts to > large values so that’s not it. > > I think we've stumbled upon a bug. We've been debugging this for 6 > weeks so we didn't come to this conclusion lightly. Could there be a > hard coded timeout that we just happen to be tripping because the proxy > imparts a > small delay? I did see in the debugger a write timeout that users don't > have access to and it was set to 60. Our entire test session only > lasts about 3 seconds. > > Our proxy is very simple. Its just a client side socket and a server > side socket with stream copiers for the 4 streams. Nothing magical. > It should be completely transparent. The stream copiers are doing > line-based IO to match FTP with auto-flush on. We know the socket > close logic is at the heart of the problem. Apache is closing the > command socket and I'll bet there is some internal hidden error causing that. > > Thanks. > Ken > > -----Original Message----- > From: David Latorre <dvl...@gmail.com> > Sent: Thursday, August 22, 2019 12:51 PM > To: ftpserver-users@mina.apache.org > Subject: Re: FTP Data Channels Through Reverse Proxy > > From your first line I understand that the problem is with the proxy > that's not closing the data channel, isn't it? > > I guess that for a proxy that understands FTP ,you shouldn't do > anything to ensure the proxy closes the channel when the transfer is done. > > It's not clear from your message but I would think that if the > connection gets closed it will be due to some inactivity timeout being > reached. Other than the idle time, I cannot understand why the control > channel would be closed, I can guarantee it's never happened to us. > > Overriding PASV to provide a different IP address works fine and is > correctly supported. I don't know why you want to capture anything > after MLSD completes. If the data channel is closed the FTP proxy will > close it's own data channel to the client. If it doesn't, you need to > see how to fix the proxy. Why would you try to workaround it for just the > MLSD command? > don't you ever need to download files? > I wouldn't try to find a workaround in any case to be honest. > > > Best regards, > David > > On Thu, 22 Aug 2019, 17:33 Kenneth Taylor, > <kenneth.tay...@dataexpress.com > > > wrote: > > > It looks like the problem is in the closing of the data channel. Its > > happening on the server but not on the proxy so the client hangs > > waiting for that to happen, unless the command channel closes. > > Apache, for some reason, closes the command channel after the "226 > Closing data connection" > > is sent. Why? I found this by preventing the command channel from > > closing on the proxy, and that caused the client to hang after > > receiving the MLSD response. So the previous behavior was > > misleading because Apache closing the command channel was being seen > > by the client which was giving up the session entirely but > > complaining about > the MLSD failing. > > > > So is this a bug in Apache or just a missing piece? How would I > > catch the > > 226 response so I could tell the proxy to close the DC? > > > > We can't use afterCommand() for PASV because that would cause Apache > > to open its own DC. So it seems like we have to do an > > afterCommand() for MLSD, but then we'd have to know this was for a > > DC and close that DC instead of the Command Channel. Ugh! > > > > Thanks. > > Ken > > > > -----Original Message----- > > From: Kenneth Taylor <kenneth.tay...@dataexpress.com> > > Sent: Thursday, August 22, 2019 9:01 AM > > To: ftpserver-users@mina.apache.org > > Subject: RE: FTP Data Channels Through Reverse Proxy > > > > Thanks for the response David. I have used 3 different clients > > (FileZilla, WinSCP and custom) and they all respond the same. I can > > see the MLSD response on the Proxy and that its sending it to the client. > > That's why we are stumped. We cannot see any difference between > > Proxy and No Proxy. But Apache is closing the Data Channel & > > Session after sending the MLSD response, and it should not. > > Connecting direct, the Data Channel does not get closed, but I > > suppose that could be because Apache is detecting some kind of error > > that it is not reporting. I've turned the logs all the way up and don't > > see anything that would help. > > > > A network trace might be a good idea. We will try a wireshark scan. > > > > One thing that just occurred to me is that Apache may be opening > > more than one Data Channel and sending the MLSD response down the > > wrong pipe. Not sure why it would do that since our PASV handler is > > a direct copy of the Apache one (PASV), except with the change of IP > > and > Port in the reponse. > > Our Ftplet returns the ResultType of SKIP in the beforeCommand() for > > PASV which should stop Apache from opening its own data channel, right? > > > > Has Apache ever tested this? I would think putting an FTP Server > > behind a reverse proxy would be VERY common. Our customers demand it. > > > > I will have to look at the FTP RFCs again but I don't remember if > > any messages are supposed to go through the data channel from the > > client to the server. Could it be that Apache is expecting > > something on that stream, even if its just an ACK or disconnect? > > > > Thanks. > > Ken > > > > -----Original Message----- > > From: David Latorre <dvl...@gmail.com> > > Sent: Wednesday, August 21, 2019 6:09 PM > > To: ftpserver-users@mina.apache.org > > Subject: Re: FTP Data Channels Through Reverse Proxy > > > > Ah sorry, > > I was rechecking something and made a mistake regarding the IP > > address in the PASV response. In any case, unless you have proof > > (logs from the server or network traces of the server <-> proxy bit) > > that the server returns invalid data in the proxy case, I think you > > should focus on verifying why the client is failing and, if it is > > caused by an invalid response, why the proxy is generating it. > > > > > > > > > > On Thu, 22 Aug 2019, 00:03 David Latorre, <dvl...@gmail.com> wrote: > > > > > Hi Kenneth, > > > > > > Shouldn't you receive the proxy address as a response to PASV when > > > going through the proxy? Otherwise it would bypass the proxy, but > > > I'm assuming you should not connect directly from the client to > > > the server and that's why you need a proxy. I'm a bit puzzled > > > since you state that the proxy receives the response to MLSD but > > > the IP that the client sees for PASV is the same in both cases. > > > > > > Your client log for the proxy case does not show any listing as a > > > response for MLSD, so it's difficult to know what's going on. If > > > you cannot get any logs for that, get some network traces with > > > tcpdump and check what's going on. Enabling more detailed logging > > > in the client might also help (you can configure that in the > > > Filezilla > settings). > > > Server logs might also prove useful. > > > > > > In any case I don't see how this would be caused by Ftpserver > > > with the information you've given. > > > > > > > > > > > > On Wed, 21 Aug 2019, 21:56 Kenneth Taylor, > > > <kenneth.tay...@dataexpress.com> > > > wrote: > > > > > >> We are trying to get Apache FTP Server to work behind a reverse proxy. > > >> The only thing not working is the data channels. We use an Ftplet > > >> to intercept and handle the PASV command and substitute our > > >> proxy’s IP and port in the PASV response. That all seems to work > > >> fine. We are testing it with FileZilla client and our server > > >> works perfectly when connecting directly to the server but fails > > >> when going through the > > proxy. > > >> > > >> > > >> > > >> We can see the data connections being established on both the > > >> server and the proxy. Our proxy logs every forwarded message and > > >> we can see that all client and server messages are being > > >> forwarded both > ways. > > >> > > >> > > >> > > >> The problem is that after the PASV command happens the next > > >> comand is MLSD and we can see the server sending the correct > > >> response and the proxy is forwarding the listing. But the client > > >> interprets the response incorrectly and closes the session. > > >> After looking at this for many many hours we’re stumped. > > >> > > >> > > >> > > >> Our Proxy is a very simple IO streams forwarder, with a control > > >> connection to tell the Proxy when to open data channels that > > >> forward back to the data channel opened by Apache on the Server. > > >> > > >> > > >> > > >> This is the client log when connecting direct: > > >> > > >> > > >> > > >> 14:10:44 Command: PASV > > >> > > >> 14:10:44 Response: 227 Entering Passive Mode > > >> (192,168,0,173,40,122) > > >> > > >> 14:10:44 Trace: Binding data connection source IP to control > > >> connection source IP 192.168.0.92 > > >> > > >> 14:10:44 Command: MLSD > > >> > > >> 14:10:44 Response: 150 File status okay; about to open data > connection. > > >> > > >> 14:10:44 Response: 226 Closing data connection. > > >> > > >> 14:10:44 Listing: Size=0;Modify=20190814174646.670;Type=dir; > > Monthly > > >> > > >> 14:10:44 Listing: Size=0;Modify=20190814180307.721;Type=dir; > Weekly > > >> > > >> 14:10:44 Listing: Size=0;Modify=20190814174833.915;Type=dir; > Daily > > >> > > >> 14:10:44 Listing: Size=0;Modify=20190814175140.777;Type=dir; > > Archive > > >> > > >> 14:10:44 Status: Directory listing of "/" successful > > >> > > >> > > >> > > >> This is the client log when connecting through the proxy: > > >> > > >> > > >> > > >> 14:37:16 Command: PASV > > >> > > >> 14:37:16 Response: 227 Entering Passive Mode > > >> (192,168,0,91,202,228) > > >> > > >> 14:37:16 Trace: Binding data connection source IP to control > > >> connection source IP 192.168.0.173 > > >> > > >> 14:37:16 Command: MLSD > > >> > > >> 14:37:16 Response: 150 File status okay; about to open data > connection. > > >> > > >> 14:37:16 Response: 226 Closing data connection. > > >> > > >> 14:37:16 Error: Disconnected from server: ECONNABORTED - > > Connection > > >> aborted > > >> > > >> 14:37:16 Error: Failed to retrieve directory listing > > >> > > >> > > >> > > >> One weird thing we see is that the “226 Closing data connection” > > >> shows up on the proxy after the MLSD listing is returned. When > > >> connecting direct it is before the listing in the client log, but > > >> in all cases these messages are coming from the same server so > > >> that may be > > a logging anomaly. > > >> > > >> > > >> > > >> Thanks for any help. > > >> > > >> Ken > > >> > > >> > > >> > > >> Disclaimer: This email from DMBGroup LLC, DMB Consulting Services > > >> LLC, or the personnel associated with either entity (collectively > > >> "*DMB*") and attachments, contain *CONFIDENTIAL, PRIVILEGED AND > > >> PROPRIETARY *information for exclusive use of the addressee > > >> individual(s) or entity. Unauthorized viewing, copying, > > >> disclosure, distribution or use of this e-mail or attachments may > > >> be subject to legal restriction or sanction. If received in > > >> error, notify sender immediately by return e-mail and delete > > >> original message and attachments. Nothing contained in this > > >> e-mail or attachments shall > > satisfy the requirements for a writing unless specifically stated. > > >> Nothing contained herein shall constitute a contract or > > >> electronic signature under the Electronic Signatures in Global > > >> and National Commerce Act, any version of the Uniform Electronic > > >> Transactions Act or any other statute governing electronic transactions. > > >> Opinions and statements expressed in this e-mail and any > > >> attachments are those of the individual sender and not > > >> necessarily of DMB. DMB does not guarantee this e-mail > > >> transmission is secured, > error or virus-free. > > >> Neither DMB nor the sender of this e-mail accepts liability for > > >> errors or omissions in the contents of this e-mail, which arise > > >> as a > > result of e-mail transmission. . > > >> > > > > > > > Disclaimer: This email from DMBGroup LLC, DMB Consulting Services > > LLC, or the personnel associated with either entity (collectively > > "DMB") and attachments, contain CONFIDENTIAL, PRIVILEGED AND > > PROPRIETARY information for exclusive use of the addressee > > individual(s) or entity. Unauthorized viewing, copying, disclosure, > > distribution or use of this e-mail or attachments may be subject to > > legal restriction or sanction. If received in error, notify sender > > immediately by return e-mail and delete original message and > > attachments. Nothing contained in this e-mail or attachments shall > > satisfy the requirements for a > writing unless specifically stated. > > Nothing contained herein shall constitute a contract or electronic > > signature under the Electronic Signatures in Global and National > > Commerce Act, any version of the Uniform Electronic Transactions Act > > or any other statute governing electronic transactions. Opinions and > > statements expressed in this e-mail and any attachments are those of > > the individual sender and not necessarily of DMB. DMB does not > > guarantee this e-mail transmission is secured, error or virus-free. > > Neither DMB nor the sender of this e-mail accepts liability for > > errors or omissions in the contents of this e-mail, which arise as a > > result of > e-mail transmission. . > > > > > > Disclaimer: This email from DMBGroup LLC, DMB Consulting Services > > LLC, or the personnel associated with either entity (collectively > > "DMB") and attachments, contain CONFIDENTIAL, PRIVILEGED AND > > PROPRIETARY information for exclusive use of the addressee > > individual(s) or entity. Unauthorized viewing, copying, disclosure, > > distribution or use of this e-mail or attachments may be subject to > > legal restriction or sanction. If received in error, notify sender > > immediately by return e-mail and delete original message and > > attachments. Nothing contained in this e-mail or attachments shall > > satisfy the requirements for a > writing unless specifically stated. > > Nothing contained herein shall constitute a contract or electronic > > signature under the Electronic Signatures in Global and National > > Commerce Act, any version of the Uniform Electronic Transactions Act > > or any other statute governing electronic transactions. Opinions and > > statements expressed in this e-mail and any attachments are those of > > the individual sender and not necessarily of DMB. DMB does not > > guarantee this e-mail transmission is secured, error or virus-free. > > Neither DMB nor the sender of this e-mail accepts liability for > > errors or omissions in the contents of this e-mail, which arise as a > > result of > e-mail transmission. . > > > > > > Disclaimer: This email from DMBGroup LLC, DMB Consulting Services LLC, > or the personnel associated with either entity (collectively "DMB") > and attachments, contain CONFIDENTIAL, PRIVILEGED AND PROPRIETARY > information for exclusive use of the addressee individual(s) or > entity. Unauthorized viewing, copying, disclosure, distribution or use > of this e-mail or attachments may be subject to legal restriction or > sanction. If received in error, notify sender immediately by return > e-mail and delete original message and attachments. Nothing contained > in this e-mail or attachments shall satisfy the requirements for a writing > unless specifically stated. > Nothing contained herein shall constitute a contract or electronic > signature under the Electronic Signatures in Global and National > Commerce Act, any version of the Uniform Electronic Transactions Act > or any other statute governing electronic transactions. Opinions and > statements expressed in this e-mail and any attachments are those of > the individual sender and not necessarily of DMB. DMB does not > guarantee this e-mail transmission is secured, error or virus-free. > Neither DMB nor the sender of this e-mail accepts liability for errors > or omissions in the contents of this e-mail, which arise as a result of > e-mail transmission. . > > Disclaimer: This email from DMBGroup LLC, DMB Consulting Services LLC, or the personnel associated with either entity (collectively "DMB") and attachments, contain CONFIDENTIAL, PRIVILEGED AND PROPRIETARY information for exclusive use of the addressee individual(s) or entity. Unauthorized viewing, copying, disclosure, distribution or use of this e-mail or attachments may be subject to legal restriction or sanction. If received in error, notify sender immediately by return e-mail and delete original message and attachments. Nothing contained in this e-mail or attachments shall satisfy the requirements for a writing unless specifically stated. Nothing contained herein shall constitute a contract or electronic signature under the Electronic Signatures in Global and National Commerce Act, any version of the Uniform Electronic Transactions Act or any other statute governing electronic transactions. Opinions and statements expressed in this e-mail and any attachments are those of the individual sender and not necessarily of DMB. DMB does not guarantee this e-mail transmission is secured, error or virus-free. Neither DMB nor the sender of this e-mail accepts liability for errors or omissions in the contents of this e-mail, which arise as a result of e-mail transmission. .
