On Mon, Jun 21, 2010 at 10:22 AM, Bengt Rodehav <[email protected]> wrote: > Claus, > > Just tried and it works. Very simple way of achieving our goals I must say. > Two questions: >
It was just a quick test to see if that resolved it here and now as a workaround. > 1. How does the consumer side work? I haven't tried it yet but wouldn't it > have the same problems? > Yes there is only one instance of the consumer and its not pooled. So it may have the issue of not being able to re-connect due Apache FTP issue. So we may have to implement a better re-connect logic in camel-ftp. Maybe re-creating the FTPClient which works > 2. Regarding the option you would introduce to control this behaviour, will > it default to "no-caching" for ftps? I guess it must, given the problems in > FTPSClient. > Its a bit confusing if some is pooled and others are not. But I do also think its best that it just works out of the box. We should add a WARN / INFO etc to the wiki page so its easy to spot. But maybe we should fix the re-connect logic and then the producer could still be pooled. > /Bengt > > 2010/6/21 Claus Ibsen <[email protected]> > >> On Mon, Jun 21, 2010 at 9:11 AM, Bengt Rodehav <[email protected]> wrote: >> > OK >> > >> >> Bengt could you try changing camel-ftp in the RemoteFileProducer >> public class RemoteFileProducer<T> extends GenericFileProducer<T> >> implements ServicePoolAware { >> And remove the implements ServicePoolAware which should cause Camel to >> not pool it and thus you get a new fresh instance each time. >> >> Then try testing with that to see if that fixes the re-connect issue. >> >> What we can then do is to add some nice option in the endpoint URI so >> you can configure the pooling behavior. >> >> >> > 2010/6/21 Claus Ibsen <[email protected]> >> > >> >> On Mon, Jun 21, 2010 at 8:52 AM, Bengt Rodehav <[email protected]> >> wrote: >> >> > I have now created a JIRA in commons-net for this issue: >> >> > >> >> > https://issues.apache.org/jira/browse/NET-327 >> >> > >> >> >> >> Good >> >> >> >> > <https://issues.apache.org/jira/browse/NET-327>However, I still think >> we >> >> > must fix this in camel-ftp by some kind of work-around (maybe similar >> to >> >> the >> >> > one I proposed in >> https://issues.apache.org/activemq/browse/CAMEL-2829). >> >> > Have you had a chance to look at it Claus? >> >> > >> >> >> >> No sorry I am to busy with other issues but we got the ticket so we >> >> will look into it before 2.4 is cut. >> >> >> >> >> >> > /Bengt >> >> > >> >> > 2010/6/18 Bengt Rodehav <[email protected]> >> >> > >> >> >> JIRA created: >> >> >> >> >> >> https://issues.apache.org/activemq/browse/CAMEL-2829 >> >> >> >> >> >> <https://issues.apache.org/activemq/browse/CAMEL-2829>/Bengt >> >> >> >> >> >> 2010/6/18 Bengt Rodehav <[email protected]> >> >> >> >> >> >> Claus, >> >> >>> >> >> >>> I'm not sure what you mean by: >> >> >>> >> >> >>> "I think we should add an option on the endpoint to allow people to >> >> >>> turn this on/off. >> >> >>> Some would like to reuse existing connections to avoid the connect >> -> >> >> >>> upload -> disconnect cycle when they upload many files etc." >> >> >>> >> >> >>> IMO we would not disconnect more often than today. We would just >> change >> >> >>> the connect() method in FtpsOperations so that it creates a new >> >> instance of >> >> >>> FTPSClient prior to a connection attempt. I assume that since the >> >> connect() >> >> >>> method is called in the first place, we are not currently connected >> and >> >> >>> there is no way to reuse an existing connection. Am I right? >> >> >>> >> >> >>> I have made some modifications in camel-ftp along these lines. I >> will >> >> >>> create a JIRA ticket for this bug and attach my changes as diff >> files >> >> to the >> >> >>> issue. I just tested my changes and they do solve the problem I have >> >> >>> encountered but I haven't tested whether I have introduced any new >> >> problems >> >> >>> or not. >> >> >>> >> >> >>> Anyway, this is what I did: >> >> >>> >> >> >>> *FtpsEndpoint* >> >> >>> I added the following members to keep track of the configuration of >> the >> >> >>> FTPSClient: >> >> >>> >> >> >>> private boolean isNeedClientAuth = false; >> >> >>> private KeyManager keyManager; >> >> >>> private TrustManager trustManager; >> >> >>> >> >> >>> I modified the createFtpClient() method so that it doesn't actually >> >> create >> >> >>> a FTPSClient but only calculates and stores the configuration in the >> >> above >> >> >>> members. I also renamed it to createFtpsConfiguration() to reflect >> >> that. >> >> >>> >> >> >>> The createRemoteFileOperations() method was also changed >> accordingly. >> >> It >> >> >>> no longer interacts directly with an FTPSClient (since it is not >> >> created >> >> >>> yet). >> >> >>> >> >> >>> *FtpsOperations* >> >> >>> I added the private createFtpsClient() method that takes care of >> >> creating >> >> >>> an FTPSClient from configuration stored in the FtpsEndpoint. The >> >> >>> createFtpsClient() is called in the connect() method just before the >> >> call to >> >> >>> the "real" connect() method in the superclass (FtpOperations). >> >> >>> >> >> >>> I changed the constructor so that it no longer takes a client as an >> >> >>> argument. I send null as client to the super class constructor. This >> is >> >> of >> >> >>> course a major change in behavior that you need to take a look at >> it. >> >> >>> Neither FtpsOperations nor its super class FtpOperations can no >> longer >> >> >>> assume that a client exists until the first connection attempt has >> been >> >> >>> made. >> >> >>> >> >> >>> *FtpOperations* >> >> >>> I had to make the "client" member non-final since I now set it on >> every >> >> >>> call to the connect() method in FtpsOperations. >> >> >>> >> >> >>> >> >> >>> BTW, I agree with you that the SFTPClient in commons-net is flawed. >> It >> >> >>> should provide a way to re-initialize itself so that a new connect() >> >> could >> >> >>> be made. The FTPClient seems to work that way and it also provides a >> >> >>> disconnect() method. If they do not intend to support this, then >> they >> >> should >> >> >>> at least state, as part of the documentation, that the SFTPClient >> >> cannot be >> >> >>> reused across multiple connections. I've posted about this on their >> >> mailing >> >> >>> list but hasn't got any reply yet. Meanwhile the above can be >> regarded >> >> as a >> >> >>> (complicated...) work-around. >> >> >>> >> >> >>> /Bengt >> >> >>> >> >> >>> >> >> >>> 2010/6/18 Claus Ibsen <[email protected]> >> >> >>> >> >> >>> On Thu, Jun 17, 2010 at 11:56 PM, Bengt Rodehav <[email protected]> >> >> >>>> wrote: >> >> >>>> > Claus, >> >> >>>> > >> >> >>>> > Having looked a bit more at the commons-net code it's becoming >> clear >> >> to >> >> >>>> me >> >> >>>> > that you probably can't just reconnect an FTPSClient once it has >> >> >>>> entered >> >> >>>> > secure communication mode. I can't find a way to re-initialise an >> >> >>>> > FTPSClient. I tried the disconnect() method (in the super class >> >> >>>> FTPClient) >> >> >>>> > but it didn't get the FTPSClient out of secure communication >> mode. >> >> >>>> > >> >> >>>> > To be safe I think we should always start with a newly >> instantiated >> >> >>>> > FTPSClient when we try to connect. Today the FTPSClient is >> created >> >> by >> >> >>>> the >> >> >>>> > FtpsEndpoint. I don't think it has to. It could just get all the >> >> righ >> >> >>>> > configuration parameters and pass it to FtpsOperations which >> would >> >> then >> >> >>>> > instantiate a new FTPSClient at every connect attempt. >> >> >>>> > >> >> >>>> > If we do it this way for ftps I guess it would make sense with >> the >> >> same >> >> >>>> > setup for plain ftp. >> >> >>>> > >> >> >>>> > What do you think? >> >> >>>> > >> >> >>>> >> >> >>>> I think we should add an option on the endpoint to allow people to >> >> >>>> turn this on/off. >> >> >>>> Some would like to reuse existing connections to avoid the connect >> -> >> >> >>>> upload -> disconnect cycle when they upload many files etc. >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> And frankly I can't see the problem why a secure connection cannot >> >> >>>> self heal and be able to re-establish itself. After all FTPSClient >> got >> >> >>>> all the connection information. And hence I would consider it a >> >> >>>> missing feature / glitch in the FTP library. >> >> >>>> >> >> >>>> Imagine if using secure connections for JMS brokers which couldn't >> >> >>>> self heal just because a network outage. That would cause a lot of >> >> >>>> pain for clients having to re-start. >> >> >>>> >> >> >>>> >> >> >>>> > /Bengt >> >> >>>> > >> >> >>>> > 2010/6/17 Bengt Rodehav <[email protected]> >> >> >>>> > >> >> >>>> >> Claus, >> >> >>>> >> >> >> >>>> >> Yeah, maybe this is a commons-net issue. As always I find it >> >> >>>> surprising >> >> >>>> >> that no one else has encountered this problem. >> >> >>>> >> >> >> >>>> >> As I started writing an email on the commons user list, I took >> >> another >> >> >>>> look >> >> >>>> >> at the code in commons-net. It turns out that the >> FTPSSocketFactory >> >> >>>> class >> >> >>>> >> actually does override the createSocket() methods in >> SocketFactory. >> >> A >> >> >>>> >> similar work-around that I was talking about but on the factory >> >> class >> >> >>>> >> instead of in the SocketClient class. FTPSSocketFactory >> delegates >> >> the >> >> >>>> >> createSocket() calls to its SSLContect's socket factory. >> >> >>>> >> >> >> >>>> >> So why then are we initially able to connect but not after we >> have >> >> >>>> called >> >> >>>> >> execProt() and thus changed the connection factory? >> >> >>>> >> >> >> >>>> >> Moving back to the FTPSClient class now. When the initial >> >> connection >> >> >>>> is up, >> >> >>>> >> the _connectAction() method is called. It in turn calls >> >> >>>> sslNegotiation() >> >> >>>> >> which in turn sets up the secure socket. In the sslNegotiation() >> >> >>>> method, an >> >> >>>> >> SSLSocketFactory is not instantiated directly. It is given by >> the >> >> >>>> >> SSLContext's getSocketFactory() method. This connection factory >> >> >>>> obviously >> >> >>>> >> works otherwise the secure connection wouldn't work even >> initially. >> >> >>>> >> >> >> >>>> >> Compare this with what's being done in the execProt() method. >> Here >> >> the >> >> >>>> >> FTPSSocketFactory is instantiated directly with the SSLContext >> >> passed >> >> >>>> as a >> >> >>>> >> constructor argument. Obviously this factory does NOT work... >> >> >>>> >> >> >> >>>> >> Another question that comes to mind is why the socket factory >> >> created >> >> >>>> in >> >> >>>> >> the sslNegotiation() method is NOT set as the SocketClient's >> socket >> >> >>>> factory >> >> >>>> >> but the one created in the execProt() method is? It turns out >> that >> >> >>>> >> execProt() calls a generic sendCommand() method that sends the >> >> command >> >> >>>> and >> >> >>>> >> if the reply is OK, then sets the socket factory to null! One >> >> wonders >> >> >>>> why >> >> >>>> >> (and there is also a comment in the code: "Check this - is this >> >> >>>> necessary at >> >> >>>> >> all?"). Anyway, this is probably the reason why execProt() needs >> to >> >> >>>> "reset" >> >> >>>> >> the connection factory. >> >> >>>> >> >> >> >>>> >> Still sounds like a commons-net problem though. I'll try to post >> >> this >> >> >>>> at >> >> >>>> >> their user mailing list. >> >> >>>> >> >> >> >>>> >> Just one thought. Do we need to reuse the FTPSClient instance? >> >> Can't >> >> >>>> we >> >> >>>> >> just create a fresh new instance everytime we need to connect? >> >> >>>> >> >> >> >>>> >> /Bengt >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> >> >> >>>> >> 2010/6/17 Claus Ibsen <[email protected]> >> >> >>>> >> >> >> >>>> >> Hi Bengt >> >> >>>> >>> >> >> >>>> >>> I think the issue should be created as a ticket for Apache >> Commons >> >> >>>> >>> Net. Then at least the people there can take a look, and maybe >> >> they >> >> >>>> >>> got some ideas. And can fix it in a future release. >> >> >>>> >>> >> >> >>>> >>> Bengt fell free to experiment yourself with the subclassing, as >> it >> >> >>>> >>> sounds like a good workaround. >> >> >>>> >>> >> >> >>>> >>> I assume the FTPSClient doesn't offer other methods to re >> connect >> >> >>>> >>> which can recover this error? >> >> >>>> >>> >> >> >>>> >>> >> >> >>>> >>> On Wed, Jun 16, 2010 at 2:44 PM, Bengt Rodehav < >> [email protected] >> >> > >> >> >>>> wrote: >> >> >>>> >>> > I have mentioned this problem in a previous conversation but >> >> after >> >> >>>> >>> > investigating the subject further I decided to start a new >> >> thread >> >> >>>> for >> >> >>>> >>> this. >> >> >>>> >>> > >> >> >>>> >>> > If some kind of problem is encountered while using ftps (I >> >> tested >> >> >>>> this >> >> >>>> >>> using >> >> >>>> >>> > Filezilla server by copying a file that already existed which >> >> >>>> Filezilla >> >> >>>> >>> does >> >> >>>> >>> > not allow), then at the next attempt to connect to the ftps >> >> server, >> >> >>>> I >> >> >>>> >>> get >> >> >>>> >>> > the following exception in my logfile: >> >> >>>> >>> > >> >> >>>> >>> > 13:03:27,606 | ERROR | %7Bfile%3Aext%7D | >> >> GenericFileOnCompletion >> >> >>>> >>> | >> >> >>>> >>> > rg.apache.camel.processor.Logger 248 | Caused by: >> >> >>>> >>> > >> >> >>>> >> [org.apache.camel.component.file.GenericFileOperationFailedException - >> >> >>>> >>> File >> >> >>>> >>> > operation failed: null Unconnected sockets not implemented. >> >> Code: >> >> >>>> 221] >> >> >>>> >>> > >> >> >>>> >> org.apache.camel.component.file.GenericFileOperationFailedException: >> >> >>>> >>> File >> >> >>>> >>> > operation failed: null Unconnected sockets not implemented. >> >> Code: >> >> >>>> 221 >> >> >>>> >>> > at >> >> >>>> >>> > >> >> >>>> >>> >> >> >>>> >> >> >> org.apache.camel.component.file.remote.FtpOperations.connect(FtpOperations.java:108)[76:org.apache.camel.camel-ftp:2.4.0.SNAPSHOT] >> >> >>>> >>> > at >> >> >>>> >>> > >> >> >>>> >>> >> >> >>>> >> >> >> org.apache.camel.component.file.remote.FtpsOperations.connect(FtpsOperations.java:40)[76:org.apache.camel.camel-ftp:2.4.0.SNAPSHOT] >> >> >>>> >>> > at >> >> >>>> >>> > >> >> >>>> >>> >> >> >>>> >> >> >> org.apache.camel.component.file.remote.RemoteFileProducer.connectIfNecessary(RemoteFileProducer.java:170)[76:org.apache.camel.camel-ftp:2.4.0.SNAPSHOT] >> >> >>>> >>> > at >> >> >>>> >>> > >> >> >>>> >>> >> >> >>>> >> >> >> org.apache.camel.component.file.remote.RemoteFileProducer.preWriteCheck(RemoteFileProducer.java:123)[76:org.apache.camel.camel-ftp:2.4.0.SNAPSHOT] >> >> >>>> >>> > at >> >> >>>> >>> > >> >> >>>> >>> >> >> >>>> >> >> >> org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:75)[65:org.apache.camel.camel-core:2.4.0.SNAPSHOT] >> >> >>>> >>> > at >> >> >>>> >>> > >> >> >>>> >>> >> >> >>>> >> >> >> org.apache.camel.component.file.remote.RemoteFileProducer.process(RemoteFileProducer.java:49)[76:org.apache.camel.camel-ftp:2.4.0.SNAPSHOT] >> >> >>>> >>> > at >> >> >>>> >>> > >> >> >>>> >>> >> >> >>>> >> >> >> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:91)[65:org.apache.camel.camel-core:2.4.0.SNAPSHOT] >> >> >>>> >>> > >> >> >>>> >>> > I added some logging to see the actual stack trace which is: >> >> >>>> >>> > >> >> >>>> >>> > java.net.SocketException: Unconnected sockets not implemented >> >> >>>> >>> > at >> >> >>>> javax.net.SocketFactory.createSocket(SocketFactory.java:104) >> >> >>>> >>> > at >> >> >>>> >>> > >> >> org.apache.commons.net.SocketClient.connect(SocketClient.java:175) >> >> >>>> >>> > at >> >> >>>> >>> > >> >> >>>> >>> >> >> >>>> >> >> >> org.apache.camel.component.file.remote.FtpOperations.connect(FtpOperations.java:91) >> >> >>>> >>> > at >> >> >>>> >>> > >> >> >>>> >>> >> >> >>>> >> >> >> org.apache.camel.component.file.remote.FtpsOperations.connect(FtpsOperations.java:40) >> >> >>>> >>> > at >> >> >>>> >>> > >> >> >>>> >>> >> >> >>>> >> >> >> org.apache.camel.component.file.remote.RemoteFileProducer.connectIfNecessary(RemoteFileProducer.java:170) >> >> >>>> >>> > at >> >> >>>> >>> > >> >> >>>> >>> >> >> >>>> >> >> >> org.apache.camel.component.file.remote.RemoteFileProducer.preWriteCheck(RemoteFileProducer.java:99) >> >> >>>> >>> > at >> >> >>>> >>> > >> >> >>>> >>> >> >> >>>> >> >> >> org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:75) >> >> >>>> >>> > at >> >> >>>> >>> > >> >> >>>> >>> >> >> >>>> >> >> >> org.apache.camel.component.file.remote.RemoteFileProducer.process(RemoteFileProducer.java:49) >> >> >>>> >>> > >> >> >>>> >>> > commons-net SocketClient class tries to call the socket >> >> factory's >> >> >>>> >>> > createSocket() method (with no parameters): >> >> >>>> >>> > >> >> >>>> >>> > _socket_= _socketFactory_.createSocket(); >> >> >>>> >>> > _socket_.connect(new InetSocketAddress(hostname, >> port), >> >> >>>> >>> > connectTimeout); >> >> >>>> >>> > >> >> >>>> >>> > Then intention is to create an "unconnected socket" (hence >> the >> >> >>>> error >> >> >>>> >>> > message) and then subsequently connect it. In my case, the >> >> >>>> connection >> >> >>>> >>> > factory being used is an FTPSSocketFactory >> >> >>>> >>> > (package org.apache.commons.net.ftp) that inherits from >> >> >>>> >>> > SocketFactory(package javax.net). The SocketFactory class >> >> >>>> implements >> >> >>>> >>> the >> >> >>>> >>> > createSocket() method but throws the exception ("Unconnected >> >> >>>> sockets not >> >> >>>> >>> > implemented"). >> >> >>>> >>> > >> >> >>>> >>> > The reason why this happens is that after having executed the >> >> >>>> >>> > FTPSClient.execPROT() method (package >> >> org.apache.commons.net.ftp), >> >> >>>> with >> >> >>>> >>> any >> >> >>>> >>> > other parameter than "C" (in my case I used "P"), the socket >> >> >>>> factory is >> >> >>>> >>> set >> >> >>>> >>> > to an FTPSSocketFactory as follows: >> >> >>>> >>> > >> >> >>>> >>> > setSocketFactory(new FTPSSocketFactory(context)); >> >> >>>> >>> > >> >> >>>> >>> > After this, no connection attempt will succeed. This problem >> has >> >> >>>> been >> >> >>>> >>> > introduced recently when Claus (on my initiative) added >> support >> >> for >> >> >>>> >>> secure >> >> >>>> >>> > data channel in ftps. I'm thus to blame... >> >> >>>> >>> > >> >> >>>> >>> > I'm not sure how this should be fixed. >> >> >>>> >>> > >> >> >>>> >>> > It's a bit strange that all the connect methods >> >> >>>> >>> > in org.apache.commons.net.SocketClient (which is the ultimate >> >> base >> >> >>>> class >> >> >>>> >>> of >> >> >>>> >>> > FTPSClient) always try to create an unconnected socket first >> and >> >> >>>> then >> >> >>>> >>> > connects it. If commons-net uses this pattern, then it should >> >> make >> >> >>>> sure >> >> >>>> >>> that >> >> >>>> >>> > all its connection factor's support unconnected sockets but >> the >> >> >>>> >>> > FTPSSocketFactory dosn't. This sounds like a bug in >> commons-net. >> >> >>>> >>> > >> >> >>>> >>> > Knowing that commons-net doesn't release very frequently I >> think >> >> we >> >> >>>> need >> >> >>>> >>> a >> >> >>>> >>> > workaround for this. I guess it would be possible to subclass >> >> the >> >> >>>> >>> FTPClient >> >> >>>> >>> > class and override all the connect methods to make sure that >> no >> >> >>>> attempt >> >> >>>> >>> is >> >> >>>> >>> > made to create unconnected sockets. We would then change the >> >> >>>> >>> > createFtpClient() method in the FtpsEndpoint class to >> >> instantiate >> >> >>>> our >> >> >>>> >>> own >> >> >>>> >>> > subclass instead of an FTPSClient. Doesn't sound like a nice >> >> clean >> >> >>>> >>> solution >> >> >>>> >>> > though. >> >> >>>> >>> > >> >> >>>> >>> > Any ideas? >> >> >>>> >>> > >> >> >>>> >>> > /Bengt >> >> >>>> >>> > >> >> >>>> >>> >> >> >>>> >>> >> >> >>>> >>> >> >> >>>> >>> -- >> >> >>>> >>> Claus Ibsen >> >> >>>> >>> Apache Camel Committer >> >> >>>> >>> >> >> >>>> >>> Author of Camel in Action: http://www.manning.com/ibsen/ >> >> >>>> >>> Open Source Integration: http://fusesource.com >> >> >>>> >>> Blog: http://davsclaus.blogspot.com/ >> >> >>>> >>> Twitter: http://twitter.com/davsclaus >> >> >>>> >>> >> >> >>>> >> >> >> >>>> >> >> >> >>>> > >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> -- >> >> >>>> Claus Ibsen >> >> >>>> Apache Camel Committer >> >> >>>> >> >> >>>> Author of Camel in Action: http://www.manning.com/ibsen/ >> >> >>>> Open Source Integration: http://fusesource.com >> >> >>>> Blog: http://davsclaus.blogspot.com/ >> >> >>>> Twitter: http://twitter.com/davsclaus >> >> >>>> >> >> >>> >> >> >>> >> >> >> >> >> > >> >> >> >> >> >> >> >> -- >> >> Claus Ibsen >> >> Apache Camel Committer >> >> >> >> Author of Camel in Action: http://www.manning.com/ibsen/ >> >> Open Source Integration: http://fusesource.com >> >> Blog: http://davsclaus.blogspot.com/ >> >> Twitter: http://twitter.com/davsclaus >> >> >> > >> >> >> >> -- >> Claus Ibsen >> Apache Camel Committer >> >> Author of Camel in Action: http://www.manning.com/ibsen/ >> Open Source Integration: http://fusesource.com >> Blog: http://davsclaus.blogspot.com/ >> Twitter: http://twitter.com/davsclaus >> > -- Claus Ibsen Apache Camel Committer Author of Camel in Action: http://www.manning.com/ibsen/ Open Source Integration: http://fusesource.com Blog: http://davsclaus.blogspot.com/ Twitter: http://twitter.com/davsclaus
