On Fri, Jun 29, 2012 at 5:52 PM, David Capwell <dcapw...@gmail.com> wrote: > I have added idempotent=true and this seems to work the way i wont but > doesn't persist. So to make sure I don't download the same file even after > reboot the best option is the filter, correct? If so what am I missing > from getting my code to work? >
There are details about persistent idempotent repositories in the Camel docs. http://camel.apache.org/file2.html And the EIP has some details as well http://camel.apache.org/idempotent-consumer.html Its just that the file/ftp component has baked in idempotent consumer, so you do not need to use .idempotentConsumer in the route. This is an optimization to avoid consuming the file if really not needed. An alternative is as you said a custom filter, where you can return true|false, whether to consume the file or not. > thanks for all the help i have been given so far! > > On Fri, Jun 29, 2012 at 8:45 AM, David Capwell <dcapw...@gmail.com> wrote: > >> Sorry, I seemed to miss the exception that I am getting: >> >> Caused by: java.lang.IllegalArgumentException: Could not find a suitable >> setter for property: filter as there isn't a setter method with same type: >> java.lang.String nor type conversion possible: No type converter available >> to convert from type: java.lang.String to the required type: >> org.apache.camel.component.file.GenericFileFilter with value >> #fileExistsFilter >> >> Not sure why but it seems that the filter is not getting the param from >> the registry but trying to pass the value in as a normal string. Based on >> the last comment it sounds like this should be working? >> >> >> On Thu, Jun 28, 2012 at 11:21 PM, Claus Ibsen <claus.ib...@gmail.com>wrote: >> >>> On Fri, Jun 29, 2012 at 12:32 AM, David Capwell <dcapw...@gmail.com> >>> wrote: >>> > Playing with the delay fields defined in file2 seems to improve download >>> > speeds. Now that this is working better I want to be able to skip any >>> file >>> > that has already been processed (should persist on restart). It looked >>> to >>> > me that I should be able to set this up by using the filter option, but >>> I >>> > am having a hard time figuring out how to use this option without using >>> > spring. >>> > >>> >>> Yeah you can either use the filter or the idempotent consumer. The >>> latter is the EIP pattern that fits this problem. >>> The file/ftp component has baked in of both. And the latter has >>> plugins for various persistent storages. >>> >>> The code below seems okay, if you configure the endpoint from the uri, >>> then you need to refer to your >>> bean that implements the filter using the # notation, as you do. >>> >>> Then you need to enlist that bean in the registry. In Spring XML that >>> happens automatic when you do a <bean>. >>> When you use pure Java, you need to pass in a registry implementation >>> when you create the camel context, >>> as you do with the code below using the SimpleRegistry. >>> >>> Then its just a matter of putting the bean in the simple registry, as >>> shown in the code. >>> >>> >>> The unit test of the camel-ftp component has plenty of examples as well >>> >>> https://svn.apache.org/repos/asf/camel/trunk/components/camel-ftp/src/test/java/ >>> >>> >>> > here is the code I am using (might not compile cause its copy/paste from >>> > diff places): >>> > >>> > SimpleRegistry registry = new SimpleRegistry(); >>> > registry.put("fileExistsFilter", new CopyFilter()); >>> > >>> > CamelContext context = new DefaultCamelContext(registry); >>> > context.addRoutes(new RouteBuilder() { >>> > @Override >>> > public void configure() throws Exception { >>> > from("sftp://example.com/path/to/file?password="; + pass + >>> > "&binary=true&delay="+delay+ "&initialDelay=" + delay + >>> > "&filter=#fileExistsFilter") >>> > .to("file://data/ftp?fileExist=Ignore"); >>> > >>> > } >>> > }); >>> > context.start(); >>> > >>> > ... >>> > public static class CopyFilter implements GenericFileFilter { >>> > >>> > @Override >>> > public boolean accept(GenericFile genericFile) { >>> > // check to see if file exists locally >>> > File localFile = new File("data/ftp", genericFile.getFileName()); >>> > if(localFile.exists()) { >>> > // check to make sure the sizes match >>> > if(localFile.length() == genericFile.getFileLength()) { >>> > // file has already been copied, ignore >>> > return false; >>> > } >>> > } >>> > return true; >>> > } >>> > } >>> > >>> > >>> > Thanks for your time reading this email >>> > >>> > On Thu, Jun 28, 2012 at 6:07 AM, David Capwell <dcapw...@gmail.com> >>> wrote: >>> > >>> >> I'll take a look at file2 to see if that helps. >>> >> >>> >> My goal is to push the data from FTP to s3. Copying locally so I know >>> >> what has been processed, >>> >> On Jun 28, 2012 12:48 AM, "Claus Ibsen" <claus.ib...@gmail.com> wrote: >>> >> >>> >>> Hi >>> >>> >>> >>> As mentioned on the ftp2 page >>> >>> http://camel.apache.org/ftp2 >>> >>> >>> >>> See the file2 page for more options as the ftp component inherits >>> these >>> >>> options >>> >>> http://camel.apache.org/file2.html >>> >>> >>> >>> The delay and consumer.delay option is the same. delay is just >>> >>> shorthand for consumer.delay. >>> >>> >>> >>> And no you cannot avoid the download if you later want to upload it >>> >>> someplace else, and the file exists. >>> >>> The ftp consumer don't know about this. >>> >>> >>> >>> A tricky improvement could be to add a new option to only download the >>> >>> file on demand, but that is a bit tricky >>> >>> as you would need a live connection and the ftp client to still be >>> >>> around and connected. >>> >>> >>> >>> >>> >>> What are you trying to do? To sync files between 2 ftp servers? >>> >>> >>> >>> >>> >>> >>> >>> On Thu, Jun 28, 2012 at 2:15 AM, David Capwell <dcapw...@gmail.com> >>> >>> wrote: >>> >>> > I am trying to use sftp (camel 2.9.2) and notice that every >>> download is >>> >>> > spread out to one every 10 seconds (files are around 10kb). Based >>> off >>> >>> the >>> >>> > examples in http://camel.apache.org/ftp2.html it looks like there >>> are >>> >>> delay >>> >>> > options (they are not listened under options, but used in example) >>> but >>> >>> when >>> >>> > I use them nothing changes. How can I have the client pull files >>> >>> faster? >>> >>> > >>> >>> > Another question, if I am trying to write the ftp files locally, is >>> >>> there >>> >>> > any way to avoid the file download if the local file is around? >>> >>> > I append fileExist=Ignore in the to() but it seems that the files >>> still >>> >>> > get written again. >>> >>> > >>> >>> > Here is the route I am using: >>> >>> > >>> >>> > public void ftpToLocal() throws Exception { >>> >>> > runRoutes(new RouteBuilder() { >>> >>> > @Override >>> >>> > public void configure() throws Exception { >>> >>> > Registry registry = getContext().getRegistry(); >>> >>> > String user = (String) registry.lookup("ftpUser"); >>> >>> > String pass = (String) registry.lookup("ftpPass"); >>> >>> > >>> >>> > // we use a delay of 60 minutes (eg. once pr. hour we poll >>> the >>> >>> FTP >>> >>> > server >>> >>> > // long delay = 60 * 60 * 1000L; >>> >>> > // long delay = TimeUnit.SECONDS.toMillis(1); >>> >>> > long delay = 200; >>> >>> > >>> >>> > from("sftp://"+user+"@example.com/path/to/dir?password="; + >>> pass >>> >>> + >>> >>> > "&binary=true&delay="+delay+"&consumer.delay=" + delay) >>> >>> > .to("file://data/ftp?fileExist=Ignore"); >>> >>> > } >>> >>> > }); >>> >>> > } >>> >>> > >>> >>> > >>> >>> > Thanks for your time reading this email. >>> >>> >>> >>> >>> >>> >>> >>> -- >>> >>> Claus Ibsen >>> >>> ----------------- >>> >>> FuseSource >>> >>> Email: cib...@fusesource.com >>> >>> Web: http://fusesource.com >>> >>> Twitter: davsclaus, fusenews >>> >>> Blog: http://davsclaus.com >>> >>> Author of Camel in Action: http://www.manning.com/ibsen >>> >>> >>> >> >>> >>> >>> >>> -- >>> Claus Ibsen >>> ----------------- >>> FuseSource >>> Email: cib...@fusesource.com >>> Web: http://fusesource.com >>> Twitter: davsclaus, fusenews >>> Blog: http://davsclaus.com >>> Author of Camel in Action: http://www.manning.com/ibsen >>> >> >> -- Claus Ibsen ----------------- FuseSource Email: cib...@fusesource.com Web: http://fusesource.com Twitter: davsclaus, fusenews Blog: http://davsclaus.com Author of Camel in Action: http://www.manning.com/ibsen
