Created a work around to this by writing an independent service using just Jsch & Quartz doing an sftp every hour. >From Java we can pass in any expression we like and actually using Jsch can find the Latest Modified files from the system date. This also solves the SFTP passwords in clear text issue as now I can encrypt them in Java. So this service runs totally independent of Synapse. As a bonus this service can unzip SFTped zipped files we get from some locations and split large files as Jsch can tell us how big they are before SFTPing. A prior issue was submitting large files to some web services that required a new type of iterator in Synapse to be developed.
So now Synapse just uses VFS to poles the file system, the service downloads files too, to submit manageable size text files to the destination Web Services. >From the other Forums this solution also will resolve a possible issue with Commons VFS SFTP having problems with large files. Not sure if these issues are real but dates are fairly recent. Start Quotes: "This doesn't work with larger files. The VFS seems to have an issue with the getInputStream(), where it loads the whole file into memory before returning a memory stream. Eventually the JVM will run out of memory. In any case, it was a nice start--I'm going to try using Jsch to work around this problem. # posted by Blogger Wes : 11:34 PM, November 24, 2008 Yes. The code is not working with larger files. It's OK up to 30mb files. If it exceeds more than 30mb it's causing problems(memory issues). Any idea how to get rid of this problem. # posted by Anonymous Anonymous : 3:17 PM, December 16, 2008. End Quotes. I have used a more recent release of Jsch than the Jar used in Synapse; that requires a later version of Java, so would not be possible to do this as a Synapse class mediator at the moment, although this would be a better solution. Thanks Kim -----Original Message----- From: Kim Horn [mailto:[email protected]] Sent: Tuesday, 7 April 2009 7:19 AM To: [email protected] Subject: RE: Can VFS SFTP be used to download todays files only Hello Asankha, yes we did get some small simple scripts into production, to help our manual process, in parralel with evaluation, but had to pull them out as they did not run with the latest trunk required to solve another issue; the VFS memory not being released. The fix to this resulted in the "out" sequences producing exceptions with VFS. Thanks Kim -----Original Message----- From: Asankha Perera on behalf of Asankha C. Perera Sent: Mon 06/04/2009 20:20 To: [email protected] Subject: Re: Can VFS SFTP be used to download todays files only Hi Kim > 1) How do you specify a regex to download files with todays date that will > run in synapse unattended for weeks ? > Yes I can hard code an expression to match 31 March 2009 and after but that > is no good today, is it. > Java has the concept of Date NOW; this is what I mean by today. > > You say this is "Quite easy" please show me how, as I cannot see how this > can be done ? If it is easy then my problem is solved. Great, but this is my > question ? > > 2) If an SFTP server has 2000 old files permanently on it now that will not > be removed; thats grows every day. How to I download just the new files ? > E.G Those placed on there with todays date in the name. I have no permission > on the partners SFTP server to MOVE files or DELETE them, they just stay > there. It is not a good assumption that files can be MOVED or DELETED on > another organisations SFTP server. Very rarely in real B2B do you get this > luxury. > The real world cases for which the transport has been originally written is to poll for a file or files at some specific location that matches some pattern, process them, and either 1) delete or 2) move them since the transport does not have state to remember what has been polled already. The commercial Aqualogic service bus follows this pattern too [1], while both Mule and Camel additionally supports the specification of date info in the file pattern. I couldn't readily find much about Fuse or JBoss etc. The solution I gave will not work, if you cannot move or delete processed files on the target sftp server, to prevent re-processing them later. However, supporting of date information in the pattern would be fairly easy to implement. > 3) You suggest SYnapse has a concept of new files ? How does this work ? > This is if you are creating a new file.. e.g. saving a message received via http/s, JMS, email etc as a file - maybe to an sftp location like in [2] > 4) Are you saying that when I start Synapse with an SFTP mediator it > remembers every file downloaded ever in the past; so that when we re-start > Synapse (after maintenence) it remembers the old files. If this is true this > is great. I am interested to know where this record of files is kept. Where > is the file ? > > 5) Otherwise I am missing some very interesting behaviour of the Synapse VFS > Mediator that does not seem to be provided by the samples or the > documentation or any prior Forum discussions. > No, the transport is implemented as whats called an "Axis2 transport" .. its actually out of the Synapse core, and not directly dependent or linked. Implementing a record of processed files - even if possible would not be worth it - I think the use of Date in the regex is the most simple way to handle this > I am quite happy to raise a Jira but there is no need if Synapse can do > this, easily, as you say ? > Well.. the point of a JIRA is that all good ideas are captured by those who propose them, so that new developers or the current developers - when they have free time, can look at real world user suggestions for improvement and implement them for future versions > I do not understand this comment about being in a hurry; we are evaluating > Synapse to solve a commercial problem and unfortunately we have encountered > many issues that we are trying to resolve. I interpreted your multiple emails in the past about dropping or almost dropping the use of Synapse to mean that you are in a hurry to get this implemented in Synapse, and thats why I even suggested a pure Java implementation. Of course if you are still in evaluation, the best is to try to select another few ESBs (open source and/or commercial) and try to implement the same, and consider the total cost and the complexity of the final solution. In this case, consider the effort to implement whats missing in Synapse into the total cost of the solution. However, from some of your emails, I see production requirements being talked about, and thats why sometime back I suggested getting commercial support - in which case, someone else would have taken care of all of these issues - implementing what was missing as required. To be honest, you could have got a complete solution for your real world problem, with commercial support and a guarantee, probably for much less than the number of hours you've already spent and would spend * a reasonable hourly rate for the cost of your time, which again is money. > Unfortunately when I think all the hurdles are solved; then we stumble over new ones. > Yes unfortunately real world projects have definite requirements and budgets and deadlines. > I do not see the point of open source software if it cannot solve commercial > problems; unless Synapse is just an Academic Research project ? I am sure > WS02 are using Synapse to solve real world commercial problems and not just > for research. I am sure they make money from using the code. > Of course, many real world users use Synapse - but some are shy to use the public forums, and risk exposing confidential information, security and passwords and sensitive data etc. I usually privately inform users whenever this happens.. since logs could reveal a lot of information which users sometimes are not aware of; and I remember I informed you too when you posted a Base64 encrypted password of a live server. Synapse is not an academic project at all.. it never was. Multiple large health care providers in the US, the US Army, Governments in Asia and Europe, mobile resource management companies, one of the largest auto makers (who is still doing good in this economy), and SOA appliance vendor that embeds Synapse, European / Asian banks and a large media company across the US and Europe all use it. These are some of the users I know about.. and WSO2 would have more users too, that I am not aware of Here are some "public" comments by a few users in the past: "Thanks for the quick fix. Synapse continues to rock." <http://markmail.org/message/qbk4wvncjvtvat2s> "Synapse continues to impress. Looking forward to 1.1 features upon its release" <http://markmail.org/message/zwyrzleaop25bqad> "Thanks everyone for taking time to look into this -- you guys are great (know wonder Synapse is such a great product!)" <http://markmail.org/message/5lldrfesfb2wjv44> "I have been using the excellent Apache Synapse lightweight Enterprise Service Bus over the last year or so, including on a live, production project." <http://thejavamonkey.blogspot.com/2008/09/apache-synapse-how-to-run-wit hout.html> "We are using Apache Synapse version 1.1 to develop webservices for our mission critical "Surface Deployment and Distribution command" of U.S Army" <http://markmail.org/message/b67h6ggfh7vspt2u> >"I do not see the point of open source software if it cannot solve commercial problems;" I took this line out of the previous block, so that I can address it separately.. As far as I am aware, Open source software is not about building free software by volunteers to commercial companies who makes money out of it. For users its more about having lifetime access to the source code of the products, and knowing that licensing is free when you are ready to drop paying for support, so that you do not lock into proprietary vendor products, or with vendors who may not survive long term. When the code is available freely, you can compile and support it yourself, or get someone else to support it for you. In Apache, its about a community of people who are interested in building something.. In Apache we even consider 'Community over code', and enjoy working with diverse people and learning from them for us to improve. But Apache does not pay us! we are free to sell support or consulting to make sure we also survive :-) Open source developers who volunteer their time, do not owe anything to the users - besides making sure the source code is available in full. Anyone who users the software is free to fix any issues, and if they like to contribute them back to the community. > Unfortunately not all Synapse users are expert enough to contribute yet; so > once again I mention the Catch 22 of your comments. It takes time for new > people to get up to speed so they can contribute. > > So I spent about 4 hours researching this today before posting; and I > believe this is a contribution. We have real world requirements that would improve Synapse; this is a > contribution. > > Asking questions that do not seem to be covered by documentation; so that > others can benefit is a contribution. Thats all I can offer at the moment. > I am fully aware that all first time users would not be using Synapse like they were pro's. From my past experience at WSO2, and at present, many users ask for help on a POC and/or training to get them started. They then develop an initial prototype project themselves and/or move into production with development support or professional services. Thereafter they use dev support to ask questions and learn more advanced stuff. BTW, if I couldn't swim, I am aware there are many sites on the Internet that hosts loads of documentation, and even YouTube videos on how to swim all styles - free like open source software :-) . Now I may evaluate these techniques to see if they work in a pool of 5' depth.. but not in the open sea, not without knowing someone is there to help me if I get in trouble. I would rather pay a professional trainer to train me to swim, and then try it out more stuff when I am comfortable.. and of course blog about it later so that others would learn something from me too. When I can swim and know I can stay afloat, the YouTube videos would allow me to learn more cool stuff.. and maybe I could invent something new and contribute it back so that others could follow it too. I will not be like Adinnapubbaka in the story Mattakundali [3] and try to learn and do what I cannot. > Unfortunately we have made an extensive investment in Synapse, in time, and > at this point I am trying not to have to drop the use of this product and > trying my hardest to demonstrate that this product can solve just one single > real world problem for us. Am I in a hurry, yes, we have passed the deadline. > So does this mean that you are trying to install Synapse into production?.. I would not invest much time or effort in an evaluation, but consider more options in parallel - maybe with some dev support to help me, to reduce my overall risk. cheers asankha [1] http://edocs.bea.com/alsb/docs261/sftptransport/intro.html [2] http://synapse.apache.org/Synapse_Samples.html#Sample254 [3] http://books.google.lk/books?id=sTclW3OWyw8C&pg=PA37&dq=%22mattakundali+ (Flat-Earring)%22 -- Asankha C. Perera AdroitLogic, http://adroitlogic.org http://esbmagic.blogspot.com
