Kim, Maybe at this point it would be useful if you could give a quick overview of the open issues that you still face.
Andreas On Tue, Apr 7, 2009 at 08:55, Kim Horn <[email protected]> wrote: > Created a work around to this by writing an independent service using > just Jsch & Quartz doing an sftp every hour. > From Java we can pass in any expression we like and actually using Jsch > can find the Latest Modified files from the system date. This also > solves the SFTP passwords in clear text issue as now I can encrypt them > in Java. So this service runs totally independent of Synapse. As a bonus > this service can unzip SFTped zipped files we get from some locations > and split large files as Jsch can tell us how big they are before > SFTPing. A prior issue was submitting large files to some web services > that required a new type of iterator in Synapse to be developed. > > So now Synapse just uses VFS to poles the file system, the service > downloads files too, to submit manageable size text files to the > destination Web Services. > > From the other Forums this solution also will resolve a possible issue > with Commons VFS SFTP having problems with large files. Not sure if > these issues are real but dates are fairly recent. > > Start Quotes: > > "This doesn't work with larger files. The VFS seems to have an issue > with the getInputStream(), where it loads the whole file into memory > before returning a memory stream. Eventually the JVM will run out of > memory. In any case, it was a nice start--I'm going to try using Jsch to > work around this problem. > # posted by Blogger Wes : 11:34 PM, November 24, 2008 > > Yes. The code is not working with larger files. It's OK up to 30mb > files. If it exceeds more than 30mb it's causing problems(memory > issues). Any idea how to get rid of this problem. > # posted by Anonymous Anonymous : 3:17 PM, December 16, 2008. > > End Quotes. > > > I have used a more recent release of Jsch than the Jar used in Synapse; > that requires a later version of Java, so would not be possible to do > this as a Synapse class mediator at the moment, although this would be a > better solution. > > Thanks > Kim > > > > -----Original Message----- > From: Kim Horn [mailto:[email protected]] > Sent: Tuesday, 7 April 2009 7:19 AM > To: [email protected] > Subject: RE: Can VFS SFTP be used to download todays files only > > Hello Asankha, > > yes we did get some small simple scripts into production, to help our > manual process, in parralel with evaluation, but had to pull them out as > they did not run with the latest trunk required to solve another issue; > the VFS memory not being released. The fix to this resulted in the "out" > sequences producing exceptions with VFS. > > Thanks > Kim > > > > -----Original Message----- > From: Asankha Perera on behalf of Asankha C. Perera > Sent: Mon 06/04/2009 20:20 > To: [email protected] > Subject: Re: Can VFS SFTP be used to download todays files only > > Hi Kim >> 1) How do you specify a regex to download files with todays date that > will >> run in synapse unattended for weeks ? >> Yes I can hard code an expression to match 31 March 2009 and after but > that >> is no good today, is it. >> Java has the concept of Date NOW; this is what I mean by today. >> >> You say this is "Quite easy" please show me how, as I cannot see how > this >> can be done ? If it is easy then my problem is solved. Great, but this > is my >> question ? >> >> 2) If an SFTP server has 2000 old files permanently on it now that > will not >> be removed; thats grows every day. How to I download just the new > files ? >> E.G Those placed on there with todays date in the name. I have no > permission >> on the partners SFTP server to MOVE files or DELETE them, they just > stay >> there. It is not a good assumption that files can be MOVED or DELETED > on >> another organisations SFTP server. Very rarely in real B2B do you get > this >> luxury. >> > The real world cases for which the transport has been originally written > > is to poll for a file or files at some specific location that matches > some pattern, process them, and either 1) delete or 2) move them since > the transport does not have state to remember what has been polled > already. > > The commercial Aqualogic service bus follows this pattern too [1], while > > both Mule and Camel additionally supports the specification of date info > > in the file pattern. I couldn't readily find much about Fuse or JBoss > etc. > > The solution I gave will not work, if you cannot move or delete > processed files on the target sftp server, to prevent re-processing them > > later. However, supporting of date information in the pattern would be > fairly easy to implement. >> 3) You suggest SYnapse has a concept of new files ? How does this work > ? >> > This is if you are creating a new file.. e.g. saving a message received > via http/s, JMS, email etc as a file - maybe to an sftp location like in > [2] >> 4) Are you saying that when I start Synapse with an SFTP mediator it >> remembers every file downloaded ever in the past; so that when we > re-start >> Synapse (after maintenence) it remembers the old files. If this is > true this >> is great. I am interested to know where this record of files is kept. > Where >> is the file ? >> >> 5) Otherwise I am missing some very interesting behaviour of the > Synapse VFS >> Mediator that does not seem to be provided by the samples or the >> documentation or any prior Forum discussions. >> > No, the transport is implemented as whats called an "Axis2 transport" .. > > its actually out of the Synapse core, and not directly dependent or > linked. Implementing a record of processed files - even if possible > would not be worth it - I think the use of Date in the regex is the most > > simple way to handle this >> I am quite happy to raise a Jira but there is no need if Synapse can > do >> this, easily, as you say ? >> > Well.. the point of a JIRA is that all good ideas are captured by those > who propose them, so that new developers or the current developers - > when they have free time, can look at real world user suggestions for > improvement and implement them for future versions >> I do not understand this comment about being in a hurry; we are > evaluating >> Synapse to solve a commercial problem and unfortunately we have > encountered >> many issues that we are trying to resolve. > I interpreted your multiple emails in the past about dropping or almost > dropping the use of Synapse to mean that you are in a hurry to get this > implemented in Synapse, and thats why I even suggested a pure Java > implementation. Of course if you are still in evaluation, the best is to > > try to select another few ESBs (open source and/or commercial) and try > to implement the same, and consider the total cost and the complexity of > > the final solution. In this case, consider the effort to implement whats > > missing in Synapse into the total cost of the solution. > > However, from some of your emails, I see production requirements being > talked about, and thats why sometime back I suggested getting commercial > > support - in which case, someone else would have taken care of all of > these issues - implementing what was missing as required. To be honest, > you could have got a complete solution for your real world problem, with > > commercial support and a guarantee, probably for much less than the > number of hours you've already spent and would spend * a reasonable > hourly rate for the cost of your time, which again is money. > >> Unfortunately when I think all the hurdles are solved; then we stumble > over new ones. >> Yes unfortunately real world projects have definite requirements and > budgets and deadlines. >> I do not see the point of open source software if it cannot solve > commercial >> problems; unless Synapse is just an Academic Research project ? I am > sure >> WS02 are using Synapse to solve real world commercial problems and not > just >> for research. I am sure they make money from using the code. >> > Of course, many real world users use Synapse - but some are shy to use > the public forums, and risk exposing confidential information, security > and passwords and sensitive data etc. I usually privately inform users > whenever this happens.. since logs could reveal a lot of information > which users sometimes are not aware of; and I remember I informed you > too when you posted a Base64 encrypted password of a live server. > > Synapse is not an academic project at all.. it never was. Multiple large > > health care providers in the US, the US Army, Governments in Asia and > Europe, mobile resource management companies, one of the largest auto > makers (who is still doing good in this economy), and SOA appliance > vendor that embeds Synapse, European / Asian banks and a large media > company across the US and Europe all use it. These are some of the users > > I know about.. and WSO2 would have more users too, that I am not aware > of > > Here are some "public" comments by a few users in the past: > "Thanks for the quick fix. Synapse continues to rock." > <http://markmail.org/message/qbk4wvncjvtvat2s> > "Synapse continues to impress. Looking forward to 1.1 features upon its > release" <http://markmail.org/message/zwyrzleaop25bqad> > "Thanks everyone for taking time to look into this -- you guys are great > > (know wonder Synapse is such a great product!)" > <http://markmail.org/message/5lldrfesfb2wjv44> > "I have been using the excellent Apache Synapse lightweight Enterprise > Service Bus over the last year or so, including on a live, production > project." > <http://thejavamonkey.blogspot.com/2008/09/apache-synapse-how-to-run-wit > hout.html> > > "We are using Apache Synapse version 1.1 to develop webservices for our > mission critical "Surface Deployment and Distribution command" of U.S > Army" <http://markmail.org/message/b67h6ggfh7vspt2u> > >>"I do not see the point of open source software if it cannot solve > commercial problems;" > > I took this line out of the previous block, so that I can address it > separately.. As far as I am aware, Open source software is not about > building free software by volunteers to commercial companies who makes > money out of it. For users its more about having lifetime access to the > source code of the products, and knowing that licensing is free when you > > are ready to drop paying for support, so that you do not lock into > proprietary vendor products, or with vendors who may not survive long > term. When the code is available freely, you can compile and support it > yourself, or get someone else to support it for you. In Apache, its > about a community of people who are interested in building something.. > In Apache we even consider 'Community over code', and enjoy working with > > diverse people and learning from them for us to improve. But Apache does > > not pay us! we are free to sell support or consulting to make sure we > also survive :-) > > Open source developers who volunteer their time, do not owe anything to > the users - besides making sure the source code is available in full. > Anyone who users the software is free to fix any issues, and if they > like to contribute them back to the community. > >> Unfortunately not all Synapse users are expert enough to contribute > yet; so >> once again I mention the Catch 22 of your comments. It takes time for > new >> people to get up to speed so they can contribute. >> >> So I spent about 4 hours researching this today before posting; and I >> believe this is a contribution. We have real world requirements that > would improve Synapse; this is a >> contribution. >> >> Asking questions that do not seem to be covered by documentation; so > that >> others can benefit is a contribution. Thats all I can offer at the > moment. >> > I am fully aware that all first time users would not be using Synapse > like they were pro's. From my past experience at WSO2, and at present, > many users ask for help on a POC and/or training to get them started. > They then develop an initial prototype project themselves and/or move > into production with development support or professional services. > Thereafter they use dev support to ask questions and learn more advanced > > stuff. > > BTW, if I couldn't swim, I am aware there are many sites on the Internet > > that hosts loads of documentation, and even YouTube videos on how to > swim all styles - free like open source software :-) . Now I may > evaluate these techniques to see if they work in a pool of 5' depth.. > but not in the open sea, not without knowing someone is there to help me > > if I get in trouble. I would rather pay a professional trainer to train > me to swim, and then try it out more stuff when I am comfortable.. and > of course blog about it later so that others would learn something from > me too. When I can swim and know I can stay afloat, the YouTube videos > would allow me to learn more cool stuff.. and maybe I could invent > something new and contribute it back so that others could follow it too. > > I will not be like Adinnapubbaka in the story Mattakundali [3] and try > to learn and do what I cannot. >> Unfortunately we have made an extensive investment in Synapse, in > time, and >> at this point I am trying not to have to drop the use of this product > and >> trying my hardest to demonstrate that this product can solve just one > single >> real world problem for us. Am I in a hurry, yes, we have passed the > deadline. >> > So does this mean that you are trying to install Synapse into > production?.. I would not invest much time or effort in an evaluation, > but consider more options in parallel - maybe with some dev support to > help me, to reduce my overall risk. > > cheers > asankha > > [1] http://edocs.bea.com/alsb/docs261/sftptransport/intro.html > [2] http://synapse.apache.org/Synapse_Samples.html#Sample254 > [3] > http://books.google.lk/books?id=sTclW3OWyw8C&pg=PA37&dq=%22mattakundali+ > (Flat-Earring)%22 > > -- > Asankha C. Perera > AdroitLogic, http://adroitlogic.org > > http://esbmagic.blogspot.com > > > > > >
