Hi Grant, Quite a few questions here. Thanks for taking the time and evaluating Apache Camel.
Yes, Camel is ideal for such scenarios, because: 1. It's specifically designed for that, systems integration. 2. It's simple to use and requires minimal coding 3. It supports out of the box a large number of protocols and data formats, which means that it's very likely to support future requirements once your application grows. 4. EIPs provide a comprehensive set of atomic operations which, combined, allow for quite complex use cases. In addition to the EIP book or site I would suggest starting with the examples and the camel-bindy unit tests. You will probably need a splitter there too. The Rider Auto Parts example in the Camel in Action book also has many similarities with the scenario you describe. Spring is not actually required, but we (and many users) use it intensively. The camel-bindy dependency on camel-spring is only for testing. So it's really up to you. Cheers, Hadrian On Feb 20, 2011, at 11:11 AM, Grant Rettke wrote: > Hi, > > For however as far as you read, thank you, I value your expertise, and > your time... you are simply the only folks with whom I could share > these "dumb" questions and discuss these topics. > > My story: > > A few months ago I inherited a custom data management system. Its goal > in life is to allow clients to look up data via a service call. Data > had always been loaded by hand-coding an Excel spreadsheet using > copy-and-pasted functions to generate SQL. This needs to go away. > Recently we got a "little" requirement like this: > > For each DTO, load data into the DB like this: > 1. Read the CSV file. > 2. Populate the DTO. > 3. Call a service method 'save' which accepts any DTO of the same supertype. > > We had some proof of concept work lying around (that I had written a > few months ago, with less understanding of the eventual goal) for > parsing CSV files and loading DTOs. It is really basic, it uses > SimpleCSV to parse the files, the specification for mapping columns to > properties of the DTO are done in an intermediary "model" class so the > BAs can > just look at the class and say "Oh yea, the property state actually > parses a string from column 4", and then Apache Commons Beanutils > copies the properties from the model over to the DTO (can't do this in > the DTOs because of dependency issues). This works well enough but for > the horrible hand-coding of the mappings.Friday night I figured it > would be nice to use annotations instead, and then decided that it is > a bad idea to write yet another framework and set out searching for > one. I found Apache Camel Bindy. I was delighted. It is exactly what I > wanted. The only problem is that I am totally ignorant of Apache > Camel. > > I spent some time reading up on the EIPs and about how Camel Works. I > need help figuring out of I should dig deeper and whether our project > is a good fit. > > I feel like the basic flow of above is a good candidate for > implementation as a route in Camel. > > <file rows> -> [ftp endpoint] -> [import queue] -> [CSV translation > via Bindy] -> <POJOs> -> [POJO Import Queue] -> (Database) > > What do you think, is it a good candidate? > > I keep in mind that even if we didn't use Camel, we would end up doing > this exact same thing, using custom code... and again we do not want > to write a custom framework! :). I think it is a good candidate since > in the future, it can only get more complex. > > Thinking ahead to the future, I do know that the business wants to do > things like: > 1. Allow a user to submit specific files to the input queue via a UI; > so now there are two ways to get get files queued up for importation. > 2. Allow a user to define logical "jobs" for importing files that would allow: > A. Tracking of what files came in with the job. > B. What rows of data came in with the job. > C. Such that we could view all of the information involved, and > even "back it out" at some point. > 3. Import data from the legacy system. > 4. Eventually we'll have hundreds of DTOs to map, every couple of weeks. > > It seems like #1 is straightforward two sources into one end point scenario. > > It seems like #2 is a content enricher scenario. > > It seems like #3 is a scenario that would benefit from an Enteprise > Integration System. > > It seems like #4 is again a good scenario. > > It looks like long term it would be a good addition because I don't > want to write a framework or custom code to handle the intricacies > (though the world would not end if we did, it certainly would not be > any easier). > > What do you think? > > At this point, nobody in the organization will be using an ESB, > however, I feel like Camel would work fine as an Enteprise Integration > System, deployed in the EAR that contains the service that provides > this system I've inherited. > > Does that make sense? Is there a precedent? Is there any > documentation/resources for this usage scenario other than the one in > the user guide (the one that doesn't use Spring)? > > Is Spring a best-practice for using Camel? I would prefer to avoid it > if I can and keep things as simple as possible, then again I never > learned Spring, so perhaps I am the problem :). > > Is it easy to do things like add a file to the queue via a web > application? What I mean is rather than dropping in a file somewhere, > just upload it via the web app and plunk it into the queue? I'm > thinking that it is, just manually get a handle on the endpoint and > drop in the message. > > How has your experience been with testing? Must you use Spring? > > If we move forward with this, I'm planning on using the Camel In > Action book (along with reading EIP as I know next to nothing about > both). Is that the best one? > > Best wishes, > > Grant Rettke > > -- > http://www.wisdomandwonder.com/ > ACM, AMA, COG, IEEE
