I think Jeremy's point that this might be an example of using the "wrong hammer on the screw" depends on how structured (vs. open) the physical model actually is. More generally than just whether SDO is the right approach for this particular application, I think it's a question of whether it's worth using any Java binding technology on a model that is full of open and mixed content - and even worse, the open content is not described by a schema (i.e., lax). Whether it's SDO, JAXB, or some other XML binding technology, the programming model that results is more DOM-like then a nice static model that we'd like. In the SDO case you end up doing a lot of DOM-like access using Sequence's (accessing "mixed", "any", and "anyAttribute" properties) - hardly a clean and beautiful Java binding. I'm not sure if the threshold of where the physical model is too loosely defined to map to a clean Java binding has been crossed or not in this case, but it looks close anyway. If it wasn't, we wouldn't need a logical model.
I think the other problems that Jeremy points out, need to get fixed anyway and SDO should be a competitive Java binding technology, as well as all the other things. Frank. Jeremy Boynes <[EMAIL PROTECTED]> 03/21/2006 02:44 PM Please respond to tuscany-dev To [email protected] cc Subject Re: Framework for StAX-based model loading Jean-Sebastien Delfino wrote: > Jeremy Boynes wrote: > >> Jim Marino wrote: >> >> >>> Hi Jeremy, >>> >>> Could you briefly enumerate what you see as the benefits to the StAX >>> framework over alternatives? >>> >>> >> >> >> The final straw that prompted me to do this was the amount of >> classloader wrangling we ended up doing in the Tomcat code a couple of >> weeks ago. We need to keep track of context classloader switching >> between loading the model and loading any application code. There is >> plenty of room for subtle errors to creep in. >> > > The classloader issues we ran into with the current SDO implementation > need to be solved. I am not sure that they are a sufficient reason for > stopping to use SDO and moving to a different technology. I'm surprised > to see that we have a databinding technology in Tuscany but we are > running away from it when we encounter our first problems with it. I > think we should spend a little more time trying to fix it instead of > running away from it. By the way the classloader problems we ran into a > couple weeks ago were not just caused by SDO, this was a combination of > SDO, Axis2 and some of the factories used under the cover by Axis2, all > having different requirements in terms of "current" class loader. > I wouldn't say "running away" (that's a little dramatic) but I do think it is fair to say that we ran into problems due to the way our SDO implementation couples the type system to the thread's classloader. The goal of SDO is not to be just another XML data binding technology, its purpose is to "simplify data programming so developers can focus on business logic instead of the underlying technology." It does this through abstraction of that technology, by providing a data-graph oriented API and capabilities like containment and change tracking. We are not using those capabilities. We're actually only using a tiny fraction of its capabilites. I think it is reasonable to evaluate whether it is the right hammer to use on this screw. >> The SDO solution (actually this would be true of any XML->POJO binding) >> was fine when the logical model was an exact replica of the XML files. >> However, to support more logical unit testing (and other uses) the model >> has now shifted back to being more of a true configuration model. This >> means we can't just slurp the XML into objects and use them directly, we >> need to read in the POJOs and then run a transformation on them. This >> adds an additional phase to the load process that needs maintenance. >> >> > > The logical model was never an exact replica of the XML files. Whatever > technology you use you'll need to do the following: > 1. handle the parsing/loading from XML > 2. transform what you get from the XML into a logical model > The current SDO based approach separated the two concerns. With this new > StAX approach we do (1) and (2) together. I think this will create > complexity over time. > I would describe our problem slightly differently saying we need: 1) to parse the incoming bytestream 2) to use the parse results to build the logical model The first of these is being handled by StAX, the second the code we provide that reacts to parse events and builds the model. This seems like a fairly clear separation of concerns. I would also break down the SDO-based solution differently: 1) SDO parses the incoming bytestream 2) SDO builds its physical model to represent the XML 3) the SCDLModelContentHandler parses the physical model and generates a partial logical model 4) linkers generated in 3) run to complete the logical model I think this is less separated. The implementation of SCDLContentModelHandler is tightly coupled to the physical model generated by SDO. The linkers are also coupled to the element handlers (the caseXXXX methods) that parse each physical model object. I'd also point out that the implementation of the content handler for the core assembly model is different from other extensions: the former uses a (code-generated) case dispatcher, the others tend to use a single method with manually coded instanceof tests. >> Having a container system there able to manage the loaders means >> extending the model is easy - an extension just needs to contribute its >> model elements and a XML handler. There is no need to codegen a separate >> XML model and write a transformer. There is also only one extension >> registry rather than two (the SDO type registry and the SCDL loader >> registry). >> >> > > I don't see why codegen is a problem. In general I'd rather get some > code generated than write it myself. I agree that with the SDO approach > you have to register the generated model and your handler/transformer. I > don't really understand the difference you make between a handler and a > transformer and why it's easier to write a handler than to codegen a > model and write a transformer. With the SDO based approach you need to > write code that gets data out of an SDO model with nice generated getter > methods. With a StAX based approach (and it would be very similar with a > DOM or SAX based approach) you get the data out of a more weakly typed > model. Frankly I prefer to write: > String name=component.getName(); > than > String name=reader.getAttributeValue(null, "name"); > I wouldn't have such a big issue with codegen if it generated the code that we wanted. However, the codegen here is generating objects that represent the physical structure of the XML rather than ones that map to the logical model. This leads to the need for another parser/transform (the SCDLModelContentHandler) to convert from the physical representation to the logical one. The result is we have both the complexity of code generation *and* the complexity of a manually coded parser/transformer. >> The XML handling is pluggable - it just uses the standard StAX APIs >> rather than internal hooks to our SDO implementation and/or EMF. A >> validating StAX parser can be used if required; semantic validation is >> still being performed in the model and builders. >> > > We have been working to remove the dependencies on EMF, so again I don't > see how this can motivate moving to StAX. One of the goals of Tuscany is > to provide a good SDO story anyway to people who want to load an XML > document into an SDO model, without requiring any hooks into our SDO > implementation or EMF. Again I think we should all work to improve our > SDO story instead of using something else. For example I think that we > should improve SDO to provide a good integration with StAX. > The point I was making here is that there are multiple implementations of StAX available that can be used and that the loader is not tied to one of them. This may be useful for people embedding Tuscany in other environments where they have some preference over which implementation should be used. We are already working to integrate StAX support with SDO to allow SDO's to be (de)-serialized from/to StAX event streams. We are already using this for better AXIOM integration in the web-services stack. Having a good SDO implementation is a primary objective for the project. However, as I pointed out at the start, SDO is more than just an XML binding technology and we need to make sure its SDO-ness excels. We also need to realize that it is not the universal solution for data binding and that other technologies may work better in specific scenarios. I think we have one of those scenarios. >> Code footprint is better as there is no intermediate form. Performance >> and memory footprint are probably better too. However, I don't see that >> as a major factor as we are only reading config data here (i.e. it's >> once per deployment not once per request). >> > > I agree, code and memory footprint are better with the StAX approach. > >> On the downside, StAX is a technology that may not be as familiar to >> people. However, I think it has enough similarity to DOM/SAX to be >> readily understood. It is also heavily used in Axis2 so we will be >> seeing it anyway. >> We will need to modify the parsing code if the XML changes whereas with >> SDO or another XML->POJO solution that would be handled by generation. >> However, the need to transform the model after load means that we have >> custom parsing code anyway just running on the POJOs (see >> SCDLModelContentHandler). >> > > This is the point that raises the biggest concern for me. I have been > there before multiple times, implemented models and loaders for changing > XML specifications. Usually the kind of approach demonstrated by this > StAX loader looks very simple and very tempting at the beginning of the > project, but ends up in a mess and maintenance nightmare in the longer > term. > Unfortunately I think that just an aspect of this problem. I have seen XML-binding based solutions used in other projects and they have also proved to be very fragile even in the face of static schemas. If we had a pure SDO solution I would be less concerned; my concern here is about the parse/transform code that we have to maintain to convert the physical to the logical model. > If we want to avoid that we need the following: > - more work on this StAX loader to clearly separate the pure > loading/de-serialization aspect from the physical -> logical > transformation aspects, if you mix the two aspects it will be a recipe > for disaster after a year of maintenance adjusting to changes in the XML See above - I think the same applies to the SCDLModelContentHandler > - define intermediate data structures (close to the physical model) to > avoid polluting the logical model with XML specific info (I'm thinking > of all the cases where we're going to have to store intermedaite / half > loaded data and complete/resolve things in a second pass, properties and > wires are two examples that come to mind); if we don't do that and start > adding stuff to the logical model to facilitate the loading phase we > will make a mess of the logical model I don't think we have needed to populate the logical model with physical artifacts. I did make some changes to support the StAX loader such as storing reference targets as pointers rather than using a Reference object directly. I would contend that is a normalization that better represents the logical model - it certainly cut out a couple of bugs that were the result of inconsistent updates to the denormalized References. > - make the StAX loader approach more complete before we jump to switch > to it, to really understand the impact, for example the StAX loaders do > not handle subsystems or property types at the moment, I think that the > support for properties in particular is going to be interesting and will > generate some work to integrate between StAX and SDO if we want to > support SDO properties. Given subsystems are changing and not really supported yet by the runtime this does not seem like a major issue. In fact, looking at the how subsystems are handled in the SCDLModelContentHandler, adding StAX loader support form them would be trivial. How about you have a go at doing it and see how hard/easy adding things to the StAX framework actually is? For properties, I don't think either solution is fully fleshed out yet so using that as a basis for comparison is a little unfair. For example, I would ask how user-defined types are loaded into the SDO type system, or how non-SDO property type support would be added (e.g. to load properties as JAXB objects)? With the StAX solution we have the advantage that by using a standard technology it will be easier to add in other binding frameworks. We are already working on a StAX->SDO deserializer that would give us a solution when the user was using an SDO; further JAXB, XMLBeans and (I believe) Castor all support StAX sources so it should be trival to integrate those. > - come up with brand new solution for serializing/saving models, I am > sure we will run into use cases where we want to save an assembly, a set > of wires, a module component configuration etc; StAX only handles the > loading part, so if we're not using SDO here again we'll have to invent > something else to handle serialization/saving of the models... > We need to do that anyway. If the logical model was an SDO then we could just write it out, but it isn't. We would need to develop and maintain a serializer equivalent to the SCDLModelContentHandler and that is likely to be just as much manual code to maintain as a StAX serializer. > My main concern is about the complexity of maintaining all this code. > Just the (incomplete) support for the core SCDL is already about 750 > lines of code, mixing parsing logic and mapping/construction of the > logical model, using dynamic APIs like reader.getAttribute(null, "name") > compared to the 550 lines of model transformation code using strongly > typed APIs generated from the SCDL XSD in the SCDLModelContentHandler > (with more complete support for the core SCDL). I anticipate many > changes in the SCDL XSD and the logical model during the course of this > year, the StAX based loader approach may look appealing now at the > beginning of the project, but we will only succeed with it if we have > committed contributors and committers ready to maintain this code, make > it complete, and adjust it each time we change the SCDL XSD (and it's > going to be pretty painful, compared to just rebuild to regen the code > from XSD and adjust the transformer where it's broken by the changes). > I think the "adjust the transformer" part here is the key phrase. This is going to be just as much work as maintaining any StAX handler (as they are essentially doing the same thing). If you're going to compare lines of code, the honest metric here would be to compare the number of lines related to parsing and transformation and to exclude the code used to create the logical model. But lines of code alone are not the only metric; when I compare blocks out of SCDLModelContentHandlerImpl with the StAX equivalent the latter seems less complex (at least to me). For example, the <service> code in SCDLModelContentHanlderImpl is: public Object caseService(Service object) { final org.apache.tuscany.model.assembly.Service service=factory.createService(); service.setName(object.getName()); linkers.add(new Runnable() { public void run() { currentComponentType.getServices().add(service); }; }); currentService=service; return service; } and for the contained <interface.java> public Object caseJavaInterface(JavaInterface object) { final JavaServiceContract serviceContract=factory.createJavaServiceContract(); serviceContract.setScope(Scope.INSTANCE); serviceContract.setInterfaceName(object.getInterface()); serviceContract.setCallbackInterfaceName(object.getCallbackInterface()); linkServiceContract(object, serviceContract); return serviceContract; } and private void linkServiceContract(Object object, final ServiceContract serviceContract) { Object container=((DataObject)object).getContainer(); if (container instanceof Service) { // Set a service contract on a service final org.apache.tuscany.model.assembly.Service service=currentService; linkers.add(new Runnable() { public void run() { service.setServiceContract(serviceContract); } }); } else if (container instanceof Reference) { // Set a service contract on a reference final org.apache.tuscany.model.assembly.Reference reference=currentReference; linkers.add(new Runnable() { public void run() { reference.setServiceContract(serviceContract); } }); } else if (container instanceof ExternalService) { // Set a service contract on an external service final org.apache.tuscany.model.assembly.ExternalService externalService=currentExternalService; linkers.add(new Runnable() { public void run() { externalService.getConfiguredService().getService().setServiceContract(serviceContract); } }); } else if (container instanceof EntryPoint) { // Set a service contract on an entry point final org.apache.tuscany.model.assembly.EntryPoint entryPoint=currentEntryPoint; linkers.add(new Runnable() { public void run() { entryPoint.getConfiguredService().getService().setServiceContract(serviceContract); entryPoint.getConfiguredReference().getReference().setServiceContract(serviceContract); } }); } } whereas in the StAX framework the <service> handler is: public Service load(XMLStreamReader reader, ResourceLoader resourceLoader) throws XMLStreamException, ConfigurationLoadException { assert SERVICE.equals(reader.getName()); Service service = factory.createService(); service.setName(reader.getAttributeValue(null, "name")); while (true) { switch (reader.next()) { case START_ELEMENT: AssemblyModelObject o = registry.load(reader, resourceLoader); if (o instanceof ServiceContract) { service.setServiceContract((ServiceContract) o); } reader.next(); break; case END_ELEMENT: return service; } } } and the <interface.java> handler is public JavaServiceContract load(XMLStreamReader reader, ResourceLoader resourceLoader) throws XMLStreamException, ConfigurationLoadException { assert AssemblyConstants.INTERFACE_JAVA.equals(reader.getName()); JavaServiceContract serviceContract = factory.createJavaServiceContract(); serviceContract.setScope(Scope.INSTANCE); serviceContract.setInterfaceName(reader.getAttributeValue(null, "interface")); serviceContract.setCallbackInterfaceName(reader.getAttributeValue(null, "callbackInterface")); return serviceContract; } > I am not completely opposed to a StAX based loader approach. I can even > see some benefits: > - smaller code and memory footprint compared to a solution based on > generated code > - faster loading (I'm actually not sure how much we'll gain on typical > SCA module file, but I'm guessing that it'll be faster) > - and more important IMO... more flexible parsing (for example we could > relax a little the requirement for some of the namespaces to be > specified in SCDL, this could help simplify SCDL files a little) > > But I'm concerned that we're going to have to spent a lot of energy to > make it really work well in the long term (energy which could be spent > on many other aspects we need to cover in Tuscany). Basically I will be > OK with this StAX loader approach only if we have enough people in the > group really volunteering to take responsiblity for it, maintain it, and > adjust it to all the upcoming changes to the XSD or the logical model. > People just need to think about it and realize that it's going to be a > lot of work. > I think we need to take responsibility for whichever option we choose. Maintaining the loading code in the face of an evolving spec is going to require work from all of us. The choice here comes down to what are we willing to look after: A) an SDO model, code generation, a SCDLContentModelHandler and its linker blocks, a set of SCDLModelLoader impls, and Thread classloader management code, or B) a set of StAXElementLoader's As I said at the start, I wasn't even looking at this until we ran into the Thread classloader problems during the Tomcat integration. I still don't think we have resolved all of them and think we are going to run into a new set of challenges with other property binding frameworks. I would like to put this to rest soon. Both versions are out there for folk to look at and they are both complete enough to run our current examples. I think the StAX version is simpler yet just as flexible, avoids some of the problems seen to date, and provides an easy integration path with other frameworks (including SDO) and unless there is a strong technical argument against I'd like to switch over. -- Jeremy
