A strong “second” to the OSGi discussion. And not so much support for the maven requirement. As for the war file, I’ll have to discuss with my teammates.
For OSGi: our environment has moved to an extensible and configurable set of OSGi bundles as pluggable components. With good sets of manifests, we can manage a fairly complex set of components without mu fuss. And the combination of OSGi and eclipse is quite powerful - so much of OSGi component development is not just supported but also automated. If one develops in eclipse, it’s a short hop to building from command line, using the same dependency declarations (many of which are automatically added by eclipse). As part of this, we were able to wrap all of jena as an OSGi bundle and use that to support the development and use of our other bundles. That’s not a great achievement, but is a great convenience. Now to Fuseki, which is a centerpiece of our system. I’ve been asked by our tech lead whether we can do something about Fuseki being a separately invoked, standalone server, separate from the rest of our components. What we need is a way for us to bring Fuseki up as just another plugin, to test for its presence, monitor its health and, when necessary ”kick-restart” it when it hangs :( Another appeal is to be able to send updates directly to Fuseki not via http. It we be great if service calls could be made directly to an OSGi component, sending a model, rather than requiring that model to be serialized and posted. Of course, for Fuseki to be a proper endpoint it would still need to interact via http. But in our case, I’m thinking that our application would be the sole source of updates, so if it would simplify matters I would be willing to tolerate switching off http updates in favor of updates via service invocations. This might even address my problems with too-fast updates(?). I hope this makes some sense. I’ll discuss this with my teammates and will send clarifications. Thanks for asking. Mark On Jun 19, 2014, at 11:31 AM, Andy Seaborne <[email protected]> wrote: > Stu B22, > >> A) AMQP / JMS publishing of graph change notices >> B) OSGi component bundling > > Both of those would be interesting. > > Fuseki2 (or should that be Joseki5?) tries to abstract idea of an action from > the coupling to HTTP but it was as clean as it might be 0 things leak back in > HTTP like getting the status codes right. Those two items don't depend on > the SPARQL action side of things so should be cleaner to have. > > A general framework for changes, including driving RDF patch, is going to > arise sometime. > > For OSGi packaging, adding the creation of OGSi components makes sense - > there will need to be several fuseki packaging maven modules. At the moment > distribution and war file building are all together but that is creaking. > > What it does need is expertise in OGSi - there is no point in someone just > learnign enough OGSi to get it working for it to then be inflexible and > likely rot over time. Apache Clerezza uses OGSi components for some of Jena > for core/arq/tdb - I wonder if adding build steps for OGSi components to the > main build would help everyone. > > Andy > > On 16/06/14 18:37, Stu Btwotwo wrote: >> >> Hello Jena-Users, >> >> In the jena-dev mailing list, on March 31, Andy announced a preliminary >> build of Fuseki 2: >> >> http://mail-archives.apache.org/mod_mbox/jena-dev/201403.mbox/%3C53394253.1070506%40apache.org%3E >> >> >> He ended that post with the following invitation: “If you integrate >> Fuseki with your own systems, >> then it would be good to discuss how to make that easier because now is >> a good time to make >> changes.” >> >> Here I will address that topic from my team’s point of view. I see this >> topic as somewhere near the >> boundary of “jena-users” and “jena-dev”, so perhaps we can start the >> discussion here in the user >> space, and then promote to the dev space as warranted. >> >> We use RDF and Jena as the symbolic data backbone of our Glue-AI social >> robotics platform, >> http://glue.ai. It contains a sub-project called http://appdapter.org >> which largely consists >> of wrappers of Jena features. >> >> There are several technology connection issues we face that I think may >> be of general interest. >> To boil down our platform as succinctly as I can: >> >> 1) Code is in Java and Scala >> >> 2) Build, deploy, and distribute using Maven >> >> 3) Semantic data access using Jena (ARQ, Fuseki), persistence using TDB >> and AllegroGraph. >> >> 4) Packaging of components using OSGi bundles (run under Felix or other >> container). >> >> 5) Webapp deployment using the Ops4j PAX-Web OSGi wrapper for Jetty. >> (This component is quite spiffy, and demonstrates some of the power of >> the OSGi platform). >> https://ops4j1.jira.com/wiki/display/paxweb/Pax+Web >> >> 6) Semantic data change notification and other (non-semantic) data >> plumbing using AMQP pub-sub Topics, implemented by Apache QPid, which is >> compatible with JMS 1.1. >> http://qpid.apache.org/ >> >> The interesting intersections occur between Jena/Fuseki #3, OSGi >> PAX-Web webapp deployment #5, >> and AMQP/QPid/JMS topics #6. >> >> The component we really want to run is an HTTP-enabled SPARQL(+Update) >> engine with TDB persistence >> (i.e. Fuseki), deployed as a bundle to be discovered and exposed to the >> web by PAX-Web, >> and able to supply AMQP notifications when graphs change. Our initial >> approach to the latter is to >> publish to a single Topic called “GraphChangeNotice”, using a range of >> QPid “subjects” corresponding >> to the different graphs in our datasets. Thus any consumer agent in our >> network can easily subscribe >> to notices on the graphs it is interested in. An unfinished design area >> is the exact representation >> of graph change notices; many forms are possible but they are naturally >> all similar to a bag of Quads >> plus some metadata indicating when, where, and how the change occurred. >> There is also a natural >> relationship between these notifications and the transaction model used >> by our ARQ datasets. >> >> We have some prototypes of each aspect of this component I have >> described, but we do not yet have a >> complete implementation, in part because of challenges stemming from the >> structure of older >> versions of Fuseki and Joseki. Of course we can always hack the >> features we want on top of >> Fuseki source, but then we worry about the cost of maintaining >> compatibility with future versions >> of the original. >> >> The new ability of Fuseki 2 to compile into a deployable .WAR webapp >> will help us somewhat. From >> there we will be looking for ways to layer on the kind of graph >> monitoring feature that Andy kindly >> sketched out for us in this thread from May 2014: >> “Notification of Graph Changes by ARQ+TDB, during or after SPARQL Update” >> http://mail-archives.apache.org/mod_mbox/jena-users/201405.mbox/%3C5378C885.3030306%40apache.org%3E >> >> >> ...and then connect those notifications to our AMQP topics as I >> described above. >> >> Given all this context, I could make suggestions about what I would most >> like to see done with the >> Fuseki codebase going forward, but I think first it would be good to >> find some consensus about how >> valuable these two feature categories are seen to be (or not to be) by >> the Jena community: >> >> A) AMQP / JMS publishing of graph change notices >> B) OSGi component bundling >> >> The purpose and utility of #A is presumably obvious, while the benefits >> of #B are more complex and >> subtle. Overall, I think the entire Jena stack would benefit from >> some attention to the OSGi >> universe (which does not require breaking compatibility with the >> non-OSGi universe), but I don’t want >> to launch into a manifest-o about that unless other folks are really >> interested in the subject. >> >> I look forward to hearing what other Jena users and developers think >> about these topics! >> >> peace, >> >> Stu B22 >> >
