Does it mean there is no isolation between different modules that are dynamically loaded. What happens if there are conflicting jars? Can we control this isolation problematically ?
On Thu, Oct 13, 2011 at 9:13 AM, adam wojtuniak <[email protected]>wrote: > Hi Leo, > > Maybe it is possible with JBossModules, I haven't noticed that they are not > really OSGi container. > "JBoss Modules is designed to work with any existing library or application > without changes, and its simple naming and resolution strategy is what > makes > that possible. Unlike OSGi, JBoss Modules does not implement a container; > rather, it is a thin bootstrap wrapper for executing an application in a > modular environment. The moment your application takes control, the modular > environment is ready to load and link modules as needed. Furthermore, > modules are never loaded (not even for resolution purposes) until required > by a dependency, meaning that the performance of a modular application > depends only on the number of modules actually used (and when they are > used), rather than the total number of modules in the system. And, they may > be unloaded by the user at any time." > > I am going to investigate that what I can do with that solution. > > Regards, > Adam > > On Thu, Oct 13, 2011 at 4:44 PM, Leo Neumeyer <[email protected]> > wrote: > > > When I tried jboss modules, I didn't have to do any of this. Maybe it > > is worth investing more times in getting jboss modules to work. > > > > I started a jira for this topic here: > > https://issues.apache.org/jira/browse/S4-4 > > > > please add to this issue. > > > > -leo > > > > On Thu, Oct 13, 2011 at 3:23 AM, adam wojtuniak <[email protected] > > > > wrote: > > > Hi Leo, > > > > > > We can't avoid class loading issue even if we want only to load apps > > > dynamically. Every app is going to have dependencies to s4 (plus s4 > > > dependencies) and also can happen that app can have their own > > dependencies, > > > all these dependencies must be visible to app classloader. One solution > > is > > > to expose all these dependencies as a system packages then they will be > > > visible to all bundles in the osgi container but that is a hack which > are > > > going to lead to maintenance nightmare (every time is new dependency it > > must > > > be added to system package). Class.forName is another problem, for > > example > > > kryo lib is loading classes which in the container aren't visible to > kryo > > > until kryo will be bundlized with DynamicImport-Package: *. These > > problems > > > can't be avoided and is no matter what solution is used (JBossModules > or > > > Felix etc.). One solution is to find proper bundlized version of some > s4 > > > dependencies or wrap them using http://www.aqute.biz/Code/Bnd and put > > for > > > some of them DynamicImport-Package: *. > > > > > > Cheers, > > > Adam > > > > > > On Wed, Oct 12, 2011 at 12:53 AM, Leo Neumeyer < > [email protected] > > >wrote: > > > > > >> Hi Adam, > > >> > > >> very nice. > > >> > > >> We are only interested in loading apps dynamically. Is it possible in > > OSGI > > >> to just use it for the apps. The rest of the jars can be loaded at > start > > >> time in the same way we do it now. > > >> > > >> Anyway, I will take a look. Can you post your comments to the dev list > > so > > >> everyone can see it and comment? > > >> > > >> [email protected]**che.org > > >> > > >> Thanks! > > >> -leo > > >> > > >> > > >> On Oct 11, 2011, at 4:41 PM, adam wojtuniak wrote: > > >> > > >> Hi Leo, > > >> > > >> Thanks for description. > > >> I've implemented ideas which I sent you before. Now I can load the > apps > > >> using embedded osgi container but there is still class loading issue > > >> (com.esotericsoftware.kryo.SerializationException: Unable to find > class: > > >> io.s4.example.counter.UserEvent). To be able to run s4 I had to put > > plenty > > >> system packages in configuration file for felix. > > >> Around 15 jars defined in libraries are not osgi bundles (don't have > > >> proper manifest file). The biggest issue are dependencies and > > classloading > > >> (for example Kryo is trying to load UserEvent class but that class is > > not > > >> visible for him so Class.forName is not going to work).I will try to > fix > > >> these problems, in the meantime you can have a look at my repo > > >> https://github.com/adamwojtuniak/s4-piper. I added task createRepo, > it > > >> will create bundles directory with all dependencies and project jars. > > When > > >> you launch Main class from s4-core, it launches embedded felix which > > will > > >> auto deploy all bundles located in bundles directory. In the console > you > > >> will see felix gogo shell ( > > >> http://www.packtpub.com/article/apache-felix-gogo ). > > >> > > >> Cheers, > > >> Adam > > >> > > >> On Sat, Oct 8, 2011 at 5:56 PM, Leo Neumeyer <[email protected] > > >wrote: > > >> > > >>> Responding to this: > > >>> > > >>> > Maybe you can write me some small description of s4 concept. > > >>> > > >>> Here is the very high level s4-piper (which is quite different than > s4 > > >>> v0.3). We expect the next version of S4 to be based on s4-piper. This > > >>> is work in progress! > > >>> > > >>> API: > > >>> > > >>> * Event: all messages sent between PEs inherit from Event. > > >>> * ProcessingElement: The unit that processes an event. > > >>> * Stream: Streams are the edges between PEs. A source pushes events > > >>> into streams and streams push events into target PEs. Streams are > > >>> associated with a specific Key. > > >>> * Key: A value derived from the content of an event. Could be the > > >>> value of a field, a combination of fields, or a more complex > > >>> transformation of the fields. Each PE instance is associated with a > > >>> unique key value. Keys can also be used to map PEs to partitions in a > > >>> cluster. > > >>> * KeyFinder: A helper class that Implements the actual logic used by > > the > > >>> Key. > > >>> * App: A unit that encapsulates a complex task carried out by a graph > > >>> comprised of PEs and Streams. The App has a reference to all the PE > > >>> and Stream objects associated with the graph. A deployment may have > > >>> several Apps that may communicate with each other using EventSource. > > >>> Apps are associated with owners and units of work. A deployment may > > >>> include Apps that are or are not related to each other. Apps can also > > >>> be used to implement security controls. By default, PEs and Streams > > >>> are not exported outside an App. > > >>> * Streamable: classes that accept S4 events must implement > streamable, > > >>> currently: Stream and EventSource. > > >>> * EventSource: use it to export events outside of an App. Any app in > > >>> the cluster can potentially subscribe to an event source at run time. > > >>> > > >>> Some internal classes: > > >>> > > >>> * Main: starts a node. > > >>> * Server: the server instance in a node. > > >>> * Sender: send an event to its destination in a cluster. > > >>> * Receiver: receive an event from somewhere in the cluster. > > >>> > > >>> Stories: > > >>> > > >>> PEs do all the application processing. PE use a prototype pattern > > >>> which is the key to the S4 design. The first PE object is called a PE > > >>> prototype which is configured for a specific task but will never be > > >>> used for actual processing. Events sent to PEs come with a specific > > >>> key. For example, key=user, key_value=joe. A PE instance, is > > >>> associated with a specific key, In this case the PE prototype will > > >>> check if PE for (user, joe) exists. If not, it will clone itself and > > >>> create a PE instance for (user, joe). The S4 framework guarantees > that > > >>> all future events of the same type sent to a given PE prototype will > > >>> always arrive at the same PE instance. In this way, we can now do > > >>> computation for joe. The PE instance may emit an output event sent to > > >>> another prototype with another key. An App is a graph of PEs and > > >>> Streams. > > >>> > > >>> PEs put events into Stream{s). > > >>> > > >>> Stream(s) put events into PEs. > > >>> > > >>> Stream(s) are configured for a specific PE target and key finder. > > >>> > > >>> To send a user event with field "user" and value "joe" to a UserPE we > > >>> do the following: > > >>> > > >>> - Define UserEvent that includes "user" field. > > >>> - Define a KeyFinder that extracts the value of field "user" (will > > >>> return "joe" in the example.) > > >>> - Define UserPE that accepts events of type UserEvent and does > > >>> something with it, say count. > > >>> - Define a Stream whose target is UserPE configured with the > KeyFinder > > >>> that gets the value of "user". > > >>> - Now put an object of type UserEvent into the stream and it will get > > >>> to the unique UserPE instance for (user, joe). The instance could be > > >>> anywhere in the cluster. > > >>> > > >>> Once a server is started, we expect it to run indefinitely until > there > > >>> is a failure or we need to upgrade the framework software or we > change > > >>> the cluster configuration. In S4, the cluster is symmetric, that is, > > >>> all the nodes are identical, with the same code base and the same > > >>> configuration. > > >>> > > >>> Applications are loaded dynamically after the cluster is started. > Most > > >>> apps need an EventSource to receive events. App must be able to find > > >>> EventSource and subscribe to it during initialization. > > >>> > > >>> When an app is unloaded, it will close its EventSource(s). This in > > >>> turn, will close all the streams that are subscribed. The apps that > > >>> own the streams are responsible for taking action once their > > >>> dependencies are gone. > > >>> > > >>> -leo > > >>> > > >>> > > >>> On Sat, Oct 8, 2011 at 8:37 AM, Leo Neumeyer <[email protected]> > > >>> wrote: > > >>> > [sending to the new s4-dev-subscribe list!] > > >>> > Thank you Adam, let me read the links you sent to understand this > > >>> better. > > >>> > BTW. I reorganized the project into subprojects to make it easier > to > > >>> > separate examples from the framework. > > >>> > To implement inter-app communication I added EventSource which an > app > > >>> can > > >>> > use to publish its output to other apps and other apps can > subscribe > > to. > > >>> We > > >>> > can also use it to build adaptors to non-s4 events. We can use the > > osgi > > >>> > mechanisms for declaring dependencies such as app2 depends on app1 > > and > > >>> for > > >>> > app2 to do: app1.subscribeStream(anApp2Stream) so app2 can get > events > > >>> from > > >>> > app1. Unless an app uses an EventSource, its events will not be > > >>> exported. > > >>> > This will create good isolation and make it easier to add a > security > > >>> layer > > >>> > around apps. Both Stream and EventSource implement Streamable. > > >>> > My code is here until we set up the apache > > >>> > svn. https://github.com/leoneu/s4-piper > > >>> > thanks! > > >>> > -leo > > >>> > > > >>> > On Oct 8, 2011, at 5:03 AM, adam wojtuniak wrote: > > >>> > > > >>> > Hi Leo, > > >>> > Yes I have a few ideas but I need couple days to implement them. > > >>> > First, is to build S4 and examples as proper OSGi bundles using > > gradle > > >>> osgi > > >>> > plugin http://gradle.org/osgi_plugin. > > >>> > Its using bnd tool http://www.aqute.biz/Bnd/Bnd. > > >>> > Then embed osgi container (using standard osgi api) in Controller > > >>> instead > > >>> > of JBoss modules > > >>> > see > > >>> > > > http://felix.apache.org/site/apache-felix-framework-launching-and-embedding.html > > >>> . > > >>> > and configure directory from which felix should auto deploy our > > >>> > examples > > >>> > > > http://felix.apache.org/site/apache-felix-framework-usage-documentation.html#ApacheFelixFrameworkUsageDocumentation-autodeploy > > >>> . > > >>> > For dynamic deployment we can use file install > > >>> > bundle http://felix.apache.org/site/apache-felix-file-install.html > , > > it > > >>> will > > >>> > watch directory from which it loads new bundles. > > >>> > To configure properly examples we can use > > >>> > ipojo http://felix.apache.org/site/apache-felix-ipojo.html or > > >>> declarative > > >>> > services > > >>> > > http://felix.apache.org/site/apache-felix-service-component-runtime.html > , > > >>> > they both using service component model and are not intrusive in > the > > >>> code. > > >>> > That will be enough to load properly the apps. > > >>> > Later we have to think how we want to use osgi in s4, just to load > > apps > > >>> or > > >>> > properly use service management. Bundles are hiding the internals > > from > > >>> other > > >>> > bundles and should communicate through well defined services. And > > also I > > >>> > don't have clear picture how s4 is going to be distributed (have to > > look > > >>> > more at https://github.com/s4/s4). Maybe you can write me some > small > > >>> > description of s4 concept. > > >>> > Cheers, > > >>> > Adam > > >>> > > > >>> > > > >>> > > >>> > > >>> > > >>> -- > > >>> > > >>> -leo > > >>> > > >> > > >> > > >> > > > > > > > > > > > -- > > > > -leo > > >
