Re: OSGi

Sat, 28 Jan 2017 17:35:47 -0800 

N.B. Mike, thanks for raising this issue.
It's worth noting that a backward compatible api upgrade may include new classes, this would be problematic for a proxy that depended on a later version, say v1.5.0, than a client that utilised v1.0.0, developers need to be aware of this restriction, when specifiying their bundle manifest. 


So a client using the lookup service would need to check their local 
installed version against the range and consider upgrading their api 
bundle before accessing a service that requires a later version.


Cheers,

Peter.

On 29/01/2017 11:18 AM, Peter wrote:

The best way to handle that situation would be to throw a 
ClassCastException, but also gather as much information as possible re 
Bundles and Bundle versions to assist the developer to debug and 
unentangle their design.



In this case BImpl was resolved first which has caused Child2Api 
version 1.0.0 to be installed.



If the package that Child2Api resides is in the version range 1.0.0 <= 
X < 2.0.0, then the only reason that a higher bundle version may have 
been chosen is that bundle 1.0.1 etc was already installed at the 
client, prior to 1.0.0 being installed and C2 has selected it during 
resolution.



This is also a good demonstration of why version ranges should be 
chosen, why would any sensible developer avoid an internal bug fix, or 
even a backward compatible api change in an api bundle, unless the 
bundle is broken?



If we try to ensure replication of wiring from one JVM in another, 
then if we don't have full control over both JVM's then we may 
introduce incompatibility at the root object, thus causing a 
ClassCastException when the caller in the deserializing jvm tries to 
assign it to Root.



We can't design a system for all scenarios, all systems have 
limitations and must make compromises.


Cheers,

Peter.

On 28/01/2017 10:46 PM, "Michał Kłeczek (XPro Sp. z o. o.)" wrote:

Sorry - you haven't shown how to avoid situation when a single 
instance of BImpl is assignable to Child1.myImpl but NOT assignable 
to Child2.myImpl.



This is simply NOT possible without having information about exact 
bundle wiring on the other end.


Thanks,
Michal

Peter wrote:

Hmm, bundle wiring, not sure, will think about it.

Yep, guilty of lazyness.

So continuing on, with the deserialization process...

The stream reads in the stream class information, now Root's 
ClassLoader is on the stack,

Child2 is loaded using Root's Bundle.
Next, Child2's bundle C2's ClassLoader is pushed onto the stack.

Child2's fields are now read from the stream, in this case a 
reference number is read in from the stream,
this reference number is looked up from the stream reference table, 
and the same instance of BImpl contained

in Class1's field is returned.

Serialization doesn't send an object twice, unless it's written 
unshared.
Now Class2's instance is created, it's ClassLoader popped off the 
stack, root's second field is written, it's ClassLoader popped off 
the stack and its instance returned.



If Class2's field was another instance of BImpl, the class is stored 
in the ObjectStreamClass's cache and has been verified and loaded, 
so we don't need to resolve it again.  If BImpl was actually a 
different version and was loaded by Bundle BImpl2.0.0, then the 
annotation won't imply BImpl1.0.0,

so it will load a new bundle.


The Bundle wiring will depend on a combination of the local Bundle's 
or downloaded Bundle's manifest package import declarations.



The wiring may be slightly different at each end.  A PackageAdmin 
service may decide to change the loaded bundles, we need to ensure 
that we allow them to communicate accross compatible versions.


Hope this helps clarify it a little better.

Cheers & thanks,

Peter.

On 28/01/2017 10:11 PM, "Michał Kłeczek (XPro Sp. z o. o.)" wrote:

Ahh... You've missed the important part :) - child2.

You cannot assume BImpl class loaded in context of C1 is assignable 
to child2 - it depends on what other unrelated bundles have been 
loaded earlier and how the container chooses to resolve them.


Imagine:

BImpl has:
Require-Bundle bundle.with.exact.child2Api.version 1.0.0

Child2 has:
Import-Package packageOf.child2Api range [1.0.0, 2.0.0)


It might be perfectly fine in one container where only 
bundle.with.exact.child2Api.version 1.0.0 was installed. But might 
be wrong in another where there are two "packageOf.child2Api" 
exporting bundles installed.


So:

1. The information about BImpl is no longer on the stack and not 
available when loading Child2
2. Even if it was available (because for example you modify your 
algorithm so that you remember all previously loaded bundles) - it 
means that when loading Child2 you no longer can depend on the 
container bundle resolution.



And 2) is the point I am trying to make from the beginning: when 
transferring object graphs you MUST serialize the BundleWiring 
information and recreate it when deserializing. You need the 
snapshot of runtime state - declarative dependency information is 
not enough.


Thanks,
Michal

Peter wrote:

Ah yes, that's easy, assuming all classes are Serializable :)


Firstly we register a BundleListener with our OSGi framework and 
keep track of installed Bundles.



So Root, is the object graph root, Classes Child1 and Child2 are 
visible to Root.

The first thing we do is push the ClassLoader of Root onto our stack.

When the stream reads in the fields of Root, the first field 
child1, it will read the stream class information, the class 
Child1 will resolve from Bundle BR, since it is visible from there.
Now we obtain Child1's ClassLoader (which provides a 
BundleReference to Bundle C1) and push it onto our stack.
The stream reads in the fields of Child1, the first field 
Child1Api, is an instance of BImpl, it first attempts to use 
Bundle C1, however assuming that BImpl isn't visible from Bundle 
C1, we use the codebase annotation we read in from the stream 
(read in for each class), to find any installed bundles implied by 
the annotation's URI.
Upon finding BImpl's bundle from the BundleContext, we first ask 
the bundle to load the class, then we check that the class is 
assignable to field myImpl in Child1, before Child1 is 
instantiated, using class.isAssignableFrom.
If this fails and there are other versions of BImpl installed we 
might iterate through them until we have a match.
If there are no BImpl bundles installed, then we use the 
annotation to load a new Bundle.

At this time BImpl's ClassLoader is pushed onto the stack.

However BImpl has no fields to deserialize, so a new instance of 
BImpl is created and before it is used to set the field myImpl in 
Child1, BImpl's ClassLoader is popped off the stack.
Now that all fields of Child1 have been created, Child1 can be 
created, and bundle C1's ClassLoader is popped off the stack.

The same process is repeated for Child2.

Cheers,

Peter.




On 28/01/2017 7:41 PM, "Michał Kłeczek (XPro Sp. z o. o.)" wrote:

I fail to see how it could possibly work. Could you walk 
step-by-step serialize/deserialize with the following object graph:


Bundle API:
interface Api {}

Bundle BR:
class Root {
  Child1 child1;
  Child2 child2;

  Api getApi() {
    return isEven(getRandom()) ? child1.impl : child2.impl;
  }

}

Bundle C1
class Child1 {
  Child1Api myImpl;
}

Bundle C2
class Child2 {
  Child2Api myImpl;
}

Bundle Api1:
interface Child1Api extends Api {}

Bundle Api2:
interface Child2Api extends Api {}

Bundle BImpl:
class Impl implements Child1Api, Child2Api {}

Object graph:
impl = new BImpl()
root = new Root(new Child1(impl), new Child2(impl));

Serialize and deserialize root.

Thanks,
Michal

Peter wrote:

So here's how we can put it together:


Our OIS contains a stack we can use to track ClassLoader's at 
each branch in our serialized object graph.



  1. First object being read in by the OIS, check the cache see 
if the

     URI annotation is implied.
  2. If yes, use this Bundle to deserialize, place this Bundle's
     BundleReference [ClassLoader] on the top of the stack.

  3. If no, walk the stack, find the first BundleReference, 
dereference

     it's Bundle, obtain the BundleContext and request a new Bundle
     with the new URL.
  4. For each field in the current object:

       * Try to load it's class from the current Bundle .

       * If successful push Bundle's ClassLoader onto stack and 
return

         class.

       * Catch ClassNotFoundException, it's likely this is a 
dependency

         injected class the parent object's ClassLoader or Bundle
         doesn't have type visibility for.

       * Check the Bundle cache to see if the URI annotation is 
implied.

       * If yes, then iterate through each, load a class from this

         bundle, check that the returned class can be assigned 
to the
         current field's type.  If not, discard and continue 
until a

         class is found that can be assigned to the current field.
       * If not, load the new bundle.

       * Push the current Bundle's ClassLoader on the top of the 
stack.



  5. Pop the last (Object in the graph) field's ClassLoader off 
the stack.



When the process completes we have a resolved object graph with 
correct class visiblity.


Regards,

Peter.

On 28/01/2017 1:49 PM, Peter wrote:

Having implemented an ObjectInputStream (OIS), I can share the 
following insights:


   * The object at the head of the graph is externally visible and
     encapsulates all other objects and primitives.

   * The objects and primitives at the tails of the graph are 
fields.
   * Circular references in object graphs are unsafe with 
untrusted

     input...

   * A serialized object graph without circular references is a 
tree.

   * Fields may have been dependency injected.

   * Each branch in the tree has an object at the head of the 
branch.
   * The OIS knows the types of fields in the local class as 
well as
     the types of fields in the deserialized class before it 
has been

     resolved.


The information required to deserialized an OSGi graph is in 
the OIS implementation.  The OIS implementation is not visible 
to the extending MarshalInputStream implementation.


MarshalInputStream contains a ClassLoader field, defaultLoader.


So the codebase annotation is critical for the identity of the 
first object,


Regards,

Peter.

On 28/01/2017 3:39 AM, Bharath Kumar wrote:

Yes Peter. Usage of thread context class loader is discouraged 
in OSGi

environment.


http://njbartlett.name/2012/10/23/dreaded-thread-context-classloader.html 




Some of the problems are hard to solve in OSGi environment. 
For example,
creating dynamic java proxy from 2 or more interfaces that are 
located in

different bundles.


http://blog.osgi.org/2008/08/classy-solutions-to-tricky-proxies.html?m=1 




This problem can be solved using composite class loader. But 
it is
difficult to write it correctly. Because OSGi environment is 
dynamic.



I believe that it is possible to provide enough abstraction in 
river code,
so that service developers don't even require to use context 
class loader

in their services.



Thanks&  Regards,
Bharath


On 27-Jan-2017 6:25 PM, "Peter"<j...@zeus.net.au>  wrote:


Thanks Gregg,

Thoughts inline below.

Cheers,

Peter.


On 27/01/2017 12:35 AM, Gregg Wonderly wrote:


Is there any thought here about how a single client might 
use both an

OSGi deployed service and a conventionally deployed service?


Not yet, I'm currently considering how to support OSGi by 
implementing an

RMIClassLoaderSPI, similar to how Dennis has for Maven in Rio.


I think once a good understanding of OSGi has developed, we 
can consider
how an implementation could support that, possibly by 
exploiting something

like Pax URL built into PreferredClassProvider.



   The ContextClassLoader is a good abstraction mechanism for 
finding “the”

approriate class loader.  It allows applications to deploy a 
composite
class loader in some form that would be able to resolve 
classes from many

sources and even provide things like preferred classes.


Yes, it works well for conventional frameworks and is 
utilised by
PreferredClassProvider, but it's use in OSGi is discouraged, 
I'd like to

consider how it's use can be avoided in an OSGi env.



In a Java desktop application, would a transition from a 
background
thread, interacting with a service to get an object from a 
service which is
not completely resolved to applicable loaders still resolve 
correctly in an
EventDispatch Thread?  That event dispatch thread can have 
the context
class loader set on it by the thread which got the object, 
to be the class
loader of the service object, to make sure that the 
resolution of classes
happens with the correct class loader such that there will 
not be a problem
with the service object having one set of definitions and 
another service
or the application classpath having a conflicting class 
definition by the

same name.


I’ve had to spend quite a bit of time to make sure that 
these scenarios

work correctly in my Swing applications.


Have you got more information?  I'm guessing this relates to 
delayed

unmarshalling into the EventDispatch thread.


It's early days yet, I'm still working it out what 
information is required
to resolve the correct ClassLoaders&  bundles, but this is an 
important
question, Bharath mentioned Entry's can be utilised for 
versioning and this

seems like a good idea.

What follows are thoughts and observations.


A bundle can be created from a URL, provided the codebase the 
URL refers
to has an OSGi bundle manifest, so this could allow any of 
the existing URL
formats to deliver a proxy codebase for an OSGi framework.  
When OSGi loads
the bundle, the package dependencies will be wired up by the 
local env.  If
the URL doesn't reference a bundle, then we could use 
Bharath's approach
and subclass the client's ClassLoader, this does make all the 
clients
classes visible to the proxy however, but that would happen 
anyway in a

standard Jini / River environment.


OSGi gives preference to already loaded bundles when 
resolving package
dependencies, so the client should be careful not to 
unmarshall any proxy's
that might require a higher version than the bundle already 
installed when

the client bundle resolved its dependencies.


One of the proxy bundle dependencies will be the service api 
bundle.  The
proxy bundle can limit the service api package / packages to 
a specific
version or version range, which it could advertise in an 
Entry.  Boolean
logic comparison of Entry's would have to be performed 
locally, after
matching on the service type (this requires delayed 
unmarshalling to

prevent the resolution of unwanted versions).


So, the OSGi bundles installed in remote JVM's communicating 
with each
other, may not be exactly the same version, but should be 
compatible.
Multiple versions of a package or bundle may be available in 
a JVM, so we

need to ensure we've selected one imported by the proxy.


We could define a simple rule, that the first URL in an 
annotation, must
be the proxy bundle, with all dependencies to be provisioned 
by the OSGi
Framework, which would allow the same codebase annotation to 
be utilised in
a non OSGi environment, but this will probably mean the loss 
of any

trailing URL annotations if the proxy's remarshalled.


Once the proxy bundle has been loaded, its ClassLoader needs 
to be the
default for the remaining stream classes (smart proxy 
fields), since it
knows the most about what dependencies are required, these 
will be visible

to the proxy's ClassLoader.


If a class doesn't have an existing ClassLoader referenced by 
annotatation
and cannot be resolved from the proxy's Bundle, it probably 
requres a new

bundle to be resolved.


What's next?  Determine what annotations to retrieve from 
OSGi Bundles.



How PreferredClassProvider currently obtains codebase 
annotations (Trivia:
the application/system ClassLoader is a URLClassLoader up to 
Java 8, but is

no longer an instance of URLClassLoader in Java 9):

Summary, If ClassLoader:


   1. Is local loader? return java.rmi.server.codebase 
property or null

      if not defined.
   2. Is instance of ClassAnnoation, get annotation and return.

   3. Is instance of URLClassLoader, get URL's as string and 
return.
   4. else return java.rmi.server.codebase property or null 
if not defined.



/**

      * Returns the annotation string for the specified class 
loader
      * (possibly null).  If check is true and the annotation 
would be
      * determined from an invocation of 
URLClassLoader.getURLs() on
      * the loader, only return the true annotation if the 
current
      * security context has permission to connect to all of 
the URLs.

      **/

     private String getLoaderAnnotation(ClassLoader loader, 
boolean check)

{

     if (isLocalLoader(loader)) {
         return getClassAnnotation(loader);
     }

         /*

      * Get the codebase URL path for the class loader, if it 
supports
      * such a notion (i.e., if it is a URLClassLoader or 
subclass).

      */
     String annotation = null;
     if (loader instanceof ClassAnnotation) {
         /*

          * If the class loader is one of our RMI class 
loaders, we have

          * already computed the class annotation string, and no
          * permissions are required to know the URLs.
          */

         annotation = ((ClassAnnotation) 
loader).getClassAnnotation();


     } else if (loader instanceof java.net.URLClassLoader) {
         try {

         URL[] urls = ((java.net.URLClassLoader) 
loader).getURLs();

         if (urls != null) {
             if (check) {
             SecurityManager sm = System.getSecurityManager();
             if (sm != null) {
                 Permissions perms = new Permissions();
                 for (int i = 0; i<  urls.length; i++) {
                 Permission p =
                     urls[i].openConnection().getPermission();
                 if (p != null) {
                     if (!perms.implies(p)) {
                     sm.checkPermission(p);
                     perms.add(p);
                     }
                 }
                 }
             }
             }
             annotation = PreferredClassLoader.urlsToPath(urls);
         }
         } catch (SecurityException e) {
         /*
          * If access was denied to the knowledge of the class
          * loader's URLs, fall back to the default behavior.
          */
         } catch (IOException e) {
         /*
          * This shouldn't happen, although it is declared to be

          * thrown by openConnection() and getPermission().  
If it
          * does happen, forget about this class loader's 
URLs and

          * fall back to the default behavior.
          */
         }
     }

     if (annotation != null) {
         return annotation;
     } else {
         return getClassAnnotation(loader);
     }
     }

/**

      * Returns the annotation string for the specified class 
loader.

      *

      *<p>This method is invoked in order to determine the 
annotation
      * string for the system class loader, an ancestor of 
the system
      * class loader, any class loader that is not an 
instance of
      * {@link ClassAnnotation} or {@link URLClassLoader}, or 
(for an

      * invocation of {@link #getClassAnnotation(Class)

      * getClassAnnotation(Class)}) 
a<code>URLClassLoader</code>  for

      * which the current security context does not have the
      * permissions necessary to connect to all of its URLs.
      *

      *<p><code>PreferredClassProvider</code>  implements 
this method

      * as follows:
      *
      *<p>This method returns the value of the system property

      *<code>"java.rmi.server.codebase"</code>  (or possibly 
an earlier

      * cached value).
      *

      * @param loader the class loader to obtain the 
annotation string

      * for
      *
      * @return the annotation string for the class loader, or
      *<code>null</code>
      **/
     protected String getClassAnnotation(ClassLoader loader) {
     checkInitialized();
     return codebaseProperty;
     }


Gregg

On Jan 20, 2017, at 6:08 PM, Peter<j...@zeus.net.au>   wrote:

Looking at your modifications to ServiceDiscoveryManager, I 
noticed
you've made changes to set the context ClassLoader prior to 
calling the

lookup service.


This is eventually utilised by PreferredClassProvider to 
lookup the
necessary loader to utilise for deserialization of the 
lookup results.



I think if we develop a RMIClassLoader provider for OSGi, 
we can avoid

utilising the context ClassLoader.


Since all OSGi ClassLoader's are instances of 
BundleReference, it's easy
to utilise OSGi bundle url anotations (I think this needs 
to incorporate
bundle versions).  I'd also like to utilise Java 9 jrt 
style URL's.


Cheers,

Peter.

On 20/01/2017 11:09 PM, Bharath Kumar wrote:


Thanks Peter for the review.


While creating this POC, I tried to make RIO framework as 
set of OSGI.
bundles. Rio project extends LookupDiscoveryManager class 
in one of the
class 
.org.rioproject.impl.client.DiscoveryManagementPool.SharedDi

scoveryManager.
That's why I removed the final modifier.


Regarding groovy files,

I have made the org.apache.river as system fragment 
bundle. So we can't
import any packages from other bundles. But we can use 
system bundle's
packages,. That's why i removed groovy files. If we use 
these groovy

files,

we need to import packages from groovy bundle which is not 
possible

here. I
will check JGDMS to see how it is used.


Thanks&    Regards,
Bharath



On Fri, Jan 20, 2017 at 6:09 PM, 
Peter<j...@zeus.net.au>    wrote:


Hi Bharath,

Re your changes (I've found so far):

LookupDiscoveryManager is non final, I'm interested why?


BasicInvocationDispatcher, you've set the context class 
loader around a
block of code, to use the ClassLoader passed in during 
construction.

I'm

currently investigating addong methods where ClassLoader 
can be passed

in
for OSGi.


Regarding bundle structure, I've restructured the layout 
here (so you

don't need to delete Groovy config):


https://github.com/pfirmstone/JGDMS/tree/Maven_build/modularize/JGDMS 




The full commit history has been retained, so u can see 
all changes.


Cheers,

Peter.

Sent from my Samsung device.

   Include original message
---- Original message ----
From: Bharath Kumar<bharathkuma...@gmail.com>
Sent: 20/01/2017 09:42:38 pm
To: dev@river.apache.org
Subject: Re: OSGi

Hello all,


I have also added a package in org.apache.river bundle to 
create the

river
service in osgi environment ( Here RIver
uses NonActivatableServiceDescriptor).

package name is  org.apache.river.start.ext


As river bundle is system fragment, i have to remove
the groovy dependency.
So i removed groovy files.

net.jini.config.Component.groovy

net.jini.config.GroovyConfig.groovy



Thanks&    Regards,

Bharath



















































					Reply via email to