Thanks Pete, good to hear from you again, I must admit it's been too
long. We last spoke when I was refactoring a class dependency tool to
use ASM instead of the jdk's tools.jar. You once asked, how do you find
a dependency for calls to Class.forName?
The reasons you've stated are also why I've chosen to support a very
narrow use case, in which you may have already noted that the serialized
connection is between two identical bundles, in separate jvm's with
compatible package imports.
There's no intent to support transferring any classes outside of the
Service API and that includes overriding classes. What you describe
about data hiding reminds me of Entry's, which have public fields.
I'll be the first to admit there are significant issues with the design
of Java's Serialization's extralinguistic api. Ironically though the
wire protocol is reasonably well thought out, with regards to evolution.
As an exercise to fix security issues, I have reimplemented Java
serialization with input validation using a public api, it has backward
compatible serial form, but only supports a subset of Java
serialization, it doesn't support circular object graphs for example as
this would compromise security. It performs input validation, sets
resource limits and expects periodic stream resets to avoid DOS and
gadget attacks. The problem is there's a lot of existing software that
utilises java Serialization, that's going to need support for some time.
Things like Serialization and Remote method invocation are damaged by
attempts to implement too much functionality, when a more rigid subset
would avoid a number of issues. But I guess no one was thinking of
modularity and versioning when they created these frameworks either.
James Gosling said something once about why Generics weren't included in
Java from the outset, which was because at the time they didn't know how
to do it properly, it's better leaving it out until you do.
JBoss has a nice web page with some graphics that illustrate some major
issues with implementation hiding you've mentioned with Serialization
and modular frameworks here:
https://developer.jboss.org/wiki/ModularSerialization
It's worth noting Service API of the smart proxy bundle doesn't need to
be Serializable, instead it's relegated to a communication means between
two identical bundles in different JVM's. It's also important to
recognise that it doesn't need to be the communcation mechanism either.
These bundles have an identical class namespace, although there may
be variances in package import versions.
Yes we are also looking at moving away from java serialization.
Also over time, because this is a service interface, at some point down
the track, serialization can be replaced, without impacting the public
api. So yes the underlying protocols can be stripped out to data and
message passing if that's more satisfactory.
So yes java serialization is an existing part of our application and it
has it's warts.
But we've also had a number of users over the years who have requested
support for OSGi.
This is not a greenfields project, I'm hoping that I'm not going to be
told that no, the chasm is too wide you can't cross over to OSGi,
rewrite or start again, there's just too many LOC.
So you have raised some important questions. Some of our users have had
a lot of success with Maven (recognising there a pro's and cons's with
module based versioning and transitive dependencies), where versioning
on a module level allows codebase annotations to be utilised in remote
invocation, avoiding class visibility issues by mapping module
ClassLoader's directly to a URI based identity. However with OSGi
there's a mismatch between different jvm's and how bundles and packages
imports will be resolved will end up being wired, so we can't rely on
codebase annotations for OSGi.
Jvm's using OSGi frameworks are quite likely to have different
dependency graphs (wires) between bundles and their package imports.
While I don't expect to solve the worlds problems or boil the ocean, I'm
looking for the most workable compromise, one that doesn't promise the
world and is easier to explain what users can and can't expect to do.
I'm relatively pragmatic. To me it would seem logical that a subset
where two identical bundles (that should have resolved similar package
import versions) should be a good place to start.
Hence my post on this list, as I realise many of you have already spent
a lot of time bumping into these issues.
Cheers,
Peter.
On 20/02/2017 6:38 PM, Peter Kriens wrote:
After working in this area for too many years I’ve come to the
conclusion that objects cannot be really transferred to other systems
in a reliable way, only self typed data can. JPA, RMI, and many other
systems promise heaven to the programmer that they can use their
objects local and remote transparently. The consequence of this dream
is a huge amount of complexity that far outweighs any gains in
programmer friendliness. Few things have caused so much trauma in the
software world as ORM. (Persistence is communications to a future
process.)
The reason objects are so complex to use in communications is that it
is in direct violation of the goal of OO to hide your data. However,
once you expose the internal data on the wire you have effectively
made it public but too many people they can still have the advantages
of abstract data types. OSGi is a bitch in this case because it tells
you that you’re trying to do something wrong by refusing to cooperate.
In this case, it balks at you because you create an invisible
dependency between the sender and the receiver. Though this is a good
thing too often the receivers of this message blame the messenger.
You can handle this dependency but you’ll find out is that it is a
hugely complex task that introduces a lot of frailty in the overall
system. Having tried this several times I can assure you that any
gains in programmer friendliness are dwarfed by the complexity of
creating this facade.
The best solution I found is to give up on data hiding. The fact your
objects is on the wire means that that wire format is public. I
therefore use Data Transfer Objects, in my case objects with public
fields. On both sides I have my own objects to provide behavior to
this data with methods and classes but this data record is at the core
of my code. Since this data is public because it goes over the wire it
is better to wrap you code around that ‘standardized public’ object
than to try you internal object data.
If you look at the OSGi specifications of the past 5 year then you
will notice that all applicable APIs have been designed to be useful
with Distributed OSGi. Calls do not pass objects but they pass DTOs
back and forth. They do not rely that the receiver and sender have
exactly the same type and version. In this model it is easy to replace
an endpoint using another language, which is a really good sign.
For Java developers this is often an unpleasant message, and quite
often OSGi get the blame. However, the fact OSGi gives you these
problems means that you’re trying to do something that has hidden
dependencies.
Distributed computing has 7 well known fallacies[1] but I strongly
believe that there is an eighth: ’One can communicate objects over a
network’.
Now your question. Yes, you could run a resolve and load the proper
bundles but you introduce a huge amount of error cases and a large
amount of complexity and you won’t solve the fundamental problem.
Kind regards,
Peter Kriens
[1]: https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing
On 20 Feb 2017, at 05:13, Peter <j...@zeus.net.au
<mailto:j...@zeus.net.au>> wrote:
Hello,
I'm currently working on converting an existing application to OSGi.
This application has a network service architecture based on java
interfaces. I've broken the application into modules, using a Maven
build, which uses bnd and bndtools to create bundle manifests. Some
of these modules are ServiceLoader provider's, so I've used
annotations to ensure these are loaded into the OSGi service registry
using the Service Loader Mediator.
The main issue that I face is this application is a networked
application and has it's own Remote Invocation protocols (which
currently utilise Java Serialization, but not Java RMI). As you'll
appreciate, class visiblity is a little different in OSGi. :)
The services mentioned above are remote services, these remote
services have a proxy which implements the service interface, these
services are discovered and installed at the client. There are two
types of proxy's, one, called a smart proxy, requires a codebase from
which to retrieve a jar or jar files that are downloaded and
installed at the cleint (traditionally during deserialization), the
other type of proxy is called a dynamic proxy (it's basically just an
instance of java.lang.reflect.Proxy), which is dynamically generated
at the client.
The Service implementation is broken up into three components:
1. The service api
2. The smart proxy (resolved and provisioned into in client jvm).
3. The server
The server bundle imports packages from the smart proxy bundle, while
the smart proxy imports packages from the service api as well as
exporting it's own packages, as required by the server bundle.
The server that provides the remote service has three bundles loaded;
server-impl, smart-proxy & service-api.
The client only has the service api bundle installed at deployment
and the smart proxy is resolved and provisioned before the service is
made available via the local OSGi service registry, where the client
will learn of it's existence using ServiceTracker.
At first glance only the smart proxy bundle needs to be provisioned
at the client, however for cases where a dynamic proxy is required to
implement interfaces from different packages, where class visibility
issues may exist, it may be beneficial in these cases to utilise and
provision a proxy bundle that imports all these interfaces, one might
do that by taking advantage of java's interface multiple inheritance;
create a bundle that contains one interface (annotated with
@ProviderType) which extends all interfaces, which the bundle doesn't
export, so we ensure that the dynamic proxy has a proper bundle
manifest with all package imports and version ranges correctly defined.
The inbuilt remote invocation protocol has server and client
endpoints, the protocol is extensible and has a number of
implementations (for example https, http, tls, kerberos, tcp). Each
endpoint is assigned a ClassLoader when it's created.
For classes installed at the client, these are typically installed in
a URLClassLoader, typically with the Application loader as parent
loader. In an OSGi environment however, the smart proxy bundle will
be installed at the client, it's ClassLoader utilised by the client
endpoint, the smart proxy bundle will also be installed at the server
and it's ClassLoader utilised by the server endpoint. In this case
the visibility of the bundles at each endpoint will be utilised to
resolve serializable classes. Private smart proxy serializable
classes will be resolvable at each end, but only public classes from
imported packages will be deserializable, since the client interacts
using the Service API, all serializable classes in the Service API
packages will need to be exported and public and imported by the
client and smart proxy.
Once a bundle has been provisioned its ClassLoader will be given to
the client endpoint and the marshalled state of the proxy
unmarshalled into it. At this point the service that the proxy
provides would be registered with the OSGi service registry for the
client to discover and consume. The smart proxy communicates with
it's server via an internal dynamic proxy (java.lang.reflect.Proxy),
it's used to invoke methods on the server.
While the existing protocol uses Java serialization, it doesn't use
Java serialization's method of resolving classes. Java Serialization
walks the stack and finds the first non system classloader (looking
for the application ClassLoader). The existing class resolution
method isn't suitable for OSGi, however the mechanism is extensible,
so can be replaced with something suitable.
Does anyone have any advise or experience utilising the OSGi
Enterprise Resolver Service Specification (chapter 136) and the OSGi
Enterprise Repository Service Specification (chapter 132) to resolve
and provision a bundle for the smart proxy at the client?
The intent here is the bundle manifests at each endpoint will be used
to determine class visiblity, so the resolution and provisioning
process will be of critical importance.
For anyone curios, the application is a fork of Apache River / Jini
and I'm experimenting with support for OSGi. I'm also a committer
and PMC member of Apache River. This isn't the old Jini we all know
and love however, there are some additional features that allow
provisioning to occur using a feature called delayed unmarshalling,
so we can avoid the need for codebase annotations and URLClassLoaders.
The work in progress can be found here, for anyone who's curious:
https://github.com/pfirmstone/JGDMS/tree/Maven_build/modularize/JGDMS
Regards,
Peter.
_______________________________________________
OSGi Developer Mail List
osgi-dev@mail.osgi.org
https://mail.osgi.org/mailman/listinfo/osgi-dev
_______________________________________________
OSGi Developer Mail List
osgi-dev@mail.osgi.org
https://mail.osgi.org/mailman/listinfo/osgi-dev
_______________________________________________
OSGi Developer Mail List
osgi-dev@mail.osgi.org
https://mail.osgi.org/mailman/listinfo/osgi-dev
