Re: River & OSGi

[Peter Firmstone]

Sam Chance wrote:
All,
I'm a relative simpleton software engineer compared to many of you on
this list. Nevertheless, I remain convinced the "group think"
continues to stagnate River. That situation seems unlikely to change
without intervention. I find myself rooting for those who get
chastised for offering ways to say "yes". Don't get me wrong! I love
you all - as brothers! :-) However, revolution seems in order.
  

I have no beef with you Sam, I liked the way that you and Gregg could 
see each other's points of view in recent discussions, we need more of 
that sort of discussion on this list, so we can all gain a better 
understanding.

I had avoided OSGi purely due to the controversy it generates on this 
list, however without the Service Pattern one cannot upgrade a package 
without first persisting everything and shutting down the entire JVM, 
then restarting.  At least OSGi allows you to stop a bundle and any 
dependents, persist what you need to then start with a later bundle 
version if desired, without having to persist or shut down the entire 
JVM.  I figured eventually someone would want the capability, although 
using OSGi does complicate development, there is already a significant 
volume of developers and products using the technology.  However it 
isn't for everyone so I don't want to make it mandatory.

You know, on the topic of OSGi, I'm thinking about an upgrade 
persistence solution for migrating / harvesting data from domain objects 
utilising Serialization but without enforcing the implementation of 
Serialization.  I could take advantage of the Java 6 
ObjectStreamClass.lookupAny(Class<?> cl) method to serialize a class 
that isn't Serializable.  I was thinking that the serialVersionUID could 
deliberately be different, where an upgrader could harvest the required 
data to be utilised in creation of new objects and class structures 
containing the old data.  This could  be done with a Package Upgrader 
Service (OSGi JVM local service not distributed) and Factory or Builders 
to generate the new class structure.  Instead of implements Serializable 
it could be implements Upgradable. 

Jini Serializable classes would be responsible for their own form, 
Package version metadata could be included and used as a minimum class 
version requirement, then it is possible to upgrade objects to new class 
files without heeding backward compatibility, using OSGi in the local 
JVM to upgrade packages as required.  This might all be done in memory 
for fast response, only upgrading objects as required perhaps.

Back to what your saying, I think it comes down to a vocal minority not 
doing their research and not respecting the position of others.  Some 
people also have to understand that this is a developer list, when 
you're formulating ideas, you want assistance with the formulation, not 
unyielding criticism, demanding answers, based on technicalities of 
manager techno jargon.  There will always be people who contribute 
nothing, looking for arguments, they are not welcome on this list.  As a 
committer I certainly won't be paying attention to the needs of people 
who attack.  My time here is purely voluntary.  People also need to 
understand that ideas expressed on dev lists aren't set in stone, they 
evolve as you solve implementation problems.  Sometimes you have to 
throw away an implementation and start again, you haven't lost, It's the 
learning process that counts.

If people have concerns about impacts of change in the River codebase, 
instead of attempting to squash idea's in their infancy, wait for the 
implementation, then raise an issue if it affects you.  You might notice 
that I've submitted patches for ideas I'm experimenting with, I haven't 
committed them, I don't want to be attacked just because I've proposed 
what appears to me to be a solution, after I have spent much time 
researching or pondering problems.   We can't all be experts, I'm not 
and who cares anyway.  You learn more acting humble than pretending you 
know everything.

If you can see an issue make a suggestion, assist the developer to solve 
the root cause, sometimes this requires example code to make it obvious, 
create an issue on Jira, send in a patch and as much information as 
possible, constructive criticism is welcome.  Asking questions is far 
more effective than criticism by the way, provided they're not loaded, 
it's the intent that matters.

Also note Apache is a meritocracy, which I happen to like.  If you 
haven't or contributed, code, documentation, patches or funds, don't 
expect to be able to take a seniority decision making position or make 
unreasonable demands of people who have.

N.B. We could use some assistance tidying up the documentation available 
through svn, please feel free to edit at your leisure and send patch 
updates.  This is very important.

More comments below.
As for the subject of this email, I must be missing something.
Lengthy, verbose and detailed narratives that most people probably
don't really digest pervade threads - at least the threads about River
& OSGi.

If I said something like the following, would it be concise and
accurate? Let's try.

OSGi is a service architecture for services in the same address space. [1]

Jini/River is a service architecture for distributed systems built out
of services separated by a network. [1]

How in the world can it be so hard to grasp - even embrace - that the
intersection of these two constructs is synergistic?
  

By getting caught up in buzzwords and not understanding the problems 
that each technology is attempting to solve.

I have an idea! Let the "Negative Nancys" hide and watch as Paremus -
who get it - lead the industry and literally change the paradigm of
application service architecture and provisioning.
  

I'm hoping they'd like to provide some assistance, I believe one of 
their engineers were going to make some comments?

Newton is under an incompatible AGPLv3 license. 

Perhaps there might be some functionality there that could be pushed up 
to River?

Cooperation - everybody benefits.

Best Regards,

Peter.

[1] Before you leap, be advised that these descriptions were provided
by Jim Waldo in the "Artima" link someone provided earlier.

In fact, I pulled the thread on the referenced articles. And if I
understood Peter Kriens' posts correctly, Jini dropped out of
prominence from the OSGi specs basically due to lack of passion - dare
I say leadership? - from the Jini community. While I like to consider
myself part of this community, there is no way I could write those
lengthy, verbose and detailed emails discussing things like
classloaders.

But those who can write those novels should be contributing to the
specifications and not acting like consultants: always finding ways to
say "no".
  
IMHO,

Sam


On 11/6/09, Mike McGrady <mmcgr...@topiatechnology.com> wrote:
  
I guess this means you would prefer not to address the issues and move
to personal attacks.  Weak but not atypical.  In fact many who follow
this will know me and will find this amusing.

Sent from my iPhone

On Nov 6, 2009, at 6:20 PM, Peter Firmstone <j...@zeus.net.au> wrote:

    
Michael,

I just went back over the list to see what contribution you've made
to River by reading some of your posts.  Your the bloke that gave
cause for Dan Creswell to resign from River.

This is a developer list, I'm trying to contribute something and get
this project moving again as have others like, Jukka, Jonathan, Tom
and Dennis among others.

This software won't affect you, the Felix crowd are quite competent
and won't consider accepting any patches that compromise their
distribution.  It wont affect existing users of River that don't
want to utilise OSGi either.  My suggestion to you is; if you don't
like it, don't use it, but don't try to roadblock the hard work of
others with rot.

Now if you don't mind I'd like to get something done.  If someone
knows this man, please take that iPhone off him, he's a nuisance.

Peter Firmstone.

Mike McGrady wrote:
      
I am getting ready to jump on an airplane, so will but initiate a
multipart reply.



Sent from my iPhone

On Nov 6, 2009, at 3:32 AM, Peter Firmstone <j...@zeus.net.au> wrote:

        
Mike McGrady wrote:
          
First, I would like to know what the "shared" "Service Pattern" is.
            
Sharing - Objects moved between jvm's in a distributed environment.
          
"Shared" in your previous post meant a shared pattern.
        
Your reading too much into it, "Service Pattern" as in Pattern
literature, I didn't refer to "Shared Pattern".

2. Jini utilises the Service Pattern for sharing distributed
   pluggable java software.

      
Distributed Service Pattern -  Dynamic Runtime Discovery - Lookup
of Objects that share a common Interface, receiving a proxy that
enables the transmission of as Parameters and Returned Methods
using Jeri / RMI / Serialization etc.
          
The terminology just changed,  "Service Pattern" now has
distributed added.  Please indicate what this "pattern" is where it
occurs in Felix.  Looks like a loose and inappropriate use of
"pattern" if you mean design pattern, like inapropriately calling
MVC a design pattern.

        
Second, I would like to note that in my opinion changing Felix,
which is architecturally on firm ground to accommodate Jini is a
huge mistake.
            
Why? Felix is designed to be embedded into systems, are you
suggesting Felix isn't suitable, would you prefer I use another
Modular infrastructure to manage package visibility?  If so which
one? The current alternative JSR-294 depends on Java 7 and hasn't
been released yet.  I fail to see why utilising mature code as
opposed to expending much time reimplementing when something
already exists would be a huge mistake?   I have considered
extending Classworlds, however this would require jvm restarts
when packages needed to be replaced.  The Java Package Version
Specification is optional and isn't widely implemented.
          
OSGi is designed with services that are not inherently remote and
are dynamic in that they appear and disappear in the course of a
runtime.  This is very different from "managing package
visibility".  You need to distinguish.modularity, lifecycle
management and services and to more appreciate their varied levels
of abstraction.

        
I am not convinced that this analysis understands the depth of
OSGi.  Certainly the module, lifecycle, service levels of
abstraction in OSGi seem to me to not coincide with the notion of
service in Jini.  This difference is greater when distribution is
considered.  I would suggest that no decisions be made without a
more developed and better expressed alternative than this.  Felix
does not need to compromise it's architectural integrity to
accommodate less mature concepts in Jini.
            
OSGi segregates bundles of software (jar files with metadata) into
separate classloaders, controls visibility of packages between
bundles based on compatibility information (metadata) in jar files
and enables the bundles to be stopped, replaced and restarted.
          
This is a serious but not unusual over simplification of OSGi.
This is the lowest level of OSGi.

        
To utilise OSGi, a Jini program must be distributed in a bundle
containing dependency information on package version imports.
          
This is ambiguous in important respects.

        
To be made available outside its Classloader (one for each bundle)
the visibility of Packages must be controlled, it's public API
Packages must be registered ie the bundle started.  Since Java
cannot reload class files when a Bundle is upgraded, it must
reside in another Classloader, so Classloaders are Bundle and
version specific.  The service pattern enables the bundle to be
rediscovered by identity locally within the JVM after an upgrade.
          
I don't think you understand what an OSGi service is.  An OSGi
service is a contract.

        
A Jini service must be exported and registered to be available for
lookup, if it is being managed by an OSGi framework it must first
have it's bundle started and registered.
          
What you mean by "managed" needs to be specified.  JIni
registration and OSGi registration are not the same of course.

        
Jini currently loads class files into classloaders based on the
local availability of classes or http coadebase URL origin,
imposing restrictions on sharing of Objects, this is suboptimal,
OSGi can assist in resolving this issue by providing a framework
that manages package visibility.  However I don't believe that
OSGi alone is enough in a distributed environment
          
What do you mean here?  OSGi is what it is.

        
Please provide me with facts so we can have a meaningful discussion,
          
I have no idea how this relates to this discussion.  What facts
about what?

        
from your reaction I detect your gut feel is telling you this
isn't the right approach, my gut feel is that it is. I've based my
reasoning on facts, I need some factual information to be
convinced otherwise.
          
What facts?  You have not explained what you mean and you speak
very loosely where precision is required, but yet make very
sweeping and important recommendations.  My mind, not my gut, does
not like this since we do important business with Felix and do not
want it fiddled with without clarity, understanding and open
discussion at depth.

I don't think, again, you understand the nature and proper
architectural positioning of OSGi services.  Rather, you seem to
focus on the sorts of things or understandings we could get from
sources like Wikipedia, I.e., superficial and misguided. Maybe your
understanding is deep and accurate but we have no way of knowing
and what you say indicates the opposite.

Ideas - nor facts - are at issue.

I have asked you to to tell me about your view of services on this
context, which is critical, and have gotten a gloss of liwer level
functionallity.

Let me remake my request that you explain what you initially meant
about Jini and OSGi services because it sounds all wrong to me.

Thanks.

        
No offense intended.
          
None taken - ditto.

        
Regards,

Peter.
          
Mike

Sent from my iPhone

On Nov 5, 2009, at 7:03 PM, Peter Firmstone <j...@zeus.net.au>
wrote:

            
Well,  I've had a good think about it, this is presently the way
I'd like to utilise OSGi:

First let me make one thing clear:

1. Both OSGi and Jini utilise the Service Pattern.
2. Jini utilises the Service Pattern for sharing distributed
  pluggable java software.
3. OSGi utilises the Service Pattern for making java software
  modular, versioned and pluggable without JVM restarts.

When I refer to Jini I'm referring to the Jini Platform, when I
refer to River, I'm referring to the Apache River implementation
of Jini.

Java Also uses the Service Pattern with SPI for 3rd Party
Vendors to Provide Pluggable implementations of JVM components
or extensions, eg JDBC, Encryption software etc. Gregg inspired
me by pointing out possible PreferredClassLoader changes
utilising the RMIClassLoaderSPI.

By embedding an OSGi implementation such as Felix into River in
a pluggable manner (usage is optional) River can support OSGi
applications and provide Jini Services to those local
applications.

Doing so will require modification to the Felix Codebase, which
is probably best managed as a Patch for a Particular Version of
Felix.

Additionally, I figure further modifying to Felix to support
interactions with Remote Codebase Services utilising Static
Analysis to identify Package API.  The Static Analysis will be
used for three reasons, one to identify compatibility between
software in addition to the Bundle & PackageVersion metadata,
two to identify later versions of import packages that are
compatible using a Subset of their API used by a Bundle.  Lastly
but perhaps most significantly to allow software to evolve
without fear of runtime exceptions caused by binary
incompatibilities.  Modification to Felix would be done in a
manner that didn't break compatibility with OSGi.

Let me make one thing clear, OSGi does not compete with River,
nor is OSGi's remote service standard intended to compete with
distributed frameworks, nor are they intending to implement a
distributed framework, the API is there to be utilised by
distributed frameworks, OSGi made a statement to this effect.
Benefits:

1. Existing OSGi Applications can be hosted by River.
2. Existing OSGi Applications can utilise Jini Services without
  modification to OSGi applications, in a pluggable fashion.
3. The River Platform (with OSGi felix embedded) is the minimum
  installation requirement (apart from Java), all sofware can be
  downloaded on demand from codebase services, guaranteed
  compatible, including additional security benefits from having
  bytecode API identified by Static Analysis from a code base
with a
  trust relationship.  The codebase would never execute uploaded
  bytecode, just analyse and distribute it.  Hence codebases are
  mediators/proxy's for disconnected or untrusted service
providers
  or clients.  All code and API apart from a minimal core platform
  code could evolve dynamically over time.
4. OSGi enables local bundles to be restarted, this would allow
River
  to locally update older bundle's when a need no longer exists to
  utilise an older version bundle or if another bundle requires a
  later version, complementing software evolution.


I know that this is perhaps a somewhat bold ambition, however I
believe it would increase the interest in the River project,
especially from the OSGi community.

There are plenty of details to work out, such as how to
implement persistence services for bundles and their import
packages over restarts, how to coordinate starting and stopping
of bundles that contain Jini remote services.  How to proxy Jini
Lookup services from within the OSGi framework to make these
services available to existing OSGi applications.  OSGi would
not be able to work with Jini Services that didn't themselves
utilise OSGi, Felix wouldn't know which bundles were required
for compatibility reasons.  That's where codebase services would
come to the rescue, by checking for known OSGi bundles that are
in fact compatible and substituting them.  If a compatible
bundle couldn't be found the codebase service would have to
create a bundle using available code.

One remaining concern I have is the approach of integrating
Felix into River.  I'd like to make the required changes within
Felix Pluggable components if possible, so that the existing
felix implementation would only require some minor changes that
we might be able to get them to accept.  That would allow us to
have a totally independent implementation which could continue
to work with future versions of Felix, and possibly other OSGi
implementations, in case word spreads about River in the OSGi
community.  I can't see that I could use OSGi for replacing OSGi
system components, but it might be possible using SPI,
ironically again another Service Pattern.

I acknowledge that the alternative method of making River a
bundle within OSGi is possible without changes to OSGi, however
it doesn't make it possible then for OSGi to utilise the
codebase services I'm interested in.  If someone can show me
that it is possible to do so I'll consider the option seriously.
Cheers,

Peter.