Complexity and Codebase annotations

[Peter Firmstone]

Hello,

Michael Warres identified a number of issues with class loading and 
codebase annotation loss, none of which were fully addressed. 
https://dl.acm.org/doi/pdf/10./1698139


I wanted to share with you an new JERI InvocationLayerFactory I've been 
working on, on and off for some time.


Existing implementations that marshal objects between two endpoints, 
associate a codebase annotation with every class in the stream.


I have done something much different.

First I don't send any codebase annotations in the stream.  I recognised 
that we shouldn't have a second class resolution mechanism, that only 
ClassLoaders should control class resolution and that each node may not 
only have different ClassLoader trees or maps, but have different ways 
of assigning and wiriing them up.


Instead I decided to make the codebase annotation a Configuration 
concern of the service.


Both endpoints of the service are wired up with ClassLoaders, the 
ServerEndpoint ClassLoader is defined in the Configuration by specifying 
a class (typically the services proxy implementation class) whose 
ClassLoader will be used for class resolution at the ServerEndpoint.


The client Endpoint is assigned a ClassLoader using a Service Provider 
interface  ProxyCodebaseSpi, which can be used to support modular 
environments.   The default service provider implementation assigns a 
PreferredClassLoader.   The codebase annotation is configured by the 
service implementation, the codebase annotation is obtained at the 
client using a bootstrap proxy that implements an interface called 
CodebaseAnnotation.


These marshalling streams are only responsible for resolving classes 
related to one service, if another service proxy is passed as a 
parameter to a service, it will be marshalled and resolved with it's own 
independent endpoints and client ClassLoader assigned specifically for 
that service.


Services now have independent class resolution determined by the 
ClassLoader's at their Endpoint's.


All services must implement the ProxyAccessor interface, whenever the 
stream detects an instance of ProxyAccessor, it obtains the service 
proxy from the service, then marshalls it into a MarshalledInstance.  
The stream also asks the service for a bootstrap proxy which implements 
CodebaseAccessor ( classes for which are always resolvable by the 
stream), then the CodebaseAccessor proxy is sent with the 
MarshalledInstance , over the marshalling stream, so as to avoid 
polluting the stream with classes from other services it cannot resolve.


When the MarshalledInstance is deserialized/unmarshalled by the client 
endpoint, the CodebaseAccessor is used to obtain the codebase 
annotation, a ClassLoader is created by the provider and the service 
proxy is unmarshalled using the ClassLoader provided.   The 
CodebaseAccessor proxy's InvocationHandler (which contains the 
ObjectEndpoint) is used along with the codebase annotation to represent 
the service's Identity.   This allows service proxy's to be assigned a 
different ClassLoader to other services who share identical annotations, 
this is important as the ClassLoader represents the ServerEndpoint's 
Subject and whatever permissions it's granted at the cleint.


Note that constraints are applied and authentication occurs prior to 
unmarshalling the proxy.


Modular environments can choose how to create this ClassLoader and how 
to resolve it, using the CodebaseAccessor (which can also provide code 
signer certificates, the codebase annotation and the identity of the 
service).


The default provider implementation PreferredProxyCodebaseProvider 
utilises PreferredClassLoader and typically uses the client's 
ClassLoader as the parent ClassLoader for the service proxy ClassLoader, 
so the service proxy has the same class visibility as the client (with 
the exception of preferred classes).   The default provider is not 
perfect for modular environments like Maven and OSGi, since it is a 
compromise that must function in all environments, but it is an 
acceptable compromise.


1. Firstly if a service proxy returns to a node from which it was
   exported, it will always be loaded by the ClassLoader from which it
   was exported, in this case it is the responsibility of the local
   environment to ensure visibility.
2. Secondly if a service proxy is unmarshalled by itself, it will not
   create a new ClassLoader, it must confirm the annotation doesn't
   match the stream loader.
3. Thirdly, if a proxy has been unmarshalled previously, has the same
   InvocationHandler, annotations and parent loader as the stream, it
   will be loaded by the cached ClassLoader.
4. A new ClassLoader will be created with the stream ClassLoader as
   it's parent loader (the client that's unmarshalled it).

This provides a reasonable implementation that works in all cases, in 
OSGi for example, different bundles may download the same service proxy, 
each will have a separate copy that resolves to the classes 

Re: Board feedback - Request discuss attic for River

[Peter Firmstone]

Thanks Dan. :)

On 5/21/2020 12:20 PM, Dan Rollo wrote:

Hi Peter,

I acknowledge the contributions are slow to come, but I agree with your 
observation that contributions are still coming. I would prefer the River 
project not be moved to the attic just yet. (“I don’t want to go on the cart. I 
feel better”….to soon? ;)

Dan


From: Peter Firmstone 
Subject: Board feedback - Request discuss attic for River
Date: May 20, 2020 at 8:51:42 PM EDT
To: dev@river.apache.org


Hello River Folk,

I've received feedback from the Board this morning, they are requesting that we 
discuss the Attic for River.

Personally I think the project still has a lot of potential to pick up again 
once the modular build is complete, and it is a useful place to send patches 
and discuss changes.

A very important patch was sent in June last year by Shawn Ellis for Java 
11.0.3 and later for services using SSL/TLS.

Another important change to the JERI protocol was discussed in September last 
year.

http://mail-archives.apache.org/mod_mbox/river-dev/201909.mbox/browser

What are your thoughts?

I don't think the board is asking that we send River to the attic, just that we 
discuss it.

Regards,

Peter.

Re: Board feedback - Request discuss attic for River

[Peter Firmstone]

Thanks Patricia,

I'm not sure we're going to see a significant number of developers 
contributing in the near future.


Unfortunately the technical matters, which are complex and difficult 
have caused a lot of argument in the past and have been the cause of a 
high barrier to entry for new developers.   At least that's what I 
think, feel free to provide your perspective.


I have been working on solutions to address some of the complexity.

I noticed that the board thought we hadn't had any commits for 3+ years, 
I'll be sure to add some commit statistics to the next board report.   I 
counted 191 commits over the last three years, not big, but not nothing 
either.


I think it's fair to say that we need Apache's infrastructure, our web 
site, Jira bug reporting system, repository and mailing lists.


The old 2.2 series code suffered from fragility due to race conditions 
and other bugs, understandably this made some existing developers very 
fearful of change (who had probably suffered from this fragility in the 
past) and the pace at which development was occurring had some 
frightened, which impacted our ability to work on the code, for this 
reason I don't talk too much about the code, as I fear the return of 
arguments of old, you might say I'm a bit shy.   In fact I am hoping 
that slowing down the pace of development as was requested has given 
people time to accept and adapt to the 3.0 release series.


I have been pinning my hopes on the Modular build to allow new 
developers to focus on smaller components without having to understand 
the whole project.


Also I think that River is constrained by the limitations of IPv4, I 
mean who only codes for the intranet these days?   Once we integrate 
multicast IPv6 support (I have been using this for at least two years 
now), we can communicate easily over the internet.


Another concern I have is security, we should have fixed security issues 
a long time ago, however the mood of development at the time didn't 
foster that, I think it was 2010 when I first flagged security issues 
with Serialization.   That's another reason why I've been hesitant to 
create bug fix releases, I don't think the code is good enough for a 
release without addressing some significant security issues first.


But clearly people are using the security features of River like SSL/ 
TLS, which indicates people are using River over the internet already, 
in spite of limitations with IPv4 NAT.  I have other fixes that address 
TLS security issues in River and bring cyphers and constraints into 
2020, rather than 2004 era cyphers.


Regards,

Peter.

On 5/21/2020 10:54 AM, Patricia Shanahan wrote:
The board tends to be more concerned about an active community than 
technical matters. We need to discuss whether there is a pool of 
potential contributors who are likely to become active.


On 5/20/2020 5:51 PM, Peter Firmstone wrote:

Hello River Folk,

I've received feedback from the Board this morning, they are 
requesting that we discuss the Attic for River.


Personally I think the project still has a lot of potential to pick 
up again once the modular build is complete, and it is a useful place 
to send patches and discuss changes.


A very important patch was sent in June last year by Shawn Ellis for 
Java 11.0.3 and later for services using SSL/TLS.


Another important change to the JERI protocol was discussed in 
September last year.


http://mail-archives.apache.org/mod_mbox/river-dev/201909.mbox/browser

What are your thoughts?

I don't think the board is asking that we send River to the attic, 
just that we discuss it.


Regards,

Peter.

Re: Board feedback - Request discuss attic for River

[Dennis Reedy]

Hi Peter,

Some quick late night thoughts ...

   - Time depending, helping a Gradle build for River is something that I'd
   be glad to participate in. I think one of the biggest challenges will be
   getting the testing framework working, or choosing a modern equivalent.
   - As it relates to combining Rio and River, I'd vote no. River has
   enough complexities/challenges, no reason to muddy the waters even more
   with Rio. Perhaps looking at how to create River components within a Spring
   Boot app might be more attractive, and open the an entire open source
   eco-system.
   - I have not seen the tide turning for dynamic code, my experience has
   been the opposite.
   - Something else to consider is to look at the level of effort for
   moving River to a more current Java version though, at least Java 8 if not
   11.

Regards

Dennis

On Fri, May 22, 2020 at 12:07 AM Peter Firmstone <
peter.firmst...@zeus.net.au> wrote:

> Hi Dennis,
>
> Good to hear from you.
>
> The truth is, I'm no build expert, Dan is much more capable than I am,
> so I'm waiting to hear Dan's opinion on Gradle, it certainly looks good.
>
> Would you be interested in helping to create a Gradle build?
>
> Something else I've been thinking about is you once proposed to
> integrate Rio with River, by the time I saw the proposal it had
> unfortunately already been shot down.   I was wondering if you were
> still interested in doing that?If so, what do others on the list
> think about it?
>
> I think these are good suggestions, but we'll need some help with
> presentation and communication.   I'm low level infrastructure focused,
> not so much at the application level.
>
> River's potential future strong points with IPv6 will be secure end to
> end dynamic discovery.   The tide seems to be turning for dynamic code,
> which was both a strength and an Achilles heel, is now looking a lot
> less like an Achillies heel and more like a strength again as originally
> envisioned.  Unfortunately today Android and Apple iOS and not really
> supportable as Jini clients, so I have doubts about consumer IOT, I
> think we might be better suited to industrial IoT, time will tell.   We
> will have the only pure Java Object based remote invocation framework
> that can integrate properly with OSGi, when support for it is added.   I
> think we've had much longer to understand problems with distributed
> computing and are overcoming them now.
>
> I'll reply some more later, given time to think some more...
>
> Regards,
>
> Peter.
>
> On 5/22/2020 3:20 AM, Dennis Reedy wrote:
> > I think showing/explaing how River can fit into a larger eco-system of
> > existing applications would certainly help. How could River augment
> Spring
> > Boot? What would it look like to combine River and Kafka? A discussion of
> > what it would mean to deploy micro-services built with RIver in the
> cloud?
> > What are the set of problems that River solves in 2020 that are not
> > solvable by other technologies? Is this about IoT? If so then the project
> > page should reflect that.
> >
> > As far as the modular build is concerned, sure it may help, but if it is
> > done, lets please use Gradle. A few years ago I put this example
> >  together, it may
> help
> > with that.
> >
> > On Thu, May 21, 2020 at 9:54 AM Bryan Thompson 
> wrote:
> >
> >> I think engagement requires a few things.  +1 on the technical points
> that
> >> you have called out Peter. But we also need to engage developers beyond
> >> those who are traditional users of River/Jini to foster new development
> >> with River. Which can then lead to new development of River by a broader
> >> base of developers.
> >>
> >> The modular build can definitely help create opportunities for
> developers
> >> to get engaged. But having examples of what they can achieve and how
> easily
> >> using modern security and IPV6 Is important for there to be any new
> >> developers using River.
> >>
> >> My question would be where are we on that roadmap. I know that we have
> been
> >> making progress towards this.  When will the project reach a state
> where it
> >> will engage a new group of developers and what actions will it take to
> help
> >> make that happen beyond the technology development?
> >>
> >> Bryan
> >>
> >> On Thu, May 21, 2020 at 01:46 Peter Firmstone <
> peter.firmst...@zeus.net.au
> >> wrote:
> >>
> >>> Thanks Patricia,
> >>>
> >>> I'm not sure we're going to see a significant number of developers
> >>> contributing in the near future.
> >>>
> >>> Unfortunately the technical matters, which are complex and difficult
> >>> have caused a lot of argument in the past and have been the cause of a
> >>> high barrier to entry for new developers.   At least that's what I
> >>> think, feel free to provide your perspective.
> >>>
> >>> I have been working on solutions to address some of the complexity.
> >>>
> >>> I noticed that the board thought we hadn't had any 

Re: Board feedback - Request discuss attic for River

[Peter Firmstone]

Hi Dennis,

Good to hear from you.

The truth is, I'm no build expert, Dan is much more capable than I am, 
so I'm waiting to hear Dan's opinion on Gradle, it certainly looks good.


Would you be interested in helping to create a Gradle build?

Something else I've been thinking about is you once proposed to 
integrate Rio with River, by the time I saw the proposal it had 
unfortunately already been shot down.   I was wondering if you were 
still interested in doing that?    If so, what do others on the list 
think about it?


I think these are good suggestions, but we'll need some help with 
presentation and communication.   I'm low level infrastructure focused, 
not so much at the application level.


River's potential future strong points with IPv6 will be secure end to 
end dynamic discovery.   The tide seems to be turning for dynamic code, 
which was both a strength and an Achilles heel, is now looking a lot 
less like an Achillies heel and more like a strength again as originally 
envisioned.  Unfortunately today Android and Apple iOS and not really 
supportable as Jini clients, so I have doubts about consumer IOT, I 
think we might be better suited to industrial IoT, time will tell.   We 
will have the only pure Java Object based remote invocation framework 
that can integrate properly with OSGi, when support for it is added.   I 
think we've had much longer to understand problems with distributed 
computing and are overcoming them now.


I'll reply some more later, given time to think some more...

Regards,

Peter.

On 5/22/2020 3:20 AM, Dennis Reedy wrote:

I think showing/explaing how River can fit into a larger eco-system of
existing applications would certainly help. How could River augment Spring
Boot? What would it look like to combine River and Kafka? A discussion of
what it would mean to deploy micro-services built with RIver in the cloud?
What are the set of problems that River solves in 2020 that are not
solvable by other technologies? Is this about IoT? If so then the project
page should reflect that.

As far as the modular build is concerned, sure it may help, but if it is
done, lets please use Gradle. A few years ago I put this example
 together, it may help
with that.

On Thu, May 21, 2020 at 9:54 AM Bryan Thompson  wrote:


I think engagement requires a few things.  +1 on the technical points that
you have called out Peter. But we also need to engage developers beyond
those who are traditional users of River/Jini to foster new development
with River. Which can then lead to new development of River by a broader
base of developers.

The modular build can definitely help create opportunities for developers
to get engaged. But having examples of what they can achieve and how easily
using modern security and IPV6 Is important for there to be any new
developers using River.

My question would be where are we on that roadmap. I know that we have been
making progress towards this.  When will the project reach a state where it
will engage a new group of developers and what actions will it take to help
make that happen beyond the technology development?

Bryan

On Thu, May 21, 2020 at 01:46 Peter Firmstone 
Thanks Patricia,

I'm not sure we're going to see a significant number of developers
contributing in the near future.

Unfortunately the technical matters, which are complex and difficult
have caused a lot of argument in the past and have been the cause of a
high barrier to entry for new developers.   At least that's what I
think, feel free to provide your perspective.

I have been working on solutions to address some of the complexity.

I noticed that the board thought we hadn't had any commits for 3+ years,
I'll be sure to add some commit statistics to the next board report.   I
counted 191 commits over the last three years, not big, but not nothing
either.

I think it's fair to say that we need Apache's infrastructure, our web
site, Jira bug reporting system, repository and mailing lists.

The old 2.2 series code suffered from fragility due to race conditions
and other bugs, understandably this made some existing developers very
fearful of change (who had probably suffered from this fragility in the
past) and the pace at which development was occurring had some
frightened, which impacted our ability to work on the code, for this
reason I don't talk too much about the code, as I fear the return of
arguments of old, you might say I'm a bit shy.   In fact I am hoping
that slowing down the pace of development as was requested has given
people time to accept and adapt to the 3.0 release series.

I have been pinning my hopes on the Modular build to allow new
developers to focus on smaller components without having to understand
the whole project.

Also I think that River is constrained by the limitations of IPv4, I
mean who only codes for the intranet these days?   Once we integrate
multicast IPv6 support (I have been using this for at least 

Re: Board feedback - Request discuss attic for River

[Dennis Reedy]

I think showing/explaing how River can fit into a larger eco-system of
existing applications would certainly help. How could River augment Spring
Boot? What would it look like to combine River and Kafka? A discussion of
what it would mean to deploy micro-services built with RIver in the cloud?
What are the set of problems that River solves in 2020 that are not
solvable by other technologies? Is this about IoT? If so then the project
page should reflect that.

As far as the modular build is concerned, sure it may help, but if it is
done, lets please use Gradle. A few years ago I put this example
 together, it may help
with that.

On Thu, May 21, 2020 at 9:54 AM Bryan Thompson  wrote:

> I think engagement requires a few things.  +1 on the technical points that
> you have called out Peter. But we also need to engage developers beyond
> those who are traditional users of River/Jini to foster new development
> with River. Which can then lead to new development of River by a broader
> base of developers.
>
> The modular build can definitely help create opportunities for developers
> to get engaged. But having examples of what they can achieve and how easily
> using modern security and IPV6 Is important for there to be any new
> developers using River.
>
> My question would be where are we on that roadmap. I know that we have been
> making progress towards this.  When will the project reach a state where it
> will engage a new group of developers and what actions will it take to help
> make that happen beyond the technology development?
>
> Bryan
>
> On Thu, May 21, 2020 at 01:46 Peter Firmstone  >
> wrote:
>
> > Thanks Patricia,
> >
> > I'm not sure we're going to see a significant number of developers
> > contributing in the near future.
> >
> > Unfortunately the technical matters, which are complex and difficult
> > have caused a lot of argument in the past and have been the cause of a
> > high barrier to entry for new developers.   At least that's what I
> > think, feel free to provide your perspective.
> >
> > I have been working on solutions to address some of the complexity.
> >
> > I noticed that the board thought we hadn't had any commits for 3+ years,
> > I'll be sure to add some commit statistics to the next board report.   I
> > counted 191 commits over the last three years, not big, but not nothing
> > either.
> >
> > I think it's fair to say that we need Apache's infrastructure, our web
> > site, Jira bug reporting system, repository and mailing lists.
> >
> > The old 2.2 series code suffered from fragility due to race conditions
> > and other bugs, understandably this made some existing developers very
> > fearful of change (who had probably suffered from this fragility in the
> > past) and the pace at which development was occurring had some
> > frightened, which impacted our ability to work on the code, for this
> > reason I don't talk too much about the code, as I fear the return of
> > arguments of old, you might say I'm a bit shy.   In fact I am hoping
> > that slowing down the pace of development as was requested has given
> > people time to accept and adapt to the 3.0 release series.
> >
> > I have been pinning my hopes on the Modular build to allow new
> > developers to focus on smaller components without having to understand
> > the whole project.
> >
> > Also I think that River is constrained by the limitations of IPv4, I
> > mean who only codes for the intranet these days?   Once we integrate
> > multicast IPv6 support (I have been using this for at least two years
> > now), we can communicate easily over the internet.
> >
> > Another concern I have is security, we should have fixed security issues
> > a long time ago, however the mood of development at the time didn't
> > foster that, I think it was 2010 when I first flagged security issues
> > with Serialization.   That's another reason why I've been hesitant to
> > create bug fix releases, I don't think the code is good enough for a
> > release without addressing some significant security issues first.
> >
> > But clearly people are using the security features of River like SSL/
> > TLS, which indicates people are using River over the internet already,
> > in spite of limitations with IPv4 NAT.  I have other fixes that address
> > TLS security issues in River and bring cyphers and constraints into
> > 2020, rather than 2004 era cyphers.
> >
> > Regards,
> >
> > Peter.
> >
> > On 5/21/2020 10:54 AM, Patricia Shanahan wrote:
> > > The board tends to be more concerned about an active community than
> > > technical matters. We need to discuss whether there is a pool of
> > > potential contributors who are likely to become active.
> > >
> > > On 5/20/2020 5:51 PM, Peter Firmstone wrote:
> > >> Hello River Folk,
> > >>
> > >> I've received feedback from the Board this morning, they are
> > >> requesting that we discuss the Attic for River.
> > >>
> > >> Personally I think the project 

Re: Board feedback - Request discuss attic for River

[Bryan Thompson]

I think engagement requires a few things.  +1 on the technical points that
you have called out Peter. But we also need to engage developers beyond
those who are traditional users of River/Jini to foster new development
with River. Which can then lead to new development of River by a broader
base of developers.

The modular build can definitely help create opportunities for developers
to get engaged. But having examples of what they can achieve and how easily
using modern security and IPV6 Is important for there to be any new
developers using River.

My question would be where are we on that roadmap. I know that we have been
making progress towards this.  When will the project reach a state where it
will engage a new group of developers and what actions will it take to help
make that happen beyond the technology development?

Bryan

On Thu, May 21, 2020 at 01:46 Peter Firmstone 
wrote:

> Thanks Patricia,
>
> I'm not sure we're going to see a significant number of developers
> contributing in the near future.
>
> Unfortunately the technical matters, which are complex and difficult
> have caused a lot of argument in the past and have been the cause of a
> high barrier to entry for new developers.   At least that's what I
> think, feel free to provide your perspective.
>
> I have been working on solutions to address some of the complexity.
>
> I noticed that the board thought we hadn't had any commits for 3+ years,
> I'll be sure to add some commit statistics to the next board report.   I
> counted 191 commits over the last three years, not big, but not nothing
> either.
>
> I think it's fair to say that we need Apache's infrastructure, our web
> site, Jira bug reporting system, repository and mailing lists.
>
> The old 2.2 series code suffered from fragility due to race conditions
> and other bugs, understandably this made some existing developers very
> fearful of change (who had probably suffered from this fragility in the
> past) and the pace at which development was occurring had some
> frightened, which impacted our ability to work on the code, for this
> reason I don't talk too much about the code, as I fear the return of
> arguments of old, you might say I'm a bit shy.   In fact I am hoping
> that slowing down the pace of development as was requested has given
> people time to accept and adapt to the 3.0 release series.
>
> I have been pinning my hopes on the Modular build to allow new
> developers to focus on smaller components without having to understand
> the whole project.
>
> Also I think that River is constrained by the limitations of IPv4, I
> mean who only codes for the intranet these days?   Once we integrate
> multicast IPv6 support (I have been using this for at least two years
> now), we can communicate easily over the internet.
>
> Another concern I have is security, we should have fixed security issues
> a long time ago, however the mood of development at the time didn't
> foster that, I think it was 2010 when I first flagged security issues
> with Serialization.   That's another reason why I've been hesitant to
> create bug fix releases, I don't think the code is good enough for a
> release without addressing some significant security issues first.
>
> But clearly people are using the security features of River like SSL/
> TLS, which indicates people are using River over the internet already,
> in spite of limitations with IPv4 NAT.  I have other fixes that address
> TLS security issues in River and bring cyphers and constraints into
> 2020, rather than 2004 era cyphers.
>
> Regards,
>
> Peter.
>
> On 5/21/2020 10:54 AM, Patricia Shanahan wrote:
> > The board tends to be more concerned about an active community than
> > technical matters. We need to discuss whether there is a pool of
> > potential contributors who are likely to become active.
> >
> > On 5/20/2020 5:51 PM, Peter Firmstone wrote:
> >> Hello River Folk,
> >>
> >> I've received feedback from the Board this morning, they are
> >> requesting that we discuss the Attic for River.
> >>
> >> Personally I think the project still has a lot of potential to pick
> >> up again once the modular build is complete, and it is a useful place
> >> to send patches and discuss changes.
> >>
> >> A very important patch was sent in June last year by Shawn Ellis for
> >> Java 11.0.3 and later for services using SSL/TLS.
> >>
> >> Another important change to the JERI protocol was discussed in
> >> September last year.
> >>
> >> http://mail-archives.apache.org/mod_mbox/river-dev/201909.mbox/browser
> >>
> >> What are your thoughts?
> >>
> >> I don't think the board is asking that we send River to the attic,
> >> just that we discuss it.
> >>
> >> Regards,
> >>
> >> Peter.
> >>
>