Hi Matt,
more inline
On 9/2/26 17:14, Matt M wrote:
Hello,
I read through the proposal and it sounds interesting. My initial
impression is that it might not have a lot relevance to end users (maybe
it does for specific library writers or as a kind of "constraint" around
certain code? (I hesitate to say anything else or in other ways here in
so far as to draw incorrect or unintended correlations)). In its current
iteration, it seems focused on being a compiler level enhancement/
requirement for future refactoring and redesigns than a necessarily
highly applicable, broadly relevant user facing feature. None of which
is bad.
yeah, I guess I was emphasizing the compiler part too much.
Some problems this tries to solve:
**Extension methods**
* In the static compiler, extension methods from a library dependency
can influence your code, even if you use nothing from that library.
* Extension methods are not available for internal use only. A library
cannot provide extension methods for its own implementation without also
making them available to users.
* Extension methods may have to be written in Java to avoid interference
from the MetaClass system.
* Users cannot define their own extension methods in the same way as
library-provided extension methods.
**Categories**
Categories are the user-level variant of extension methods, but their
scope is problematic because it is thread-bound.
* Most categories are actually intended to apply to a call site, which
is why this GEP discusses lexical categories.
* Dynamic categories complicate the implementation and affect every call
site that Groovy resolves dynamically. Being able to turn them off
permits a simpler and more reliable dispatch implementation, which also
benefits tooling and other implementations of the call site.
**Groovy core and Groovy modules**
* Some parts of Groovy core, such as extension methods and the MetaClass
implementation, cannot be written in Groovy itself without depending on
the dynamic mechanisms they implement. This is not a proposal to rewrite
Groovy in Groovy tomorrow, but currently there are places where this is
not possible at all.
* The old MetaClass system has accumulated several generations of
implementation techniques: reflection, generated call sites, primitive
optimizations, invokedynamic, and others. It needs a redesign that is
largely blocked by the requirement to preserve the legacy API.
**Optimized dynamic compilation**
This is already discussed extensively in the GEP. The basic idea is
dynamic Groovy with compile-time optimizations controlled by which types
and dynamic features are available.
For the user in short, more control, more tooling support under some
specific conditions, a new construct for categories.
I believe I got the gist of the idea that the GEP is getting at. It
would probably be really useful to call out and accentuate the exact
purpose and mission statement of the problem(s) that it seeks to solve
(e.g. "Start to work towards enabling strong encapsulation and isolation
of dynamic capabilities in a controlled and deterministic way in so far
as to prevent unintended leaks of the MOP and MOP-like functionality
outside of an articulated scope.”).
the mission statement I would probably phrase along this:
The goal of realms is to make Groovy's dynamic semantics explicit and
controllable. A realm defines which dynamic capabilities are visible to
a piece of code, allowing extension methods, categories, the MetaClass
system, and other dynamic behavior to be selected, isolated, and
optimized independently of the surrounding application.
"which dynamic capabilities are visible to a piece of code" does not
mean encapsulation in the Java sense. This is not an OOP concept.
Optimization is a byproduct of having more control - control a human and
the compiler can understand.
Can a user continue writing his programs as before, most likely. The
intention is not to break existing code with this. I have to see if that
is actually realizable, but I think it should. And if there are changes,
then in the meta class api maybe.
At a linguistics level for the idea, I think the term |Realm| is not
quite a good fit for describing the concept. It's quite whimsical and
stylized without quickly and clearly explaining/alluding to what pattern
of functionality it represents or does, even colloquially. I would
suggest changing it to some version or combination of the various
explanatory terms/word-phrases already used in the GEP to explain/
describe the idea itself such as |Environment|, |Scope|, |
EnvironmentScope|, |ImmutableContext|, etc. just to name a few. For
example, an initial idea of a clearer design that came to mind was that
a |@Realm| (i.e. in usage form) is better described as a |@Scope| and a
|@RealmDefinition| is described as an |@Environment|. Therefore, one way
a user could think of it is as "You define an @Environment and then use
that environment in/as a @Scope for code to execute in/under". Makes the
mental model clearer and more explicit.
Names can still be changed, the concept is what needs acceptance first.
To give you an example of my impressions... When I hear scope, I think
block scope, variable scope and so on. Environment I think of a system,
environment scope I have no idea. If I had to do more with Kubernetes I
would probably associate realm with that world. Just see the names as
"work in progress".
On a minor hiccup I had in my readings is the word "properties” being
used. I initially got a little confused at how it was saying that these
concepts would interact with and work with Properties (i.e. Groovy's |
getters/setters|) since it said: "A realm inherits the definition of one
other realm and may replace individual properties.”. I incorrectly
conflated the term with the Groovy literal idea of |Properties| which is
not what was intended. It might be better and clearer to say that what
the GEP is referring to are environmental context attributes or
something else of that kind.
For now I will then just use attributes, which really only works better
against Groovy and Java developers, because attributes is not used so
much there to describe fields.
At a semantic level, overall, the GEP is going a bit deeper than the
idea that might be canonically known as namespaces. It's closer to
implementing specific resolution handling and isolation rules/semantics
for strong encapsulation of/for the MOP. I tried to draw a corollary to
other languages that implement the same concept or at least notionally
seem to go for the same idea. The best one that that I could arrive at
was idea of "function environments" in Lua. GEP-31 seems similar in
vision albeit highly constrained on the MOP and extension methods
exclusively in the current iteration.
Dart, Kotlin and Scala provide statically scoped extension mechanisms,
Ruby refinements are closer to lexical categories. JavaScript realms
instead isolate global execution environments. These mechanisms address
parts of the problem that realms address, but none combines them with
explicit control over Groovy's MetaClass-based dispatch.
Frankly the whole problem is home made and rooted in the complex MOP
[...]
* How does backwards compatibility work in so far as the compiler’s
public API/ABI is concerned? At first glance, it seems completely
transparent and seamless. However, I wouldn't want it to be what
project jigsaw was to Java (i.e. probably the least understood, most
hated/disliked feature ever implemented in Java; that was largely
seen as solely for the benefit of the compiler/language devs at the
expense of everyone else). [I know that I am probably editorializing
a fair bit here but based on my experience and interactions with all
kinds of other developers in the Java space and even what the JVM
architects said themselves, it's not really a disingenuous or even
an unfair perspective].
well... good that you did not see some of the mails I exchanged with
those guys back then. That is all I will say about me not liking jigsaw,
or I will rant for a while.
I don´t think the ABI will change for this. I do not see where and what
for. There will be a new bootstrap method, to take the realm as
parameter, but the old one will continue to work and assume GROOVY_DYN
realm, which is 100% compatible to today.
* What are rules for inheritance as it relates to a Realm and all of
its layers' priority and ordering? How does a Realm inherit from
another Realm and how do we envision that working (not from a syntax
perspective but rather like how do we envision that actually working
under the hood)? What is the mental model for its inheritance that
is made clear for end users (folks in general struggle to understand
OOP inheritance as it is already)? There's a lot of talk about
layers and there seem to be quite many and they have various planned
intricacies for how they each interact with each other and those
above/below each other, in regards to precedence and priority.
Realms do not strictly inherit in the OO sense. A realm definition
declares capabilities. Extending another realm means taking those
capabilities and then overwriting selected ones with your own
definition. The resulting realm is an ordered composition of layers.
The usage side determines where the definition is applied. The usages in
the proposal are use(MyRealm) { ... } and @Realm(MyRealm). MyRealm is
more or less a token given to the runtime.
At runtime, that token is associated with an ordered list of layers per
class. The exact shape is not defined yet and will depend on the
implementation, but most likely there will be one layer per extension
method class, one for the small MOP, one for the extended MOP, and one
for mutable MetaClasses.
To dispatch a method, the runtime goes through the layers in order. Each
layer determines which targets are applicable; the applicable targets
may be empty. The most specific applicable method is selected as the
dispatch target. If no layer provides an applicable target, the dispatch
results in MissingMethodException.
Think of each layer as a specialist. We already have most of this
machinery in current Groovy, both as code and partially as a model - we
call it a meta class. The proposal separates the code paths and models
the dispatch environment more explicitly. Once it is represented as a
model, we can reason about it; with the current implementation, much of
that reasoning is implicit in the code. That combination of model and
code is currently spread between the meta classes and ClassInfo, as well
as several more classes.
* How does access for Realms work? Are they public by default and/or
can they be designated as private, package-private, etc.? Would
protected be allowed?
I did not explicitly define that yet, yes. But since a definition will
be represented as class, it goes with the visibility of the class. And I
guess I should move the declaration style I used before to an explicit
class, where you can apply a modifier as well. Then simply normal class
rules apply.
* What is the compelling use case/reason for a user to want to even
use/know about/understand this feature at all? I would want to avoid
it having the same kind of confusion and/or kind of general disdain/
disinterest at understanding as there is/was around the JVM's
concepts of classpath and modulepath interplay.
More control over the dynamic semantics of a piece of code.
A library can choose which extension methods, categories and MetaClass
behavior its code uses, without implicitly depending on whatever dynamic
environment happens to surround it. An application can do the same when
it wants a particular dynamic behavior or wants to isolate parts of the
application from each other.
Jigsaw mainly restricts what code can access. Realms do not primarily
restrict access. They let the programmer choose which dynamic
capabilities are present. There is isolation, yes, but there is no
*privileged* "inside" that needs to be protected from an "outside".
Think:
My library should behave the same way whether or not the application
happens to have some unrelated extension method or MetaClass
customization installed.
* Are Realms repeatable/stackable (e.g. |@Realm(X) @Realm(Y) def
my_method() { ... }|)? I could see it going either way on allowing
or disallowing this. If allowed, there's then the challenge of
checking for overlaps/conflicting extensions, precedence handling,
and cycle detection that would have to be done at each usage site.
If not allowed, then there's the challenge of how to allow users to
compose Realms and/or extending Realms either that they didn't write
and/or don't have access to.
|@Realm(X) @Realm(Y) def my_method() { ... }| is one usage site only.
But no, I would not allow that.
At a syntax level, I know this will come off really blunt and harsh, I
apologize in advance. I /really/ don't like the shape or design of the
syntax for implementing a Realm's definition at all, as it is stated in
the GEP. It looks very out-of-place and like non-idiomatic Groovy. It
reads /very/ strangely and is awkward to understand at a glance since it
seems like a hybrid of a not-closure closure and half a class
declaration with a weird set assignment/definition thrown in for good
measure. Syntactically, it's just very confusing to read/understand and
I would imagine it will be quite kludge'y to implement at a grammar/
syntax level as is. Again, I don't mean to draw the same parallel to
jigsaw but it comes off as feeling like module-info.java all over again.
Again, apologies for being really blunt and just hating on it.
you will be then surprised to hear that this requires not changes to the
syntax at all. Instead it is a DSL, a local transform changes the
semantics. I would probably now move the block into the annotation and
annotate a class that defines modifier and name. But I doubt that
changes your verdict. I am open for a better version of course. Only
that I do not want this definition to be a runtime construct. The result
should be readable without initializing the class it is based on.
I think it would be simpler, clearer, and more consistent to not get
clever at all with the syntax here and just be super boring about it by
having it be implemented it as a typical plain-old Groovy class like
anything else. Using common semantics like a constructor and a super
class to clearly define the API contract of it all. The compiler could
just scan for any and all classes that extend the super class (or
implement a particular interface as that could be another route to go)
and process those first/separately as environments/realm definitions.
Sometimes simple and boring is best.
The class-scanning part is exactly what I do not want. A realm should be
an explicit semantic dependency, not something the compiler discovers by
scanning the classpath. Otherwise adding an unrelated realm to the
classpath can change compilation semantics, and incremental compilation
becomes harder to reason about.
I also do not want a realm definition to require class initialization or
object construction. The definition is metadata. I want the compiler and
runtime to be able to inspect it directly from the class file, using
reflection or ASM, and obtain the realm token without executing user code.
So a plain class can still be the underlying representation, but the
realm definition itself needs to be declarative and explicitly
referenced rather than discovered by scanning for subclasses.
The class is the discovery mechanism.
At a conceptual level, I like the idea of what the GEP is shooting for
in the broad strokes. There's some details to flesh out and refine. I
think it would be really helpful as well to see a few illustrative
examples and samples of the idea in motion so that someone could get a
sense of what it enables, how it would work, and the value it adds for
them and just getting a bird's eye view of it all.
**Library isolation**
```groovy
MY_LIB = {
extends GROOVY_DYN
extensions MyExtensions
metaclass isolated
}
```
and use `@Realm(MY_LIB)` for its implementation. The library then does
not accidentally acquire extension methods or MetaClass changes from the
application using it.
**Explicit extension dependencies**
Instead of discovering extension methods from everything on the
classpath, a realm explicitly declares them:
```groovy
MY_LIB = {
extensions [MyExtensions, SOME_OTHER_REALM.extensions]
}
```
**Lexical categories**
A category can be represented as a derived realm for a lexical region:
```groovy
use(MyCategory) {
// category semantics apply here
}
```
rather than changing thread-bound dynamic state. This is closer to how
categories are usually intended to be used, and it also gives the
compiler a known semantic environment for those call sites.
**Groovy itself**
The same mechanism allows different parts of Groovy to choose different
dispatch environments. For example, a new MetaClass implementation could
be used in a realm without requiring all of Groovy to use it
immediately. This is also the direction in which I see the realm concept
helping with a more optimized form of dynamic compilation.
Complete example would probably be too much for this mail.
On a meta note, what Groovy version is this tentatively planned to
target? I didn't see any mention of a target version but I would
speculate v8 or v9 and beyond.
well... maybe Groovy 7 is too short. But nothing says that it has to be
complete in version 7.
On a meta meta note, the original email came through just fine on my end
and I even saw it on the mailing list archive too. Not sure that helps
at all or anything but just throwing that out there. Emails/servers
being wonky is one of the many fun caveats of old school mailing list
style development.
I opened https://issues.apache.org/jira/browse/INFRA-28349 for this, and
we are both victims of this.
I hope this is helpful. Thank you for writing this GEP up regardless.
The worst would have been nobody caring to even answer - so thank you
for caring! I think no good idea can be good enough if it cannot
withstand some skepticism.
bye Jochen
