This is an automated email from the ASF dual-hosted git repository.
wusheng pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/skywalking-graalvm-distro.git
The following commit(s) were added to refs/heads/main by this push:
new a46fb8d Update docs: immigration status, README with native image
strategy
a46fb8d is described below
commit a46fb8d1ae5b193bbb17fe2e7fd67c5532b68319
Author: Wu Sheng <[email protected]>
AuthorDate: Thu Feb 19 12:12:12 2026 +0800
Update docs: immigration status, README with native image strategy
Update PLAN.md to reflect completed MAL immigration (Phase 2 done),
classpath scanning fully solved. Update README.md with comprehensive
overview of the build-time pre-compilation strategy, architecture
diagram, current status, and remaining phases. Mark MAL-IMMIGRATION.md
as complete. Update OAL-IMMIGRATION.md module reference.
Co-Authored-By: Claude Opus 4.6 <[email protected]>
---
MAL-IMMIGRATION.md | 2 +-
OAL-IMMIGRATION.md | 2 +-
PLAN.md | 48 ++++++++++------
README.md | 164 +++++++++++++++++++++++++++++++++++++++++++++++++++--
4 files changed, 190 insertions(+), 26 deletions(-)
diff --git a/MAL-IMMIGRATION.md b/MAL-IMMIGRATION.md
index 4121900..9ea5151 100644
--- a/MAL-IMMIGRATION.md
+++ b/MAL-IMMIGRATION.md
@@ -1,4 +1,4 @@
-# Phase 2: MAL/LAL Build-Time Pre-Compilation
+# Phase 2: MAL/LAL Build-Time Pre-Compilation — COMPLETE
## Context
diff --git a/OAL-IMMIGRATION.md b/OAL-IMMIGRATION.md
index 1917177..3f9e85d 100644
--- a/OAL-IMMIGRATION.md
+++ b/OAL-IMMIGRATION.md
@@ -10,7 +10,7 @@ OAL engine generates metrics, builder, and dispatcher classes
at runtime via Jav
## Step 1: Build-Time OAL Class Export Tool — DONE
-**Module**: `build-tools/oal-exporter`
+**Module**: `build-tools/precompiler` (originally `build-tools/oal-exporter`,
merged into unified precompiler)
**Created**: `OALClassExporter.java` — main class that:
diff --git a/PLAN.md b/PLAN.md
index a89051a..2f36c6a 100644
--- a/PLAN.md
+++ b/PLAN.md
@@ -55,14 +55,21 @@ All `.oal` scripts are known. Run OAL engine at build time,
export `.class` file
---
-## Challenge 2: MAL and LAL (Groovy + Javassist)
+## Challenge 2: MAL and LAL (Groovy + Javassist) — SOLVED
### What Happens
-- MAL uses `GroovyShell` + `DelegatingScript` for meter rule expressions
(~1188 rules). Also, `MeterSystem.create()` uses Javassist to dynamically
generate one meter subclass per metric rule.
+- MAL uses `GroovyShell` + `DelegatingScript` for meter rule expressions
(~1250 rules across 71 YAML files). Also, `MeterSystem.create()` uses Javassist
to dynamically generate one meter subclass per metric rule.
- LAL uses `GroovyShell` + `@CompileStatic` + `LALPrecompiledExtension` for
log analysis scripts (10 rules).
### Approach (this repo)
-Run full MAL/LAL initialization at build time via `build-tools/mal-compiler`.
Export Javassist-generated `.class` files + compiled Groovy script bytecode. At
runtime, load from manifests. See [MAL-IMMIGRATION.md](MAL-IMMIGRATION.md) for
details.
+Run full MAL/LAL initialization at build time via `build-tools/precompiler`
(unified tool). Export Javassist-generated `.class` files + compiled Groovy
script bytecode. At runtime, load from manifests.
+
+### What Was Built
+- **Unified precompiler** (`build-tools/precompiler`): Replaced separate
`oal-exporter` and `mal-compiler` modules. Compiles all 71 MAL YAML rule files
(meter-analyzer-config, otel-rules, log-mal-rules, envoy-metrics-rules,
telegraf-rules, zabbix-rules) producing 1209 meter classes and 1250 Groovy
scripts.
+- **Manifests**: `META-INF/mal-groovy-manifest.txt` (script→class mapping),
`META-INF/mal-groovy-expression-hashes.txt` (SHA-256 for combination pattern
resolution), `META-INF/mal-meter-classes.txt` (Javassist-generated classes),
`META-INF/annotation-scan/MeterFunction.txt` (16 function classes).
+- **Combination pattern**: Multiple YAML files from different data sources
(otel, telegraf, zabbix) may define metrics with the same name. Deterministic
suffixes (`_1`, `_2`) with expression hash tracking enable unambiguous
resolution.
+- **Same-FQCN replacements**: `DSL.java` (MAL), `DSL.java` (LAL),
`FilterExpression.java`, `MeterSystem.java` — all load pre-compiled classes
from manifests instead of runtime compilation.
+- **Comparison test suite**: 73 test classes, 1281 assertions covering all 71
YAML files. Each test validates pre-compiled classes produce identical results
to fresh Groovy compilation.
### Key finding: MAL cannot use @CompileStatic
MAL expressions rely on `propertyMissing()` for sample name resolution and
`ExpandoMetaClass` on `Number` for arithmetic operators — fundamentally dynamic
Groovy features. Pre-compilation uses standard dynamic Groovy (same
`CompilerConfiguration` as upstream). LAL already uses `@CompileStatic`.
@@ -73,18 +80,15 @@ MAL expressions rely on `propertyMissing()` for sample name
resolution and `Expa
---
-## Challenge 3: Classpath Scanning (Guava ClassPath) — PARTIALLY SOLVED
+## Challenge 3: Classpath Scanning (Guava ClassPath) — SOLVED
### What Happens
`ClassPath.from()` used in `SourceReceiverImpl.scan()`, `AnnotationScan`,
`MeterSystem`, `DefaultMetricsFunctionRegistry`, `FilterMatchers`,
`MetricsHolder`.
### What Was Solved
-`AnnotationScan` and `SourceReceiverImpl` replaced with same-FQCN classes that
read from build-time manifests. 6 annotation/interface manifests under
`META-INF/annotation-scan/`: `ScopeDeclaration`, `Stream`, `Disable`,
`MultipleDisable`, `SourceDispatcher`, `ISourceDecorator`.
-
-`DefaultMetricsFunctionRegistry`, `FilterMatchers`, `MetricsHolder` — these
only run inside the OAL engine at build time, not at runtime. Automatically
solved.
-
-### What Remains
-`MeterSystem` uses Guava `ClassPath.from()` to scan for meter function classes
at runtime. This needs a manifest or build-time scan as part of MAL immigration.
+- `AnnotationScan` and `SourceReceiverImpl` replaced with same-FQCN classes
that read from build-time manifests. 6 annotation/interface manifests under
`META-INF/annotation-scan/`: `ScopeDeclaration`, `Stream`, `Disable`,
`MultipleDisable`, `SourceDispatcher`, `ISourceDecorator`.
+- `DefaultMetricsFunctionRegistry`, `FilterMatchers`, `MetricsHolder` — these
only run inside the OAL engine at build time, not at runtime. Automatically
solved.
+- `MeterSystem` replaced with same-FQCN class that reads from
`META-INF/annotation-scan/MeterFunction.txt` manifest (16 meter function
classes). Solved as part of MAL immigration.
---
@@ -129,18 +133,25 @@ MAL expressions rely on `propertyMissing()` for sample
name resolution and `Expa
- [x] Create a JVM-mode starter with fixed module wiring (`FixedModuleManager`
+ `ModuleWiringBridge` + `GraalVMOAPServerStartUp`)
- [x] Simplified config file for selected modules (`application.yml`)
-### Phase 2: Build-Time Pre-Compilation & Verification — IN PROGRESS
+### Phase 2: Build-Time Pre-Compilation & Verification — COMPLETE
**OAL immigration — COMPLETE:**
-- [x] OAL engine → export `.class` files (`OALClassExporter`, 9 defines, ~620
metrics, ~45 dispatchers)
-- [x] Classpath scanning → export class index (6 annotation/interface
manifests in `oal-exporter`)
+- [x] OAL engine → export `.class` files (9 defines, ~620 metrics, ~620
builders, ~45 dispatchers)
+- [x] Classpath scanning → export class index (7 annotation/interface
manifests including MeterFunction)
- [x] Runtime registration from manifests (3 same-FQCN replacement classes:
`OALEngineLoaderService`, `AnnotationScan`, `SourceReceiverImpl`)
-- [x] Verification tests (`OALClassExporterTest` — 3 tests,
`PrecompiledRegistrationTest` — 12 tests)
+- [x] Verification tests (`PrecompilerTest`, `PrecompiledRegistrationTest`)
+
+**MAL immigration — COMPLETE:**
+- [x] Unified precompiler (`build-tools/precompiler`): replaces separate
`oal-exporter` and `mal-compiler`
+- [x] MAL Groovy pre-compilation: 71 YAML files → 1250 Groovy scripts + 1209
Javassist meter classes
+- [x] Combination pattern support: deterministic suffixes + expression hash
tracking for cross-source metric deduplication
+- [x] Same-FQCN replacements: `DSL.java` (MAL), `DSL.java` (LAL),
`FilterExpression.java`, `MeterSystem.java`
+- [x] Comparison test suite: 73 test classes, 1281 assertions (all 71 YAML
files, 100% coverage)
+- [x] SHA-256 staleness detection for submodule YAML drift
+- [x] `MeterSystem` classpath scan eliminated via manifest
**Remaining:**
-- [ ] MAL Groovy pre-compilation (`build-tools/mal-compiler` skeleton exists)
-- [ ] LAL Groovy pre-compilation (can be part of mal-compiler)
-- [ ] `MeterSystem` classpath scan: uses Guava scanning for meter function
classes — needs manifest or build-time scan (part of MAL immigration)
+- [ ] LAL Groovy pre-compilation (LAL `DSL.java` replacement exists but
runtime behavior not yet tested — LAL is lower priority since it has only 10
rules)
- [ ] Config generator (`build-tools/config-generator` skeleton exists) —
eliminate `Field.setAccessible` reflection in config loading
- [ ] Package everything into native-image classpath
@@ -164,5 +175,6 @@ MAL expressions rely on `propertyMissing()` for sample name
resolution and `Expa
## Upstream Changes Tracker
- [x] OAL engine: build-time class export works via existing debug API (no
upstream change needed)
-- [ ] MAL/LAL: Groovy static compilation / DSL adjustments (not started)
+- [x] MAL: No upstream changes needed — pre-compilation uses same dynamic
Groovy `CompilerConfiguration` as upstream
+- [ ] Dynamic Groovy MOP in native image: may need upstream DSL changes if
`ExpandoMetaClass` fails (Phase 3 concern)
- [ ] Other findings during implementation
diff --git a/README.md b/README.md
index a55b89b..5598c7a 100644
--- a/README.md
+++ b/README.md
@@ -1,14 +1,166 @@
# SkyWalking GraalVM Distro (Experimental)
<img src="http://skywalking.apache.org/assets/logo.svg"; alt="Sky Walking logo"
height="90px" align="right" />
-GraalVM compiles your Java applications ahead of time into standalone
binaries. These binaries are smaller, start up to 100x faster,
-provide peak performance with no warmup, and use less memory and CPU than
applications running on a Java Virtual Machine (JVM).
+GraalVM compiles your Java applications ahead of time into standalone
binaries. These binaries are smaller, start up to 100x faster, provide peak
performance with no warmup, and use less memory and CPU than applications
running on a Java Virtual Machine (JVM).
-SkyWalking GraalVM Distro is a re-distribution version of the official Apache
SkyWalking OAP server.
+SkyWalking GraalVM Distro is a re-distribution version of the official Apache
SkyWalking OAP server, targeting GraalVM native image on JDK 25.
+## Why This Is Hard
-## ⚠️ Release Policy ⚠️
-This repository is only on the experimental stage, no release will be made and
driven by the PMC by the current status, until we have a way out.
+SkyWalking OAP server relies heavily on runtime code generation and dynamic
class loading — patterns that are fundamentally incompatible with GraalVM
native image's closed-world assumption:
-# License
+| Runtime Pattern | Where Used | Scale |
+|---|---|---|
+| **Javassist bytecode generation** | OAL metrics classes, MAL meter classes |
~1,850 classes generated at startup |
+| **Groovy dynamic compilation** | MAL meter expressions, LAL log rules |
~1,260 scripts compiled at startup |
+| **Guava ClassPath scanning** | Annotation discovery, function registry | 7
scanning sites |
+| **ServiceLoader (SPI)** | Module/provider discovery | All modules |
+| **Reflection-based config** | `Field.setAccessible()` for YAML config | All
`ModuleConfig` subclasses |
+
+None of these work in a GraalVM native image out of the box.
+
+## Strategy: Build-Time Pre-Compilation
+
+The core idea is simple: **move all dynamic code generation and classpath
scanning from runtime to build time**. At build time, run the full OAL/MAL/LAL
initialization pipeline, capture all generated bytecode, and package it into
the native image classpath. At runtime, load pre-compiled classes from
manifests — no Javassist, no GroovyShell, no ClassPath scanning.
+
+```
+┌─────────────────────────────────────────────────────────┐
+│ BUILD TIME │
+│ │
+│ skywalking/ build-tools/precompiler │
+│ (submodule) ┌──────────────────────────────┐ │
+│ │ │ 1. Run OAL engine (Javassist)│ │
+│ │ │ → 620 metrics classes │ │
+│ │ │ → 620 builder classes │ │
+│ │ │ → 45 dispatcher classes │ │
+│ │ │ │ │
+│ │ │ 2. Run MAL engine (Groovy) │ │
+│ │ │ → 1250 Groovy scripts │ │
+│ │ │ → 1209 meter classes │ │
+│ │ │ │ │
+│ │ │ 3. Classpath scan │ │
+│ │ │ → 7 annotation manifests │ │
+│ │ │ │ │
+│ │ │ 4. Write manifests │ │
+│ │ └──────────┬───────────────────┘ │
+│ │ │
+│ ▼ │
+│ precompiler-generated.jar │
+│ (all .class files + manifests) │
+└─────────────────────────────────────────────────────────┘
+ │
+ ▼
+┌─────────────────────────────────────────────────────────┐
+│ RUNTIME │
+│ │
+│ oap-graalvm-server │
+│ ┌─────────────────────────────────────────────┐ │
+│ │ Same-FQCN replacement classes: │ │
+│ │ │ │
+│ │ OALEngineLoaderService │ │
+│ │ → reads oal-metrics-classes.txt manifest │ │
+│ │ → Class.forName() + register │ │
+│ │ │ │
+│ │ DSL.java (MAL) │ │
+│ │ → reads mal-groovy-manifest.txt │ │
+│ │ → Class.forName() pre-compiled scripts │ │
+│ │ │ │
+│ │ MeterSystem.java │ │
+│ │ → reads MeterFunction.txt manifest │ │
+│ │ → reads mal-meter-classes.txt manifest │ │
+│ │ → no Javassist, no ClassPath.from() │ │
+│ │ │ │
+│ │ AnnotationScan.java │ │
+│ │ → reads annotation-scan/*.txt manifests │ │
+│ │ → no Guava ClassPath.from() │ │
+│ └─────────────────────────────────────────────┘ │
+└─────────────────────────────────────────────────────────┘
+```
+
+The **same-FQCN replacement** technique makes this transparent: classes in
`oap-graalvm-server` share the exact fully-qualified name with their upstream
counterparts. Maven classpath ordering ensures the replacement is loaded. No
forking, no patching, no SPI trickery.
+
+## Current Status
+
+### Phase 1: Build System Setup — COMPLETE
+- Maven + Makefile build orchestration
+- SkyWalking as git submodule (`skywalking/`)
+- Fixed module manager (no SPI discovery)
+- Simplified `application.yml` for selected providers
+
+### Phase 2: Build-Time Pre-Compilation — COMPLETE
+- **OAL immigration**: 9 OAL defines → ~1,285 Javassist classes exported, 6
annotation manifests
+- **MAL immigration**: 71 MAL YAML files → 1,250 Groovy scripts + 1,209
Javassist meter classes exported
+- **Classpath scanning eliminated**: All 7 Guava `ClassPath.from()` sites
replaced with build-time manifests
+- **Verification**: 1,281 comparison tests validate pre-compiled classes match
fresh Groovy compilation
+
+### Phase 3: Native Image Build — NOT STARTED
+- `native-image-maven-plugin` configuration
+- GraalVM reflection/resource/JNI metadata (`reflect-config.json`,
`resource-config.json`)
+- gRPC/Netty/Protobuf native image configuration
+- Dynamic Groovy MOP compatibility (biggest unknown risk)
+- Get OAP server booting as native image with BanyanDB storage
+
+### Phase 4: Harden & Test — NOT STARTED
+- Verify all receiver plugins, query APIs, cluster mode, alarm
+- Performance benchmarking vs JVM
+- CI: automated native-image build + smoke tests
+
+## Module Selection
+
+This distro targets a **full-feature OAP server** with fixed module/provider
selection:
+
+- **Storage**: BanyanDB
+- **Cluster**: Kubernetes
+- **Configuration**: Kubernetes
+- **Receivers**: All (trace, meter, log, profile, browser, OTel, mesh, envoy,
Zipkin, Zabbix, Telegraf, etc.)
+- **Query**: GraphQL, PromQL, LogQL, Zipkin
+- **Alarm, Telemetry, Exporter**: Enabled
+
+## Build
+
+Requires GraalVM JDK 25.
+
+```bash
+# Initialize submodule
+git submodule update --init --recursive
+
+# Full build (precompiler + tests + server)
+JAVA_HOME=/path/to/graalvm-jdk-25 make build-distro
+```
+
+## Project Structure
+
+```
+skywalking-graalvm-distro/
+├── skywalking/ # Git submodule — DO NOT MODIFY
+├── build-tools/
+│ ├── precompiler/ # Build-time OAL + MAL + LAL pre-compilation
+│ └── config-generator/ # (planned) Build-time config code
generation
+├── oap-graalvm-server/
+│ └── src/
+│ ├── main/java/ # Same-FQCN replacement classes
+│ │ ├── .../oal/rt/ # OALEngineLoaderService
+│ │ ├── .../annotation/ # AnnotationScan
+│ │ ├── .../source/ # SourceReceiverImpl
+│ │ ├── .../meter/ # MeterSystem, DSL, FilterExpression
+│ │ └── .../log/ # LAL DSL
+│ └── test/java/ # 1,281 comparison tests
+├── PLAN.md # Detailed build plan with phase tracking
+├── OAL-IMMIGRATION.md # OAL pre-compilation design doc
+└── MAL-IMMIGRATION.md # MAL pre-compilation design doc
+```
+
+## Known Risks
+
+| Risk | Severity | Mitigation |
+|---|---|---|
+| Dynamic Groovy MOP in native image | High | `ExpandoMetaClass` +
`propertyMissing()` may not work. Fallback: upstream DSL changes to eliminate
dynamic dispatch. |
+| gRPC / Netty native image | Medium | GraalVM reachability metadata repo,
Netty substitutions |
+| Reflection sites beyond config | Medium | Tracing agent +
`reflect-config.json` |
+| Kubernetes client in native image | Low | Has documented GraalVM support |
+
+## Release Policy
+This repository is experimental. No releases are planned until a working
native image is achieved.
+
+## License
Apache 2.0
![http://skywalking.apache.org/assets/logo.svg"](http://skywalking.apache.org/assets/logo.svg"){kind=link}