wu-sheng commented on code in PR #13699:
URL: https://github.com/apache/skywalking/pull/13699#discussion_r2794101068
##########
oap-server/oal-rt/V2_IMPLEMENTATION_SUMMARY.md:
##########
@@ -0,0 +1,386 @@
+# OAL Engine V2 - Implementation Summary
+
+## Overview
+
+OAL Engine V2 is now the **only OAL implementation** after V1 has been
completely removed. V2 provides a clean architecture with immutable models,
type safety, and comprehensive testing.
+
+## Architecture
+
+```
+OAL Script → OALScriptParserV2 → MetricDefinition (immutable) →
+MetricDefinitionEnricher → CodeGenModel → OALClassGeneratorV2 →
+FreeMarker Templates → Javassist → Generated Classes
+```
+
+### Key Design Principles
+
+1. **Immutable Models**: All data models are immutable and thread-safe
+2. **Type Safety**: Strongly-typed filter values and function arguments
+3. **Clean Separation**: Parse → Enrich → Generate pipeline
+4. **Independent Package Structure**: All V2 code organized under
`org.apache.skywalking.oal.v2`
+5. **Comprehensive Testing**: 70+ unit tests covering all components
+
+## What Was Implemented
+
+### 1. V2 Immutable Model Classes ✅
+**Location**: `org.apache.skywalking.oal.v2.model`
+
+All model classes are immutable, type-safe, and use builder pattern:
+- **SourceLocation**: Track source file location for error reporting
+- **SourceReference**: Immutable source reference (e.g., `Service.latency`)
+- **FilterOperator**: Type-safe enum for filter operators (==, >, <, like,
etc.)
+- **FilterValue**: Strongly-typed filter values (NUMBER, STRING, BOOLEAN,
NULL, ARRAY)
+- **FunctionArgument**: Type-safe function arguments (LITERAL, ATTRIBUTE,
EXPRESSION)
+- **FilterExpression**: Immutable filter expression
+- **FunctionCall**: Function call with typed arguments
+- **MetricDefinition**: Complete metric definition
+
+### 2. V2 Parser ✅
+**Location**: `org.apache.skywalking.oal.v2.parser`
+
+- **OALListenerV2**: ANTLR listener converting parse tree to V2 models
+ - Handles all OAL grammar rules
+ - Builds immutable objects
+ - Preserves source locations
+ - Comprehensive javadoc with parsing flow examples
+
+- **OALScriptParserV2**: Facade for parsing OAL scripts
+ - Simple API: `OALScriptParserV2.parse(script)`
+ - Returns list of `MetricDefinition` objects
+ - Helper methods: `hasMetrics()`, `getMetricsCount()`
+
+### 3. V2 Registry ✅
+**Location**: `org.apache.skywalking.oal.v2.registry`
+
+- **MetricsFunctionRegistry**: Interface for function registry
+- **MetricsFunctionDescriptor**: Function metadata
+- **DefaultMetricsFunctionRegistry**: Classpath scanning implementation
+ - Scans for `@MetricsFunction` annotations
+ - Finds `@Entrance` methods
+ - Lazy initialization with thread-safety
+
+### 4. V2 Code Generation Pipeline ✅
+**Location**: `org.apache.skywalking.oal.v2.generator`
+
+- **CodeGenModel**: V2 data model for templates
+ - Contains all information needed by Freemarker templates
+ - Separate classes for source fields, persistent fields, entrance methods
+ - Serialization field models
+
+- **MetricDefinitionEnricher**: V2 enricher (replaces V1's DeepAnalysis)
+ - Looks up metrics function classes
+ - Extracts source columns from metadata
+ - Finds entrance methods via reflection
+ - Builds entrance method arguments
+ - Collects persistent fields from @Column annotations
+ - Generates serialization fields
+
+- **OALClassGeneratorV2**: V2 class generator
+ - Uses **V2 templates** from `code-templates-v2/`
+ - Generates metrics classes with Javassist
+ - Generates builder classes
+ - Generates dispatcher classes
+ - Completely independent from V1 generator
+
+### 5. V2 Engine ✅
+**Location**: `org.apache.skywalking.oal.v2`
+
+- **OALEngineV2**: Main V2 engine (extends OALKernel)
+ - Uses V2 parser for parsing
+ - Uses V2 enricher for metadata extraction
+ - Uses V2 generator with V2 templates
+ - **No V1 dependencies** in the pipeline
Review Comment:
Fixed. Updated to 'implements OALEngine'.
##########
oap-server/oal-rt/V2_IMPLEMENTATION_SUMMARY.md:
##########
@@ -0,0 +1,386 @@
+# OAL Engine V2 - Implementation Summary
+
+## Overview
+
+OAL Engine V2 is now the **only OAL implementation** after V1 has been
completely removed. V2 provides a clean architecture with immutable models,
type safety, and comprehensive testing.
+
+## Architecture
+
+```
+OAL Script → OALScriptParserV2 → MetricDefinition (immutable) →
+MetricDefinitionEnricher → CodeGenModel → OALClassGeneratorV2 →
+FreeMarker Templates → Javassist → Generated Classes
+```
+
+### Key Design Principles
+
+1. **Immutable Models**: All data models are immutable and thread-safe
+2. **Type Safety**: Strongly-typed filter values and function arguments
+3. **Clean Separation**: Parse → Enrich → Generate pipeline
+4. **Independent Package Structure**: All V2 code organized under
`org.apache.skywalking.oal.v2`
+5. **Comprehensive Testing**: 70+ unit tests covering all components
+
+## What Was Implemented
+
+### 1. V2 Immutable Model Classes ✅
+**Location**: `org.apache.skywalking.oal.v2.model`
+
+All model classes are immutable, type-safe, and use builder pattern:
+- **SourceLocation**: Track source file location for error reporting
+- **SourceReference**: Immutable source reference (e.g., `Service.latency`)
+- **FilterOperator**: Type-safe enum for filter operators (==, >, <, like,
etc.)
+- **FilterValue**: Strongly-typed filter values (NUMBER, STRING, BOOLEAN,
NULL, ARRAY)
+- **FunctionArgument**: Type-safe function arguments (LITERAL, ATTRIBUTE,
EXPRESSION)
+- **FilterExpression**: Immutable filter expression
+- **FunctionCall**: Function call with typed arguments
+- **MetricDefinition**: Complete metric definition
+
+### 2. V2 Parser ✅
+**Location**: `org.apache.skywalking.oal.v2.parser`
+
+- **OALListenerV2**: ANTLR listener converting parse tree to V2 models
+ - Handles all OAL grammar rules
+ - Builds immutable objects
+ - Preserves source locations
+ - Comprehensive javadoc with parsing flow examples
+
+- **OALScriptParserV2**: Facade for parsing OAL scripts
+ - Simple API: `OALScriptParserV2.parse(script)`
+ - Returns list of `MetricDefinition` objects
+ - Helper methods: `hasMetrics()`, `getMetricsCount()`
+
+### 3. V2 Registry ✅
+**Location**: `org.apache.skywalking.oal.v2.registry`
+
+- **MetricsFunctionRegistry**: Interface for function registry
+- **MetricsFunctionDescriptor**: Function metadata
+- **DefaultMetricsFunctionRegistry**: Classpath scanning implementation
+ - Scans for `@MetricsFunction` annotations
+ - Finds `@Entrance` methods
+ - Lazy initialization with thread-safety
+
+### 4. V2 Code Generation Pipeline ✅
+**Location**: `org.apache.skywalking.oal.v2.generator`
+
+- **CodeGenModel**: V2 data model for templates
+ - Contains all information needed by Freemarker templates
+ - Separate classes for source fields, persistent fields, entrance methods
+ - Serialization field models
+
+- **MetricDefinitionEnricher**: V2 enricher (replaces V1's DeepAnalysis)
+ - Looks up metrics function classes
+ - Extracts source columns from metadata
+ - Finds entrance methods via reflection
+ - Builds entrance method arguments
+ - Collects persistent fields from @Column annotations
+ - Generates serialization fields
+
+- **OALClassGeneratorV2**: V2 class generator
+ - Uses **V2 templates** from `code-templates-v2/`
+ - Generates metrics classes with Javassist
+ - Generates builder classes
+ - Generates dispatcher classes
+ - Completely independent from V1 generator
+
+### 5. V2 Engine ✅
+**Location**: `org.apache.skywalking.oal.v2`
+
+- **OALEngineV2**: Main V2 engine (extends OALKernel)
+ - Uses V2 parser for parsing
+ - Uses V2 enricher for metadata extraction
+ - Uses V2 generator with V2 templates
+ - **No V1 dependencies** in the pipeline
+
+### 6. V2 Freemarker Templates ✅
+**Location**: `src/main/resources/code-templates-v2/`
+
+**Metrics templates** (9 files):
+- `id.ftl` - Storage ID generation
+- `hashCode.ftl` - Hash code for metrics identity
+- `remoteHashCode.ftl` - Remote hash code
+- `equals.ftl` - Equality comparison
+- `serialize.ftl` - Remote data serialization
+- `deserialize.ftl` - Remote data deserialization
+- `getMeta.ftl` - Metadata (table name)
+- `toHour.ftl` - Hour-level aggregation
+- `toDay.ftl` - Day-level aggregation
+
+**Dispatcher templates** (2 files):
+- `doMetrics.ftl` - Individual metric dispatcher method
+- `dispatch.ftl` - Main dispatch method
+
+**Builder templates** (2 files):
+- `entity2Storage.ftl` - Convert entity to storage format
+- `storage2Entity.ftl` - Convert storage to entity format
+
+### 7. Comprehensive Tests ✅
+**Location**: `org.apache.skywalking.oal.v2.*Test`
+
+**Model Tests** (28 tests):
+- FilterValueTest (10 tests) - All value types with type safety
+- FilterExpressionTest (10 tests) - All operators and value combinations
+- FunctionCallTest (8 tests) - Arguments, equality, type safety
+
+**Parser Tests** (15 tests):
+- OALScriptParserV2Test - Complete integration tests
+ - Simple metrics
+ - Wildcards
+ - Filters (number, boolean, string)
+ - Multiple chained filters
+ - Function arguments
+ - Decorators
+ - Multiple metrics
+ - Disable statements
+ - All comparison operators
+
+**Total**: 62 tests (43 V2 + 19 V1), all passing ✅
+
+## Package Structure
+
+After V1 removal, all code is organized under `org.apache.skywalking.oal.v2`:
+
+| Package | Purpose |
+|---------|---------|
+| `model` | Immutable data models (MetricDefinition, FilterExpression, etc.) |
+| `parser` | OAL script parsing (OALListenerV2, OALScriptParserV2) |
+| `generator` | Code generation (MetricDefinitionEnricher,
OALClassGeneratorV2, CodeGenModel) |
+| `metadata` | Metadata utilities (SourceColumnsFactory, FilterMatchers,
MetricsHolder) |
+| `util` | Code generation utilities (ClassMethodUtil, TypeCastUtil) |
+| `registry` | Function registry (MetricsFunctionRegistry) |
+
+## Key Benefits
+
+### 1. Type Safety
+
+### 2. Type Safety
Review Comment:
Fixed. Removed duplicate section.
##########
oap-server/oal-rt/src/main/resources/code-templates-v2/metrics/deserialize.ftl:
##########
@@ -0,0 +1,29 @@
+public void
deserialize(org.apache.skywalking.oap.server.core.remote.grpc.proto.RemoteData
remoteData) {
+<#if serializeFields.stringFields?? && serializeFields.stringFields?size gt 0>
+ <#list serializeFields.stringFields as field>
+ if (remoteData.getDataStrings(${field_index}) != "") {
+ ${field.setter}(remoteData.getDataStrings(${field_index}));
+ }
Review Comment:
Fixed. Changed to use .isEmpty() for proper content comparison.
##########
oap-server/oal-rt/src/test/java/org/apache/skywalking/oal/v2/parser/RealOALScriptsTest.java:
##########
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oal.v2.parser;
+
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.skywalking.oal.v2.model.MetricDefinition;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+/**
+ * Test V2 parser with real OAL scripts from server-starter/resources/oal.
+ *
+ * This validates that V2 can parse all production OAL scripts.
+ */
+@Slf4j
+public class RealOALScriptsTest {
+
+ /**
+ * Find the OAL scripts directory in the project.
+ *
+ * Tries multiple paths to locate the directory:
+ * 1. From current working directory (Maven default)
+ * 2. From user.dir system property
+ * 3. Relative from this module
+ */
+ private File findOALScriptsDir() {
+ String[] possiblePaths = {
+ "oap-server/server-starter/src/main/resources/oal",
+ "../server-starter/src/main/resources/oal",
+ "../../server-starter/src/main/resources/oal"
+ };
+
+ for (String path : possiblePaths) {
+ File dir = new File(path);
+ if (dir.exists() && dir.isDirectory()) {
+ log.debug("Found OAL scripts directory at: {}",
dir.getAbsolutePath());
+ return dir;
+ }
+ }
+
+ throw new IllegalStateException("Could not find OAL scripts directory.
Tried: " +
+ String.join(", ", possiblePaths));
+ }
+
+ /**
+ * Test parsing core.oal with V2.
+ *
+ * core.oal contains the main service and endpoint metrics.
+ */
+ @Test
+ public void testParseCoreOAL() throws IOException {
+ File oalDir = findOALScriptsDir();
+ File oalFile = new File(oalDir, "core.oal");
+ assertTrue(oalFile.exists(), "core.oal not found at: " +
oalFile.getAbsolutePath());
+
+ OALScriptParserV2 parser = OALScriptParserV2.parse(new
FileReader(oalFile), "core.oal");
+
+ assertNotNull(parser);
+ assertTrue(parser.hasMetrics(), "core.oal should have metrics");
+
+ log.info("✅ Parsed core.oal: {} metrics", parser.getMetricsCount());
+
+ // Verify some expected metrics exist
+ boolean foundServiceRespTime = false;
+ boolean foundServiceSla = false;
+ boolean foundServiceCpm = false;
+
+ for (MetricDefinition metric : parser.getMetrics()) {
+ String name = metric.getName();
+ if (name.equals("service_resp_time")) {
+ foundServiceRespTime = true;
+ log.info(" - Found: service_resp_time (source: {}, function:
{})",
+ metric.getSource().getName(),
+ metric.getAggregationFunction().getName());
+ } else if (name.equals("service_sla")) {
+ foundServiceSla = true;
+ } else if (name.equals("service_cpm")) {
+ foundServiceCpm = true;
+ }
+ }
+
+ assertTrue(foundServiceRespTime, "Should find service_resp_time");
+ assertTrue(foundServiceSla, "Should find service_sla");
+ assertTrue(foundServiceCpm, "Should find service_cpm");
+ }
+
+ /**
+ * Test parsing all OAL files from server-starter/resources/oal.
+ */
+ @Test
+ public void testParseAllOALFiles() throws IOException {
+ File oalDir = findOALScriptsDir();
+ String[] oalFiles = {
+ "core.oal",
+ "java-agent.oal",
+ "dotnet-agent.oal",
+ "browser.oal",
+ "mesh.oal",
+ "ebpf.oal",
+ "tcp.oal",
+ "cilium.oal",
+ "disable.oal"
+ };
+
+ int totalMetrics = 0;
+ int successCount = 0;
+
+ for (String fileName : oalFiles) {
+ File oalFile = new File(oalDir, fileName);
+ assertTrue(oalFile.exists(), fileName + " not found at: " +
oalFile.getAbsolutePath());
+
+ try {
+ OALScriptParserV2 parser = OALScriptParserV2.parse(new
FileReader(oalFile), fileName);
Review Comment:
Fixed in same commit - all FileReader instances use try-with-resources.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
