Re: [PR] LAL: add def local variable support with toJson/toJsonArray and type cast [skywalking]
wu-sheng merged PR #13737: URL: https://github.com/apache/skywalking/pull/13737 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] LAL: add def local variable support with toJson/toJsonArray and type cast [skywalking]
wu-sheng commented on PR #13737:
URL: https://github.com/apache/skywalking/pull/13737#issuecomment-4035523755
## Generated `.class` Output Locations
All `.class` files are generated automatically during `mvn test` — no
separate generation step needed.
### Table 1: DSLClassGeneratorTest (server-starter) — all production scripts
| DSL | Output path | Classes |
|---|---|---|
| OAL | `oap-server/server-starter/target/generated-dsl-classes/oal/` |
1,285 |
| MAL | `oap-server/server-starter/target/generated-dsl-classes/mal/` |
1,269 |
| LAL | `oap-server/server-starter/target/generated-dsl-classes/lal/` | 10 |
| Hierarchy |
`oap-server/server-starter/target/generated-dsl-classes/hierarchy/` | 4 |
| **Total** | | **2,568** |
```bash
./mvnw test -pl oap-server/server-starter -Dtest=DSLClassGeneratorTest
```
### Table 2: Unit tests and v1-v2 checkers
| Maven module | Test class | Output path | Classes | Command |
|---|---|---|---|---|
| `log-analyzer` | `LALClassGenerator*Test` |
`oap-server/analyzer/log-analyzer/target/lal-generated-classes/` | 28 | `./mvnw
test -pl oap-server/analyzer/log-analyzer
-Dtest="LALClassGeneratorBasicTest,LALClassGeneratorConditionTest,LALClassGeneratorExtractorTest,LALClassGeneratorDefTest,LALClassGeneratorSinkTest"`
|
| `envoy-metrics-receiver-plugin` | `EnvoyAlsLalTest` |
`oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/target/lal-generated-classes/`
| 3 | `./mvnw test -pl
oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin
-Dtest=EnvoyAlsLalTest` |
| `mal-lal-v1-v2-checker` | `LalComparisonTest` |
`test/script-cases/scripts/lal/test-lal/**/*.generated-classes/` | 35 | `./mvnw
test -pl test/script-cases/script-runtime-with-groovy/mal-lal-v1-v2-checker
-Dtest=LalComparisonTest` |
| `mal-lal-v1-v2-checker` | `MalComparisonTest` |
`test/script-cases/scripts/mal/**/*.generated-classes/` | 1,324 | `./mvnw test
-pl test/script-cases/script-runtime-with-groovy/mal-lal-v1-v2-checker
-Dtest=MalComparisonTest` |
| `hierarchy-v1-v2-checker` | `HierarchyRuleComparisonTest` |
`test/script-cases/scripts/hierarchy-rule/*.generated-classes/` | 4 | `./mvnw
test -pl test/script-cases/script-runtime-with-groovy/hierarchy-v1-v2-checker
-Dtest=HierarchyRuleComparisonTest` |
| **Total** | | | **1,394** | |
### Shared DSL Testing Framework
New utilities in `oap-server/server-testing`
(`org.apache.skywalking.oap.server.testing.dsl`):
- `DslClassOutput.unitTestDir(dslType)` — standardized
`target/{dslType}-generated-classes/` for unit tests
- `DslClassOutput.checkerTestDir(sourceFile)` — standardized
`{baseName}.generated-classes/` for checkers
- `LalRuleLoader` / `MalRuleLoader` / `HierarchyRuleLoader` — shared rule
loading from YAML
- `LalLogDataBuilder` — shared LogData + proto extraLog building from test
input maps
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] LAL: add def local variable support with toJson/toJsonArray and type cast [skywalking]
wu-sheng commented on code in PR #13737:
URL: https://github.com/apache/skywalking/pull/13737#discussion_r2912580820
##
oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALBlockCodegen.java:
##
@@ -1209,6 +1523,19 @@ static String generateMethodArgs(
va.getSegments().get(0))).append("\"");
} else if (va.isNumberLiteral()) {
sb.append(va.getSegments().get(0));
+} else if (!va.getSegments().isEmpty()) {
+final String text = va.getSegments().get(0);
+if ("true".equals(text) || "false".equals(text)
+|| "null".equals(text)) {
+// Boolean or null literal
+sb.append(text);
+} else if (genCtx != null
+&& genCtx.localVars.containsKey(text)) {
+// Local def variable reference
+sb.append(genCtx.localVars.get(text).javaVarName);
+} else {
+sb.append("null");
Review Comment:
Fixed in c5911cc. Unknown identifier arguments now throw
`IllegalArgumentException` at compile time instead of silently emitting `null`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] LAL: add def local variable support with toJson/toJsonArray and type cast [skywalking]
wu-sheng commented on code in PR #13737:
URL: https://github.com/apache/skywalking/pull/13737#discussion_r2912580617
##
oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALBlockCodegen.java:
##
@@ -1140,6 +1172,288 @@ static String generateExtraLogAccess(
return prevVar;
}
+// Def statement codegen
+
+static void generateDefStatement(final StringBuilder sb,
+ final LALScriptModel.DefStatement def,
+ final LALClassGenerator.GenCtx genCtx) {
+final LALScriptModel.ValueAccess init = def.getInitializer();
+final String varName = def.getVarName();
+final String javaVar = "_" + varName;
Review Comment:
Fixed in c5911cc. Using `_def_` prefix now (e.g., `def config` →
`_def_config`) to avoid collision with reserved locals like `_p`, `_o`, `_tN`.
##
oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALBlockCodegen.java:
##
@@ -1174,29 +1488,29 @@ static void generateProcessRegistryCall(
// Utility methods
static String appendMethodSegment(final String current,
- final LALScriptModel.MethodSegment ms) {
+ final LALScriptModel.MethodSegment ms,
+ final LALClassGenerator.GenCtx genCtx) {
+final String mn = ms.getName();
+final String args = ms.getArguments().isEmpty()
+? "" : generateMethodArgs(ms.getArguments(), genCtx);
if (ms.isSafeNav()) {
-final String mn = ms.getName();
+// Special-cased helpers for common safe-nav methods on Object
if ("toString".equals(mn)) {
return "h.toString(" + current + ")";
} else if ("trim".equals(mn)) {
return "h.trim(" + current + ")";
-} else {
-throw new IllegalArgumentException(
-"Unsupported safe-nav method: ?." + mn + "()");
}
+// General safe-nav: null guard with ternary
+return "(" + current + " == null ? null : "
++ current + "." + mn + "(" + args + "))";
} else {
Review Comment:
Not an issue in practice. This general safe-nav path is only reachable for
untyped chains (non-def-var). In real LAL scripts, `?.` chains on untyped
values only call object-returning methods like `.get()`, `.getAsString()`,
`.toString()`, `.trim()`. The typed def-var path (`generateDefVarChain`)
already handles primitive boxing correctly. Adding type tracking to the untyped
path would require significant complexity for a case that doesn't occur in
practice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] LAL: add def local variable support with toJson/toJsonArray and type cast [skywalking]
wu-sheng commented on code in PR #13737:
URL: https://github.com/apache/skywalking/pull/13737#discussion_r2912565107
##
oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALBlockCodegen.java:
##
@@ -1140,6 +1172,288 @@ static String generateExtraLogAccess(
return prevVar;
}
+// Def statement codegen
+
+static void generateDefStatement(final StringBuilder sb,
+ final LALScriptModel.DefStatement def,
+ final LALClassGenerator.GenCtx genCtx) {
+final LALScriptModel.ValueAccess init = def.getInitializer();
+final String varName = def.getVarName();
+final String javaVar = "_" + varName;
Review Comment:
Good catch — will use `_def_` prefix to avoid collision with reserved locals
(`_p`, `_o`, `_tN`, etc.).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] LAL: add def local variable support with toJson/toJsonArray and type cast [skywalking]
Copilot commented on code in PR #13737:
URL: https://github.com/apache/skywalking/pull/13737#discussion_r2912527476
##
oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALBlockCodegen.java:
##
@@ -1140,6 +1172,288 @@ static String generateExtraLogAccess(
return prevVar;
}
+// Def statement codegen
+
+static void generateDefStatement(final StringBuilder sb,
+ final LALScriptModel.DefStatement def,
+ final LALClassGenerator.GenCtx genCtx) {
+final LALScriptModel.ValueAccess init = def.getInitializer();
+final String varName = def.getVarName();
+final String javaVar = "_" + varName;
Review Comment:
`def` variables are mapped to Java locals using `"_" + varName` (e.g., `def
p` -> `_p`). This can collide with compiler-reserved locals like `_p` (extraLog
cast), `_o` (output builder), `_tN` (proto cache), etc., leading to
uncompilable generated code when users pick those names. Consider using a
dedicated prefix for def vars (e.g., `__d_`/`_def_`) and/or detecting/resolving
collisions against reserved names before emitting declarations.
```suggestion
final String javaVar = "__d_" + varName;
```
##
oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALBlockCodegen.java:
##
@@ -1174,29 +1488,29 @@ static void generateProcessRegistryCall(
// Utility methods
static String appendMethodSegment(final String current,
- final LALScriptModel.MethodSegment ms) {
+ final LALScriptModel.MethodSegment ms,
+ final LALClassGenerator.GenCtx genCtx) {
+final String mn = ms.getName();
+final String args = ms.getArguments().isEmpty()
+? "" : generateMethodArgs(ms.getArguments(), genCtx);
if (ms.isSafeNav()) {
-final String mn = ms.getName();
+// Special-cased helpers for common safe-nav methods on Object
if ("toString".equals(mn)) {
return "h.toString(" + current + ")";
} else if ("trim".equals(mn)) {
return "h.trim(" + current + ")";
-} else {
-throw new IllegalArgumentException(
-"Unsupported safe-nav method: ?." + mn + "()");
}
+// General safe-nav: null guard with ternary
+return "(" + current + " == null ? null : "
++ current + "." + mn + "(" + args + "))";
} else {
Review Comment:
`appendMethodSegment` emits a generic safe-nav ternary for any method:
`(current == null ? null : current.method(...))`. If the method returns a
primitive (e.g., `String?.length()` -> `int`), this generates invalid Java due
to mixing `null` with a primitive in the ternary. Either restrict safe-nav
methods here (as previously) or add type-aware handling/boxing for primitive
return types when generating generic safe-nav method calls.
##
oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALBlockCodegen.java:
##
@@ -1209,6 +1523,19 @@ static String generateMethodArgs(
va.getSegments().get(0))).append("\"");
} else if (va.isNumberLiteral()) {
sb.append(va.getSegments().get(0));
+} else if (!va.getSegments().isEmpty()) {
+final String text = va.getSegments().get(0);
+if ("true".equals(text) || "false".equals(text)
+|| "null".equals(text)) {
+// Boolean or null literal
+sb.append(text);
+} else if (genCtx != null
+&& genCtx.localVars.containsKey(text)) {
+// Local def variable reference
+sb.append(genCtx.localVars.get(text).javaVarName);
+} else {
+sb.append("null");
Review Comment:
`generateMethodArgs` silently converts any non-literal identifier argument
that isn’t a known `def` variable into `null`. With `def` vars now supported as
method args, this makes typos or forward references (using a variable before
it’s declared) compile but behave incorrectly at runtime. Consider failing
compilation with a clear error when an identifier arg can’t be resolved to a
local `def` variable (instead of emitting `null`).
```suggestion
throw new IllegalStateException(
"Unknown identifier used as method argument: '" +
text + '\''
);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, ple
Re: [PR] LAL: add def local variable support with toJson/toJsonArray and type cast [skywalking]
wu-sheng commented on code in PR #13737:
URL: https://github.com/apache/skywalking/pull/13737#discussion_r2912409621
##
docs/en/concepts-and-designs/lal.md:
##
@@ -145,6 +145,89 @@ filter {
Extractors aim to extract metadata from the logs. The metadata can be a
service name, a service instance name, an
endpoint name, or even a trace ID, all of which can be associated with the
existing traces and metrics.
+ Local variables (`def`)
+
+You can use `def` to declare local variables in the extractor (or at the
filter level). This is useful when an
+expression is reused multiple times, or when you want to break a long chain
into readable steps.
+
+The syntax is:
+
+```
+def variableName = expression
+def variableName = expression as TypeName
+```
+
+The variable type is inferred from the initializer expression at compile time.
`def` is not limited to JSON — it works
+with any value access expression whose type is resolvable on the classpath,
including protobuf getter chains,
+`log.*` fields, and Gson JSON method chains. Subsequent method calls on the
variable are validated at compile time
+against the inferred type.
+
+You can optionally add an explicit `as` type cast to narrow the variable type.
The cast type can be a built-in
+type (`String`, `Long`, `Integer`, `Boolean`) or a fully qualified class name:
+
+```
+def value = someExpression as com.example.MyType
+```
+
+This is useful when the compiler infers a type that is too general (e.g.,
`Object` from a generic API return)
+and you know the concrete runtime type. The cast tells the compiler which type
to use for subsequent method chain
+validation. Note that `as` performs a Java cast — it does **not** convert
between types. For JSON conversion, use
+`toJson()` or `toJsonArray()` instead.
+
+The FQCN must be resolvable on the classpath at compile time. If the class is
not found, the OAP server will fail
+to start.
+
+Two built-in conversion functions are provided for JSON interoperability:
+
+- `toJson(expr)` — converts a value to a Gson `JsonObject`. Works with JSON
strings, `Map`, and protobuf `Struct`.
+- `toJsonArray(expr)` — converts a value to a Gson `JsonArray`. Works with
JSON array strings.
+
Review Comment:
Fixed in d990305. Added Map/List → Gson recursive conversion in
toJsonObject(Object) and toJsonArray(Object). Docs are now accurate.
##
oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/rt/LalRuntimeHelper.java:
##
@@ -180,6 +188,115 @@ public String tagValue(final String key) {
return "";
}
+// JSON conversion (for def variables)
+//
+// toJsonObject: converts to JsonObject. Overloaded for Struct, String,
Object.
+// toJsonArray: converts to JsonArray. Overloaded for String, Object.
+//
+// Primary use case: extract JWT claims from envoy ALS filter_metadata:
+// def jwt = toJson(parsed?.commonProperties?.metadata
+// ?.filterMetadataMap?.get("envoy.filters.http.jwt_authn"))
+// tag 'email':
jwt?.getAsJsonObject("payload")?.get("email")?.getAsString()
+
+/**
+ * Converts a protobuf {@link Struct} to a Gson {@link JsonObject}.
+ * Recursively converts nested Struct/Value/ListValue to Gson types.
+ */
+public JsonObject toJsonObject(final Struct struct) {
+if (struct == null) {
+return null;
+}
+return structToJsonObject(struct);
+}
+
+/**
+ * Parses a JSON string into a {@link JsonObject}.
+ */
+public JsonObject toJsonObject(final String s) {
+if (s == null || s.isEmpty()) {
+return null;
+}
+final JsonElement el = JsonParser.parseString(s);
+return el.isJsonObject() ? el.getAsJsonObject() : null;
+}
+
+/**
+ * Fallback for {@code Map.get()} erasure — dispatches to typed overloads.
+ */
+public JsonObject toJsonObject(final Object obj) {
+if (obj == null) {
+return null;
+}
+if (obj instanceof Struct) {
+return toJsonObject((Struct) obj);
+}
+if (obj instanceof String) {
+return toJsonObject((String) obj);
+}
+return null;
+}
Review Comment:
Fixed in d990305. toJsonObject(Object) now dispatches Map via a new
toJsonObject(Map) overload with recursive conversion. toJsonArray(Object) also
handles List input.
##
oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALBlockCodegen.java:
##
@@ -1140,6 +1172,283 @@ static String generateExtraLogAccess(
return prevVar;
}
+// Def statement codegen
+
+static void generateDefStatement(final StringBuilder sb,
+ final LALScriptModel.DefStatement def,
+
Re: [PR] LAL: add def local variable support with toJson/toJsonArray and type cast [skywalking]
wu-sheng commented on code in PR #13737:
URL: https://github.com/apache/skywalking/pull/13737#discussion_r2912349313
##
docs/en/concepts-and-designs/lal.md:
##
@@ -145,6 +145,89 @@ filter {
Extractors aim to extract metadata from the logs. The metadata can be a
service name, a service instance name, an
endpoint name, or even a trace ID, all of which can be associated with the
existing traces and metrics.
+ Local variables (`def`)
+
+You can use `def` to declare local variables in the extractor (or at the
filter level). This is useful when an
+expression is reused multiple times, or when you want to break a long chain
into readable steps.
+
+The syntax is:
+
+```
+def variableName = expression
+def variableName = expression as TypeName
+```
+
+The variable type is inferred from the initializer expression at compile time.
`def` is not limited to JSON — it works
+with any value access expression whose type is resolvable on the classpath,
including protobuf getter chains,
+`log.*` fields, and Gson JSON method chains. Subsequent method calls on the
variable are validated at compile time
+against the inferred type.
Review Comment:
Correct observation. Filter-level def lives in execute(), extractor/sink are
separate private methods with their own localVars scope. This is intentional —
filter-level def is useful for conditions before extractor/sink blocks, not for
passing variables into them. Will clarify the docs.
##
oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/rt/LalRuntimeHelper.java:
##
@@ -180,6 +188,115 @@ public String tagValue(final String key) {
return "";
}
+// JSON conversion (for def variables)
+//
+// toJsonObject: converts to JsonObject. Overloaded for Struct, String,
Object.
+// toJsonArray: converts to JsonArray. Overloaded for String, Object.
+//
+// Primary use case: extract JWT claims from envoy ALS filter_metadata:
+// def jwt = toJson(parsed?.commonProperties?.metadata
+// ?.filterMetadataMap?.get("envoy.filters.http.jwt_authn"))
+// tag 'email':
jwt?.getAsJsonObject("payload")?.get("email")?.getAsString()
+
+/**
+ * Converts a protobuf {@link Struct} to a Gson {@link JsonObject}.
+ * Recursively converts nested Struct/Value/ListValue to Gson types.
+ */
+public JsonObject toJsonObject(final Struct struct) {
+if (struct == null) {
+return null;
+}
+return structToJsonObject(struct);
+}
+
+/**
+ * Parses a JSON string into a {@link JsonObject}.
+ */
+public JsonObject toJsonObject(final String s) {
+if (s == null || s.isEmpty()) {
+return null;
+}
+final JsonElement el = JsonParser.parseString(s);
+return el.isJsonObject() ? el.getAsJsonObject() : null;
+}
+
+/**
+ * Fallback for {@code Map.get()} erasure — dispatches to typed overloads.
+ */
+public JsonObject toJsonObject(final Object obj) {
+if (obj == null) {
+return null;
+}
+if (obj instanceof Struct) {
+return toJsonObject((Struct) obj);
+}
+if (obj instanceof String) {
+return toJsonObject((String) obj);
+}
+return null;
+}
Review Comment:
Valid — same as the docs comment. The Map case is missing in the
toJsonObject(Object) dispatcher. Will add Map handling.
##
oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALExpressionExecutionTest.java:
##
@@ -318,6 +330,30 @@ private void executeAndAssert(
}
}
+/**
+ * Reads a field from the output builder if present, otherwise falls back
+ * to the proto value. The output builder (e.g. LogBuilder) stores
+ * extracted fields that haven't been flushed to LogData proto yet.
+ */
+private static String getOutputOrProto(final Object outputObj,
+final String fieldName,
+final String protoValue) {
+if (outputObj == null) {
+return protoValue;
+}
+try {
+final Field f = outputObj.getClass().getDeclaredField(fieldName);
+f.setAccessible(true);
+final Object val = f.get(outputObj);
+if (val != null) {
+return val.toString();
+}
+} catch (NoSuchFieldException | IllegalAccessException ignored) {
+// Field not on this output type — fall back to proto
+}
+return protoValue;
+}
Review Comment:
The test helper works correctly for the current output types. The field
names (service, serviceInstance, endpoint, layer) match LogBuilder which is the
default outputType. Custom outputTypes that use different setter n
Re: [PR] LAL: add def local variable support with toJson/toJsonArray and type cast [skywalking]
wu-sheng commented on code in PR #13737:
URL: https://github.com/apache/skywalking/pull/13737#discussion_r2912347726
##
oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALBlockCodegen.java:
##
@@ -1175,23 +1484,21 @@ static void generateProcessRegistryCall(
static String appendMethodSegment(final String current,
final LALScriptModel.MethodSegment ms) {
+final String mn = ms.getName();
+final String args = ms.getArguments().isEmpty()
+? "" : generateMethodArgs(ms.getArguments());
if (ms.isSafeNav()) {
-final String mn = ms.getName();
+// Special-cased helpers for common safe-nav methods on Object
if ("toString".equals(mn)) {
return "h.toString(" + current + ")";
} else if ("trim".equals(mn)) {
return "h.trim(" + current + ")";
-} else {
-throw new IllegalArgumentException(
-"Unsupported safe-nav method: ?." + mn + "()");
}
+// General safe-nav: null guard with ternary
+return "(" + current + " == null ? null : "
++ current + "." + mn + "(" + args + "))";
} else {
-if (ms.getArguments().isEmpty()) {
-return current + "." + ms.getName() + "()";
-} else {
-return current + "." + ms.getName() + "("
-+ generateMethodArgs(ms.getArguments()) + ")";
-}
+return current + "." + mn + "(" + args + ")";
}
}
Review Comment:
Already handled. generateDefVarChain() checks returnType.isPrimitive() at
lines 1326-1332 and wraps with boxing + null-guard. The appendMethodSegment
path is only used for non-def-var chains where the compile-time type is unknown
— in practice those only call toString()/trim() style methods.
##
oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/rt/LalRuntimeHelper.java:
##
@@ -180,6 +188,115 @@ public String tagValue(final String key) {
return "";
}
+// JSON conversion (for def variables)
+//
+// toJsonObject: converts to JsonObject. Overloaded for Struct, String,
Object.
+// toJsonArray: converts to JsonArray. Overloaded for String, Object.
+//
+// Primary use case: extract JWT claims from envoy ALS filter_metadata:
+// def jwt = toJson(parsed?.commonProperties?.metadata
+// ?.filterMetadataMap?.get("envoy.filters.http.jwt_authn"))
+// tag 'email':
jwt?.getAsJsonObject("payload")?.get("email")?.getAsString()
+
+/**
+ * Converts a protobuf {@link Struct} to a Gson {@link JsonObject}.
+ * Recursively converts nested Struct/Value/ListValue to Gson types.
+ */
+public JsonObject toJsonObject(final Struct struct) {
+if (struct == null) {
+return null;
+}
+return structToJsonObject(struct);
+}
+
+/**
+ * Parses a JSON string into a {@link JsonObject}.
+ */
+public JsonObject toJsonObject(final String s) {
+if (s == null || s.isEmpty()) {
+return null;
+}
+final JsonElement el = JsonParser.parseString(s);
+return el.isJsonObject() ? el.getAsJsonObject() : null;
+}
+
+/**
+ * Fallback for {@code Map.get()} erasure — dispatches to typed overloads.
+ */
+public JsonObject toJsonObject(final Object obj) {
+if (obj == null) {
+return null;
+}
+if (obj instanceof Struct) {
+return toJsonObject((Struct) obj);
+}
+if (obj instanceof String) {
+return toJsonObject((String) obj);
+}
+return null;
+}
+
+/**
+ * Parses a JSON string into a {@link JsonArray}.
+ */
+public JsonArray toJsonArray(final String s) {
+if (s == null || s.isEmpty()) {
+return null;
+}
+final JsonElement el = JsonParser.parseString(s);
+return el.isJsonArray() ? el.getAsJsonArray() : null;
+}
Review Comment:
Intentional fail-fast. LAL rules are configured by operators, not end users.
A malformed JSON body means the log source is sending unexpected data — failing
loudly is the right behavior so operators notice and fix the pipeline. Silently
returning null would hide data issues.
##
docs/en/concepts-and-designs/lal.md:
##
@@ -145,6 +145,89 @@ filter {
Extractors aim to extract metadata from the logs. The metadata can be a
service name, a service instance name, an
endpoint name, or even a trace ID, all of which can be associated with the
existing traces and metrics.
+ Local variables (`def`)
+
+You can use `def` to declare local variables in the extractor (or at the
filter level). This
Re: [PR] LAL: add def local variable support with toJson/toJsonArray and type cast [skywalking]
wu-sheng commented on code in PR #13737:
URL: https://github.com/apache/skywalking/pull/13737#discussion_r2912341607
##
oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALClassGeneratorTestBase.java:
##
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.skywalking.oap.log.analyzer.v2.compiler;
+
+import java.io.File;
+import javassist.ClassPool;
+import org.apache.skywalking.oap.log.analyzer.v2.dsl.LalExpression;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestInfo;
+
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+
+/**
+ * Shared base for LAL class generator tests.
+ *
+ * Provides a fresh {@link LALClassGenerator} per test and dumps generated
+ * {@code .class} files to {@code target/lal-generated-classes/} for
+ * inspection with tools like {@code javap -c}.
+ *
+ * Generated class names follow the pattern
+ * {@code {TestClassName}_{testMethodName}} for easy identification,
+ * e.g. {@code BasicTest_compileMinimalFilter}.
+ */
+abstract class LALClassGeneratorTestBase {
Review Comment:
Same previous added DSLClassGeneratorTest.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] LAL: add def local variable support with toJson/toJsonArray and type cast [skywalking]
wu-sheng commented on code in PR #13737:
URL: https://github.com/apache/skywalking/pull/13737#discussion_r2912339394
##
oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALClassGeneratorTestBase.java:
##
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.skywalking.oap.log.analyzer.v2.compiler;
+
+import java.io.File;
+import javassist.ClassPool;
+import org.apache.skywalking.oap.log.analyzer.v2.dsl.LalExpression;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.TestInfo;
+
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+
+/**
+ * Shared base for LAL class generator tests.
+ *
+ * Provides a fresh {@link LALClassGenerator} per test and dumps generated
+ * {@code .class} files to {@code target/lal-generated-classes/} for
+ * inspection with tools like {@code javap -c}.
+ *
+ * Generated class names follow the pattern
+ * {@code {TestClassName}_{testMethodName}} for easy identification,
+ * e.g. {@code BasicTest_compileMinimalFilter}.
+ */
+abstract class LALClassGeneratorTestBase {
Review Comment:
@wankai123 I added this as a kind example. All subclass tests are generating
the compiled codes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] LAL: add def local variable support with toJson/toJsonArray and type cast [skywalking]
Copilot commented on code in PR #13737:
URL: https://github.com/apache/skywalking/pull/13737#discussion_r2912090135
##
docs/en/concepts-and-designs/lal.md:
##
@@ -145,6 +145,89 @@ filter {
Extractors aim to extract metadata from the logs. The metadata can be a
service name, a service instance name, an
endpoint name, or even a trace ID, all of which can be associated with the
existing traces and metrics.
+ Local variables (`def`)
+
+You can use `def` to declare local variables in the extractor (or at the
filter level). This is useful when an
+expression is reused multiple times, or when you want to break a long chain
into readable steps.
+
+The syntax is:
+
+```
+def variableName = expression
+def variableName = expression as TypeName
+```
+
+The variable type is inferred from the initializer expression at compile time.
`def` is not limited to JSON — it works
+with any value access expression whose type is resolvable on the classpath,
including protobuf getter chains,
+`log.*` fields, and Gson JSON method chains. Subsequent method calls on the
variable are validated at compile time
+against the inferred type.
+
+You can optionally add an explicit `as` type cast to narrow the variable type.
The cast type can be a built-in
+type (`String`, `Long`, `Integer`, `Boolean`) or a fully qualified class name:
+
+```
+def value = someExpression as com.example.MyType
+```
+
+This is useful when the compiler infers a type that is too general (e.g.,
`Object` from a generic API return)
+and you know the concrete runtime type. The cast tells the compiler which type
to use for subsequent method chain
+validation. Note that `as` performs a Java cast — it does **not** convert
between types. For JSON conversion, use
+`toJson()` or `toJsonArray()` instead.
+
+The FQCN must be resolvable on the classpath at compile time. If the class is
not found, the OAP server will fail
+to start.
+
+Two built-in conversion functions are provided for JSON interoperability:
+
+- `toJson(expr)` — converts a value to a Gson `JsonObject`. Works with JSON
strings, `Map`, and protobuf `Struct`.
+- `toJsonArray(expr)` — converts a value to a Gson `JsonArray`. Works with
JSON array strings.
+
Review Comment:
The docs state `toJson(expr)` “Works with JSON strings, `Map`, and protobuf
`Struct`”, but the current runtime implementation only supports `String` and
`Struct` (and returns null for `Map`). Please either add Map conversion support
in `LalRuntimeHelper` or update this section so users don’t rely on unsupported
behavior.
##
oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALBlockCodegen.java:
##
@@ -1140,6 +1172,283 @@ static String generateExtraLogAccess(
return prevVar;
}
+// Def statement codegen
+
+static void generateDefStatement(final StringBuilder sb,
+ final LALScriptModel.DefStatement def,
+ final LALClassGenerator.GenCtx genCtx) {
+final LALScriptModel.ValueAccess init = def.getInitializer();
+final String varName = def.getVarName();
+final String javaVar = "_d" + genCtx.localVarCounter++;
+
+// Determine type and generate initializer expression
+Class resolvedType;
+final StringBuilder initExpr = new StringBuilder();
+
+if (init.getFunctionCallName() != null
+&& BUILTIN_FUNCTIONS.containsKey(init.getFunctionCallName())) {
+// Built-in function: toJson(...), toJsonArray(...)
+final Object[] info =
BUILTIN_FUNCTIONS.get(init.getFunctionCallName());
+final String helperMethod = (String) info[0];
+resolvedType = (Class) info[1];
+
+initExpr.append(helperMethod).append("(");
+if (!init.getFunctionCallArgs().isEmpty()) {
+generateValueAccess(initExpr,
+init.getFunctionCallArgs().get(0).getValue(), genCtx);
+} else {
+initExpr.append("null");
+}
+initExpr.append(")");
+} else {
Review Comment:
`generateDefStatement` ignores additional arguments for built-ins like
`toJson(...)`/`toJsonArray(...)` (it only emits the first arg, otherwise
`null`). If a script accidentally passes 2+ args, it will compile but behave
unexpectedly. Consider validating the arity for built-in functions and failing
compilation with a clear error when the arg count doesn’t match the expected
signature (exactly 1).
##
oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/rt/LalRuntimeHelper.java:
##
@@ -180,6 +188,115 @@ public String tagValue(final String key) {
return "";
}
+// JSON conversion (for def variables)
+//
+// toJsonObject: conve
[PR] LAL: add def local variable support with toJson/toJsonArray and type cast [skywalking]
wu-sheng opened a new pull request, #13737:
URL: https://github.com/apache/skywalking/pull/13737
### Add `def` local variable support in LAL DSL
- [x] Update the documentation to include this new feature.
- [x] Tests(including UT, IT, E2E) are added to verify the new feature.
**Grammar**: `def varName = expression` with optional `as TypeName` cast
(built-in types or FQCN).
**Built-in functions**: `toJson()` converts String/Map/protobuf `Struct` to
Gson `JsonObject`; `toJsonArray()` converts to `JsonArray`.
**Type inference**: Variable type inferred at compile time from initializer
expression. Supports null-safe navigation (`?.`) and method chaining on the
inferred type. Optional `as` cast narrows type when the inferred type is too
general (e.g., `Object` from generic API).
**Runtime helpers**: `LalRuntimeHelper.toJsonObject()` overloads for String,
Map, and protobuf Struct (recursive field conversion).
**Test reorganization**: Split monolithic `LALClassGeneratorTest` (641
lines) into 5 scoped test classes:
- `LALClassGeneratorBasicTest` (10 tests)
- `LALClassGeneratorConditionTest` (10 tests)
- `LALClassGeneratorExtractorTest` (10 tests)
- `LALClassGeneratorDefTest` (10 tests)
- `LALClassGeneratorSinkTest` (5 tests)
Plus shared `LALClassGeneratorTestBase` with `.class` file output to
`target/lal-generated-classes/` and `{TestClass}_{testMethod}` naming.
**ALS-specific tests**: `EnvoyAlsLalTest` in envoy receiver module — 3 tests
for JWT extraction from proto `filter_metadata` via chained `def` variables.
- [x] Update the [`CHANGES`
log](https://github.com/apache/skywalking/blob/master/docs/en/changes/changes.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
