This is an automated email from the ASF dual-hosted git repository. wu-sheng pushed a commit to branch feat/otlp-log-json-body in repository https://gitbox.apache.org/repos/asf/skywalking.git
commit c4cab41d86dd51f63c82047f52c3477a3d61f19e Author: Wu Sheng <[email protected]> AuthorDate: Thu Jul 2 21:17:08 2026 +0800 Parse OTLP text-body JSON in LAL json{} and fix abortOnFailure wiring - LAL json{} falls back to the text body when the JSON body is empty (OTLP delivers JSON-shaped bodies as text), normalizing the body to JSON on a successful parse so it persists with content type JSON. - Fix abortOnFailure being silently ignored in the v2 compiler: the flag is now baked into the generated json/yaml/text parser call, and parse failures / regexp non-matches are logged at WARN. Co-Authored-By: Claude Fable 5 <[email protected]> --- docs/en/changes/changes.md | 2 + docs/en/concepts-and-designs/lal.md | 4 ++ docs/en/setup/backend/log-otlp.md | 2 + .../analyzer/v2/compiler/LALClassGenerator.java | 8 ++- .../log/analyzer/v2/compiler/LALScriptParser.java | 11 ++-- .../analyzer/v2/dsl/spec/filter/FilterSpec.java | 64 +++++++++++++++++----- .../v2/dsl/spec/parser/AbstractParserSpec.java | 25 ++------- .../v2/dsl/spec/parser/TextParserSpec.java | 20 ++++--- .../v2/compiler/LALClassGeneratorBasicTest.java | 3 +- .../v2/compiler/LALScriptExecutionTest.java | 24 ++++++++ .../feature-cases/execution-basic.data.yaml | 36 ++++++++++++ .../test-lal/feature-cases/execution-basic.yaml | 44 +++++++++++++++ 12 files changed, 194 insertions(+), 49 deletions(-) diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md index a766359d48..b4e271e7fc 100644 --- a/docs/en/changes/changes.md +++ b/docs/en/changes/changes.md @@ -264,6 +264,8 @@ * Add a runtime-rule apply-status query. The cluster main now tracks each structural apply through a phase machine (`SchemaApplyCoordinator`: pending → DDL → fencing → rolling-out → applied, with `degraded` for a committed-but-unconfirmed apply — the cluster schema fence did not confirm within the timeout, in which case the lagging data-node ids are surfaced as `fenceLaggards` and dispatch is resumed anyway, or the local commit-tail threw — and `failed` carrying the specific reason). The [...] * Push runtime-rule convergence to peers on commit. After a successful structural apply — and on the `commit_deferred` path, where the DB row is durable but this node's commit-tail threw — the main broadcasts a `NotifyApplied` admin-internal RPC so peers reconcile against the just-persisted DB row immediately, instead of waiting up to one refresh tick (~30s) to notice it. The fan-out runs off the REST response thread (fire-and-forget on a daemon executor) so an unreachable peer's per-cal [...] * Fix BanyanDB peer nodes permanently flooding `<metric> is not registered`, and a follow-on case where a peer kept translating writes with a stale schema shape after a runtime-rule reshape, when a node held a live persist worker but its local `MetadataRegistry` schema cache was missing or stale for that model — a `withoutSchemaChange` peer apply or a runtime-rule bundled fall-over rebuilt the dispatch worker but skipped the local-cache populate, and the registry was insert-only (never e [...] +* Support LAL `json {}` parsing JSON content delivered in a plain-text log body. The parser reads the native protocol's JSON body first; when that is empty, it tries the text body as JSON — e.g. the OTLP log receiver maps every OTLP string body to a text body, even JSON-shaped ones, so previously-aborting `json {}` rules on OTLP-fed layers now work without any receiver or protocol change. A parse failure keeps the existing `abortOnFailure` behavior; on a successful parse from a text body [...] +* Fix the LAL parser `abortOnFailure` option being silently ignored in the v2 (ANTLR4) compiler. The DSL flag was parsed into the rule model but never emitted into the generated code, so every `json {}` / `yaml {}` / `text { regexp }` block always used the built-in default and `abortOnFailure false` had no effect. The compiler now bakes each rule's flag into the generated parser call (its default is `true`, as documented), and a parse failure or regexp non-match is now logged at WARN ins [...] * Fix a v2 MAL `CounterWindow` key collision: `rate()` / `increase()` / `irate()` keyed each counter's sliding window on the rule's output metric name (the same for every input metric of a rule) instead of the counter's own name, so two or more counters that reduce to the same label set after `.sum(...)` shared one window and computed rates against each other's values — fabricating non-zero rates from unchanged counters (e.g. the BanyanDB liaison gRPC error rate read a steady non-zero of [...] * Fix the v2 MAL Elvis operator `?:` to honor Groovy-falsy semantics. It compiled to `Optional.ofNullable(primary).orElse(fallback)`, applying the fallback only when the primary is `null`, so an empty-string primary kept `""` instead — e.g. a BanyanDB liaison `ServiceInstance` stored `node_type=""` rather than `n/a`, because `.sum([...,'node_type'])` fills an absent group-by label with `""`. The fallback now applies for falsy primaries such as null, false, numeric zero, and empty strings [...] * SWIP-15: rebuild BanyanDB self-observability around the cluster / container / group model (requires BanyanDB 0.11+). A BanyanDB cluster is modeled as one `Service`, each container as a `ServiceInstance` (role/tier as attributes), and each storage group as an `Endpoint`. The `otel-rules/banyandb/` rules are category-separated by role (`node_*` / `liaison_*` / `data_*` / `lifecycle_*`) and by data type (`measure_*` / `stream_*` / `trace_*` / `property_*`), mirroring the upstream FODC-pro [...] diff --git a/docs/en/concepts-and-designs/lal.md b/docs/en/concepts-and-designs/lal.md index 74acc3f7e3..793fdd7ac7 100644 --- a/docs/en/concepts-and-designs/lal.md +++ b/docs/en/concepts-and-designs/lal.md @@ -219,6 +219,10 @@ filter { } ``` +`json` reads the JSON body of the [native log protocol](../api/log-data-protocol.md). When the JSON body is empty but a plain-text body is present — for example, the [OTLP log receiver](../setup/backend/log-otlp.md) delivers every string body as text, even JSON-shaped ones — the parser tries the text body as JSON instead. A parse failure of either form follows `abortOnFailure`. + +When `json` parses successfully from a text body, the log is normalized to a JSON body: it is stored with content type `JSON` and the original text content is no longer available (`log.body` reads the JSON content). This makes a JSON-shaped log delivered over any transport persist and render as JSON once a `json {}` rule matches it. + #### `yaml` ``` diff --git a/docs/en/setup/backend/log-otlp.md b/docs/en/setup/backend/log-otlp.md index 77c56daa94..5390adac41 100644 --- a/docs/en/setup/backend/log-otlp.md +++ b/docs/en/setup/backend/log-otlp.md @@ -53,3 +53,5 @@ And several attributes are optional as add-on information for the logs before an - `service.instance`: the instance name that generates the logs. The default value is empty. Note, that these attributes should be set manually through OpenTelemetry SDK or through [attribute#insert in OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/attributesprocessor/README.md). + +The OTLP log record body is delivered to [LAL](../../concepts-and-designs/lal.md) as a plain-text body, even when the string content is JSON-shaped. A LAL rule can still parse such content with the `json {}` parser — when the JSON body is empty and a text body is present, `json {}` tries the text as JSON. On a successful parse the log is normalized to a JSON body, so it is stored and rendered as JSON. See [`json`](../../concepts-and-designs/lal.md#json) for details. diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALClassGenerator.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALClassGenerator.java index 082529d700..4c0cc8d7bc 100644 --- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALClassGenerator.java +++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALClassGenerator.java @@ -757,16 +757,18 @@ public final class LALClassGenerator { if (tp.getRegexpPattern() != null) { sb.append(" filterSpec.textWithRegexp(ctx, \"") .append(LALCodegenHelper.escapeJava(tp.getRegexpPattern())) - .append("\");\n"); + .append("\", ").append(tp.isAbortOnFailure()).append(");\n"); } else { sb.append(" filterSpec.text(ctx);\n"); } LALCodegenHelper.emitCaptureCall(sb, "Parser", genCtx.ruleName, 0, "ctx", ""); } else if (stmt instanceof LALScriptModel.JsonParser) { - sb.append(" filterSpec.json(ctx);\n"); + sb.append(" filterSpec.json(ctx, ") + .append(((LALScriptModel.JsonParser) stmt).isAbortOnFailure()).append(");\n"); LALCodegenHelper.emitCaptureCall(sb, "Parser", genCtx.ruleName, 0, "ctx", ""); } else if (stmt instanceof LALScriptModel.YamlParser) { - sb.append(" filterSpec.yaml(ctx);\n"); + sb.append(" filterSpec.yaml(ctx, ") + .append(((LALScriptModel.YamlParser) stmt).isAbortOnFailure()).append(");\n"); LALCodegenHelper.emitCaptureCall(sb, "Parser", genCtx.ruleName, 0, "ctx", ""); } else if (stmt instanceof LALScriptModel.AbortStatement) { sb.append(" filterSpec.abort(ctx);\n"); diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALScriptParser.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALScriptParser.java index 38c8942337..3ff9e58f6e 100644 --- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALScriptParser.java +++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALScriptParser.java @@ -151,7 +151,7 @@ public final class LALScriptParser { private static FilterStatement visitParserBlock(final LALParser.ParserBlockContext ctx) { if (ctx.textBlock() != null) { String pattern = null; - boolean abortOnFail = false; + boolean abortOnFail = true; if (ctx.textBlock().textContent() != null) { for (final LALParser.RegexpStatementContext regCtx : ctx.textBlock().textContent().regexpStatement()) { @@ -176,13 +176,14 @@ public final class LALScriptParser { /** * Extract the {@code abortOnFailure} flag from an optional - * {@code abortOnFailureStatement} node, defaulting to {@code false} - * when the statement is absent. Shared between the JSON and YAML - * parser blocks (text uses a list form, handled inline). + * {@code abortOnFailureStatement} node, defaulting to {@code true} + * when the statement is absent — the documented default for every parser. + * Shared between the JSON and YAML parser blocks (text uses a list form, + * handled inline). */ private static boolean extractAbortOnFail( final LALParser.AbortOnFailureStatementContext ctx) { - return ctx != null && parseBoolText(ctx.boolValue().getText()); + return ctx == null || parseBoolText(ctx.boolValue().getText()); } private static boolean parseBoolText(final String text) { diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsl/spec/filter/FilterSpec.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsl/spec/filter/FilterSpec.java index cf265c7a54..5e259ec339 100644 --- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsl/spec/filter/FilterSpec.java +++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsl/spec/filter/FilterSpec.java @@ -22,7 +22,9 @@ import com.fasterxml.jackson.core.type.TypeReference; import java.util.Arrays; import java.util.List; import java.util.Map; +import org.apache.skywalking.apm.network.logging.v3.JSONLog; import org.apache.skywalking.apm.network.logging.v3.LogData; +import org.apache.skywalking.apm.network.logging.v3.LogDataBody; import org.apache.skywalking.oap.log.analyzer.v2.dsl.ExecutionContext; import org.apache.skywalking.oap.log.analyzer.v2.dsl.spec.AbstractSpec; import org.apache.skywalking.oap.log.analyzer.v2.dsl.spec.extractor.MetricExtractor; @@ -99,36 +101,66 @@ public class FilterSpec extends AbstractSpec { /** * LAL {@code text { regexp '...' }} — applies a named-group regexp to the * log body text. Matched groups are stored in {@code ctx.parsed().getMatcher()} - * and accessed via {@code parsed.groupName} in the LAL script. + * and accessed via {@code parsed.groupName} in the LAL script. {@code abortOnFailure} + * is the rule's compile-time flag (default {@code true}): a non-matching body aborts + * the filter chain only when it is set. The flag travels as a parameter rather than + * as parser-spec state because one {@code FilterSpec} instance serves every compiled + * rule concurrently. */ - public void textWithRegexp(final ExecutionContext ctx, final String regexp) { + public void textWithRegexp(final ExecutionContext ctx, final String regexp, + final boolean abortOnFailure) { if (ctx.shouldAbort()) { return; } - textParser.regexp(ctx, regexp); + textParser.regexp(ctx, regexp, abortOnFailure); } /** - * LAL {@code json {}} — parses {@code LogData.body.json.json} into a - * {@code Map<String, Object>} and stores it in {@code ctx.parsed()}. - * Metadata fields (service, serviceInstance, endpoint, layer, timestamp) + * LAL {@code json {}} — parses the JSON log body into a {@code Map<String, Object>} + * and stores it in {@code ctx.parsed()}. Reads {@code LogData.body.json.json}; when + * that is empty but a text body is present, falls back to parsing + * {@code LogData.body.text.text} as JSON — the OTLP log receiver delivers every + * string body (even JSON-shaped ones) as a text body, and a rule that declared + * {@code json {}} means the content is JSON regardless of which body case carried it. + * A parse failure is logged at WARN and then honors {@code abortOnFailure} (the rule's + * compile-time flag, default {@code true}); the flag travels as a parameter rather than + * as parser-spec state because one {@code FilterSpec} instance serves every compiled + * rule concurrently. + * + * <p>On a successful parse that fell back to the text body, the body is rewritten to a + * JSON body carrying the same content, so the log persists as {@code ContentType.JSON} + * (persistence derives the stored content type from the body case in {@code + * LogBuilder.toLog()}); the original text case is dropped and no longer readable. + * + * <p>Metadata fields (service, serviceInstance, endpoint, layer, timestamp) * are also added to the map via {@code putIfAbsent}, so body values take * priority while metadata fields serve as fallback — matching v1 Groovy * {@code Binding.Parsed.getAt(key)} behavior. */ - public void json(final ExecutionContext ctx) { + public void json(final ExecutionContext ctx, final boolean abortOnFailure) { if (ctx.shouldAbort()) { return; } try { final LogData.Builder logData = (LogData.Builder) ctx.input(); - final Map<String, Object> parsed = jsonParser.create().readValue( - logData.getBody().getJson().getJson(), parsedType - ); + final LogDataBody body = logData.getBody(); + String content = body.getJson().getJson(); + final boolean fromText = content.isEmpty(); + if (fromText) { + content = body.getText().getText(); + } + final Map<String, Object> parsed = jsonParser.create().readValue(content, parsedType); addMetadataFields(parsed, ctx.metadata()); ctx.parsed(parsed); + if (fromText) { + logData.setBody(LogDataBody.newBuilder() + .setJson(JSONLog.newBuilder().setJson(content).build()) + .build()); + } } catch (final Exception e) { - if (jsonParser.abortOnFailure()) { + LOGGER.warn("LAL json parser failed to parse the log body (service={}, abortOnFailure={}): {}", + ctx.metadata().getService(), abortOnFailure, e.getMessage()); + if (abortOnFailure) { ctx.abort(); } } @@ -137,9 +169,11 @@ public class FilterSpec extends AbstractSpec { /** * LAL {@code yaml {}} — parses {@code LogData.body.yaml.yaml} into a * {@code Map<String, Object>} and stores it in {@code ctx.parsed()}. - * Metadata fields are added the same way as {@link #json(ExecutionContext)}. + * Metadata fields and {@code abortOnFailure} are handled the same way as + * {@link #json(ExecutionContext, boolean)}: a parse failure is logged at WARN + * before honoring the flag. */ - public void yaml(final ExecutionContext ctx) { + public void yaml(final ExecutionContext ctx, final boolean abortOnFailure) { if (ctx.shouldAbort()) { return; } @@ -151,7 +185,9 @@ public class FilterSpec extends AbstractSpec { addMetadataFields(parsed, ctx.metadata()); ctx.parsed(parsed); } catch (final Exception e) { - if (yamlParser.abortOnFailure()) { + LOGGER.warn("LAL yaml parser failed to parse the log body (service={}, abortOnFailure={}): {}", + ctx.metadata().getService(), abortOnFailure, e.getMessage()); + if (abortOnFailure) { ctx.abort(); } } diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsl/spec/parser/AbstractParserSpec.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsl/spec/parser/AbstractParserSpec.java index d27e4d42dc..64aed29305 100644 --- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsl/spec/parser/AbstractParserSpec.java +++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsl/spec/parser/AbstractParserSpec.java @@ -18,32 +18,19 @@ package org.apache.skywalking.oap.log.analyzer.v2.dsl.spec.parser; -import lombok.experimental.Accessors; import org.apache.skywalking.oap.log.analyzer.v2.dsl.spec.AbstractSpec; import org.apache.skywalking.oap.log.analyzer.v2.provider.LogAnalyzerModuleConfig; import org.apache.skywalking.oap.server.library.module.ModuleManager; -@Accessors +/** + * Base of the shared parser specs. Per-rule options such as {@code abortOnFailure} are NOT + * state on the spec — one spec instance serves every compiled rule concurrently, so the v2 + * compiler bakes each rule's flag into the generated call site (see + * {@code LALClassGenerator#generateFilterStatement}) and it travels as a method parameter. + */ public class AbstractParserSpec extends AbstractSpec { - /** - * Whether the filter chain should abort when parsing the logs failed. - * - * Failing to parse the logs means either parsing throws exceptions or the logs not matching the - * desired patterns. - */ - private boolean abortOnFailure = true; - public AbstractParserSpec(final ModuleManager moduleManager, final LogAnalyzerModuleConfig moduleConfig) { super(moduleManager, moduleConfig); } - - @SuppressWarnings("unused") // used in user LAL scripts - public void abortOnFailure(final boolean abortOnFailure) { - this.abortOnFailure = abortOnFailure; - } - - public boolean abortOnFailure() { - return this.abortOnFailure; - } } diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsl/spec/parser/TextParserSpec.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsl/spec/parser/TextParserSpec.java index a0f07926c2..a404b70590 100644 --- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsl/spec/parser/TextParserSpec.java +++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsl/spec/parser/TextParserSpec.java @@ -20,32 +20,38 @@ package org.apache.skywalking.oap.log.analyzer.v2.dsl.spec.parser; import java.util.regex.Matcher; import java.util.regex.Pattern; +import lombok.extern.slf4j.Slf4j; import org.apache.skywalking.apm.network.logging.v3.LogData; import org.apache.skywalking.oap.log.analyzer.v2.dsl.ExecutionContext; import org.apache.skywalking.oap.log.analyzer.v2.provider.LogAnalyzerModuleConfig; import org.apache.skywalking.oap.server.library.module.ModuleManager; +@Slf4j public class TextParserSpec extends AbstractParserSpec { public TextParserSpec(final ModuleManager moduleManager, final LogAnalyzerModuleConfig moduleConfig) { super(moduleManager, moduleConfig); } - public void regexp(final ExecutionContext ctx, final String regexp) { - regexp(ctx, Pattern.compile(regexp)); + public void regexp(final ExecutionContext ctx, final String regexp, final boolean abortOnFailure) { + regexp(ctx, Pattern.compile(regexp), abortOnFailure); } - public void regexp(final ExecutionContext ctx, final Pattern pattern) { + public void regexp(final ExecutionContext ctx, final Pattern pattern, final boolean abortOnFailure) { if (ctx.shouldAbort()) { return; } - final LogData.Builder log = (LogData.Builder) ctx.input(); - final Matcher matcher = pattern.matcher(log.getBody().getText().getText()); + final LogData.Builder logData = (LogData.Builder) ctx.input(); + final Matcher matcher = pattern.matcher(logData.getBody().getText().getText()); final boolean matched = matcher.find(); if (matched) { ctx.parsed(matcher); - } else if (abortOnFailure()) { - ctx.abort(); + } else { + log.warn("LAL text parser regexp did not match the log body (service={}, abortOnFailure={})", + ctx.metadata().getService(), abortOnFailure); + if (abortOnFailure) { + ctx.abort(); + } } } diff --git a/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALClassGeneratorBasicTest.java b/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALClassGeneratorBasicTest.java index 4e9c3310af..0330d7a1e1 100644 --- a/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALClassGeneratorBasicTest.java +++ b/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALClassGeneratorBasicTest.java @@ -80,7 +80,8 @@ class LALClassGeneratorBasicTest extends LALClassGeneratorTestBase { compileAndAssert(dsl); final String source = generator.generateSource(dsl); assertNotNull(source); - assertTrue(source.contains("filterSpec.json(ctx)")); + // The codegen bakes the rule's abortOnFailure flag (default true) into the call. + assertTrue(source.contains("filterSpec.json(ctx, true)")); assertTrue(source.contains("filterSpec.sink(ctx)")); } diff --git a/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALScriptExecutionTest.java b/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALScriptExecutionTest.java index 3a24bb294b..1cdca79c46 100644 --- a/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALScriptExecutionTest.java +++ b/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/v2/compiler/LALScriptExecutionTest.java @@ -30,6 +30,7 @@ import java.util.ServiceLoader; import com.google.protobuf.Message; import org.apache.skywalking.apm.network.common.v3.KeyStringValuePair; import org.apache.skywalking.apm.network.logging.v3.LogData; +import org.apache.skywalking.apm.network.logging.v3.LogDataBody; import org.apache.skywalking.oap.log.analyzer.v2.module.LogAnalyzerModule; import org.apache.skywalking.oap.log.analyzer.v2.provider.LogAnalyzerModuleConfig; import org.apache.skywalking.oap.log.analyzer.v2.spi.LALSourceTypeProvider; @@ -249,6 +250,10 @@ class LALScriptExecutionTest { case "timestamp": assertOutputField(ruleName, output, "timestamp", expected); break; + case "contentType": + assertEquals(expected, bodyContentType(logBuilder), + ruleName + ": persisted body content type mismatch"); + break; default: if (key.startsWith("tag.")) { final String tagKey = key.substring(4); @@ -275,6 +280,25 @@ class LALScriptExecutionTest { } } + /** The content type {@code LogBuilder.toLog()} would persist, derived from the body + * oneof case of the (possibly LAL-rewritten) input. */ + private static String bodyContentType(final LogData.Builder logBuilder) { + if (logBuilder == null) { + return "NONE"; + } + final LogDataBody body = logBuilder.getBody(); + if (body.hasJson()) { + return "JSON"; + } + if (body.hasYaml()) { + return "YAML"; + } + if (body.hasText()) { + return "TEXT"; + } + return "NONE"; + } + private static final Map<String, String[]> FIELD_GETTER_CANDIDATES; static { diff --git a/oap-server/analyzer/log-analyzer/src/test/resources/scripts/lal/test-lal/feature-cases/execution-basic.data.yaml b/oap-server/analyzer/log-analyzer/src/test/resources/scripts/lal/test-lal/feature-cases/execution-basic.data.yaml index 7c67febff0..579ed5fc2c 100644 --- a/oap-server/analyzer/log-analyzer/src/test/resources/scripts/lal/test-lal/feature-cases/execution-basic.data.yaml +++ b/oap-server/analyzer/log-analyzer/src/test/resources/scripts/lal/test-lal/feature-cases/execution-basic.data.yaml @@ -199,3 +199,39 @@ auto-layer-abort: os.name: "Android" expect: abort: true + +# json {} falls back to the text body when the JSON body is empty — the OTLP log +# receiver wraps every string body as text, even JSON-shaped ones. On success the +# body is normalized to JSON, so the log persists with content type JSON. +json-text-fallback: + body-type: text + body: '{"service":"otlp-svc","env":"prod"}' + expect: + save: true + abort: false + service: otlp-svc + tag.env: prod + contentType: JSON + +# Non-JSON text under the fallback still aborts (default abortOnFailure true). +json-text-fallback-invalid: + body-type: text + body: 'plain non-json log line' + expect: + abort: true + +# abortOnFailure false: a json parse failure must NOT abort — the log flows through. +json-abort-on-failure-false: + body-type: text + body: 'plain non-json log line' + expect: + abort: false + save: true + +# abortOnFailure false on a text regexp: a no-match must NOT abort. +text-abort-on-failure-false: + body-type: text + body: 'a line that does not match the level pattern' + expect: + abort: false + save: true diff --git a/oap-server/analyzer/log-analyzer/src/test/resources/scripts/lal/test-lal/feature-cases/execution-basic.yaml b/oap-server/analyzer/log-analyzer/src/test/resources/scripts/lal/test-lal/feature-cases/execution-basic.yaml index 45a1398854..ea4e180e0c 100644 --- a/oap-server/analyzer/log-analyzer/src/test/resources/scripts/lal/test-lal/feature-cases/execution-basic.yaml +++ b/oap-server/analyzer/log-analyzer/src/test/resources/scripts/lal/test-lal/feature-cases/execution-basic.yaml @@ -324,3 +324,47 @@ rules: } sink {} } + + - name: json-text-fallback + layer: GENERAL + dsl: | + filter { + json {} + extractor { + service parsed.service as String + tag env: parsed.env as String + } + sink {} + } + + - name: json-text-fallback-invalid + layer: GENERAL + dsl: | + filter { + json {} + extractor { + service parsed.service as String + } + sink {} + } + + - name: json-abort-on-failure-false + layer: GENERAL + dsl: | + filter { + json { + abortOnFailure false + } + sink {} + } + + - name: text-abort-on-failure-false + layer: GENERAL + dsl: | + filter { + text { + regexp "^(?<level>INFO|WARN|ERROR) (?<msg>.*)$" + abortOnFailure false + } + sink {} + }
