This is an automated email from the ASF dual-hosted git repository. wu-sheng pushed a commit to branch fix/envoy-als-lal-debug-rendering in repository https://gitbox.apache.org/repos/asf/skywalking.git
commit 94693a6d2d49123701a45c1e9d08b860f5b01e27 Author: Wu Sheng <[email protected]> AuthorDate: Tue Jun 30 09:57:22 2026 +0800 Decode Istio metadata-exchange peer in Envoy ALS rendering; harden proto-to-JSON The LAL live-debugger's INPUT sample for Envoy ALS traffic showed a `jsonformat-failed` envelope instead of the entry, because the printer in `LalPayloadDebugDump` had no `JsonFormat.TypeRegistry` and threw on the `google.protobuf.Any`-typed `common_properties.filter_state_objects`. The same bare `ProtoBufJsonUtils.toJSON` on the output builder's `bindInput` (which the generated `execute()` calls eagerly, before debug capture) could throw on an unregistered injected `Any` and abort the whole rule, dropping the mesh log. This change: - Adds a well-known-type `TypeRegistry` plus a sanitizer so a single un-printable field can no longer blank the whole entry: an unresolvable, no-slash, or corrupt-bytes `Any` degrades to an `@unresolved` placeholder, and a non-finite `Value` double (NaN/Infinity) is rendered as a string. - Decodes the predictable Istio metadata-exchange peer into readable metadata (pod/namespace/labels) — legacy Wasm `wasm.*_peer` (`Any{BytesValue}` -> FlatBuffer, via the existing `ServiceMetaInfoAdapter`) and modern `*_peer` (`Any{Struct}`) — for both the live-debug INPUT sample and the persisted log `content`, covering HTTP and TCP access logs. - Wires decoding through a new `LalInputDebugRenderer` SPI (`EnvoyAlsHttpDebugRenderer` / `EnvoyAlsTcpDebugRenderer`) so `log-analyzer` reaches the receiver-side decoders without depending on the Envoy receiver, mirroring the existing `LALSourceTypeProvider` seam. --- docs/en/changes/changes.md | 1 + .../v2/dsldebug/LalInputDebugRenderers.java | 56 ++++ .../analyzer/v2/dsldebug/LalPayloadDebugDump.java | 245 ++++++++++++++++- .../log/analyzer/v2/spi/LalInputDebugRenderer.java | 63 +++++ .../v2/dsldebug/LalPayloadDebugDumpTest.java | 154 +++++++++++ .../envoy/als/mx/EnvoyAlsHttpDebugRenderer.java | 47 ++++ .../receiver/envoy/als/mx/EnvoyAlsJsonUtils.java | 157 +++++++++++ .../envoy/als/mx/EnvoyAlsTcpDebugRenderer.java | 49 ++++ .../envoy/persistence/EnvoyAccessLogBuilder.java | 13 +- ...g.oap.log.analyzer.v2.spi.LalInputDebugRenderer | 20 ++ .../envoy/als/mx/EnvoyAlsDebugRendererTest.java | 300 +++++++++++++++++++++ .../persistence/EnvoyAccessLogBuilderTest.java | 70 +++++ 12 files changed, 1167 insertions(+), 8 deletions(-) diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md index 3ef88c95d8..6385d78b30 100644 --- a/docs/en/changes/changes.md +++ b/docs/en/changes/changes.md @@ -317,6 +317,7 @@ * Align the default BanyanDB stage `segmentInterval` values so each coarser stage is an integer multiple of the finer one (`records` cold `3` → `4`, `metricsMinute` cold `5` → `6`, `metricsHour` warm `7` → `10` and cold `15` → `20`), keeping hot → warm → cold lifecycle migration on the cheap whole-segment fast path. * Fix: `layer-extensions.yml` is now excluded from the `skywalking-oap` jar and shipped to the distribution `config/` directory, so an operator-edited `config/layer-extensions.yml` is no longer shadowed by the empty template bundled in the jar. Because the OAP launch script puts `oap-libs/*.jar` ahead of `config/` on the classpath, `ResourceUtils.read("layer-extensions.yml")` previously always resolved the jar-bundled `layers: []` and silently ignored the operator's file — custom layers [...] * Fix: the v2 MAL compiler now resolves custom layers referenced as `Layer.NAME` in an expression. A custom layer declared through a `layerDefinitions:` block (or `layer-extensions.yml` / the `LayerExtension` SPI) has no generated `Layer.*` static field, so `service(['svc'], Layer.IOT_FLEET)` previously failed code generation because `Layer` has no `IOT_FLEET` field. The compiler now lowers every `Layer.NAME` static-field reference to a runtime `Layer.nameOf("NAME")` registry lookup, so [...] +* Fix Envoy ALS rendering for the LAL live-debugger and the persisted log `content`: an Istio metadata-exchange peer in `common_properties.filter_state_objects` (legacy Wasm `wasm.*_peer` = `Any{BytesValue}` wrapping a FlatBuffer, or modern `*_peer` = `Any{Struct}`) is now decoded into the readable peer metadata (pod / namespace / labels) instead of an opaque `jsonformat-failed` envelope or base64. The serialization is hardened so a single un-printable field can no longer blank the whole [...] #### UI * Add Airflow layer dashboards and menu i18n under Workflow Scheduler in Horizon UI (SWIP-7). diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsldebug/LalInputDebugRenderers.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsldebug/LalInputDebugRenderers.java new file mode 100644 index 0000000000..51f82b780b --- /dev/null +++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsldebug/LalInputDebugRenderers.java @@ -0,0 +1,56 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.log.analyzer.v2.dsldebug; + +import com.google.protobuf.Message; +import java.util.HashMap; +import java.util.Map; +import java.util.ServiceLoader; +import org.apache.skywalking.oap.log.analyzer.v2.spi.LalInputDebugRenderer; + +/** + * Indexes {@link LalInputDebugRenderer} SPI implementations by their declared + * input type. Discovery is pull-based via {@link ServiceLoader} (the same + * mechanism the {@code LALSourceTypeProvider} seam uses), so a receiver + * plugin's renderer is found on first use without any registration call. + */ +final class LalInputDebugRenderers { + + private static final Map<Class<?>, LalInputDebugRenderer> BY_TYPE = load(); + + private LalInputDebugRenderers() { + } + + private static Map<Class<?>, LalInputDebugRenderer> load() { + final Map<Class<?>, LalInputDebugRenderer> map = new HashMap<>(); + for (final LalInputDebugRenderer renderer : ServiceLoader.load(LalInputDebugRenderer.class)) { + map.put(renderer.inputType(), renderer); + } + return map; + } + + /** + * Render {@code msg} via a registered renderer, or {@code null} when no + * renderer is registered for its type or the renderer declined. + */ + static String render(final Message msg) { + final LalInputDebugRenderer renderer = BY_TYPE.get(msg.getClass()); + return renderer == null ? null : renderer.render(msg); + } +} diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsldebug/LalPayloadDebugDump.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsldebug/LalPayloadDebugDump.java index c10db4954e..48d5a23d4d 100644 --- a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsldebug/LalPayloadDebugDump.java +++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/dsldebug/LalPayloadDebugDump.java @@ -21,9 +21,28 @@ package org.apache.skywalking.oap.log.analyzer.v2.dsldebug; import com.google.gson.JsonElement; import com.google.gson.JsonObject; import com.google.gson.JsonParser; +import com.google.protobuf.Any; +import com.google.protobuf.BoolValue; +import com.google.protobuf.BytesValue; +import com.google.protobuf.Descriptors.Descriptor; +import com.google.protobuf.Descriptors.FieldDescriptor; +import com.google.protobuf.DoubleValue; +import com.google.protobuf.Duration; +import com.google.protobuf.DynamicMessage; +import com.google.protobuf.FloatValue; +import com.google.protobuf.Int32Value; +import com.google.protobuf.Int64Value; +import com.google.protobuf.ListValue; import com.google.protobuf.Message; import com.google.protobuf.MessageOrBuilder; +import com.google.protobuf.StringValue; +import com.google.protobuf.Struct; +import com.google.protobuf.Timestamp; +import com.google.protobuf.UInt32Value; +import com.google.protobuf.UInt64Value; +import com.google.protobuf.Value; import com.google.protobuf.util.JsonFormat; +import java.util.Base64; import org.apache.skywalking.apm.network.logging.v3.LogData; import org.apache.skywalking.oap.server.core.dsldebug.LogDataDebugDump; import org.apache.skywalking.oap.server.core.dsldebug.ToJson; @@ -58,8 +77,40 @@ import org.apache.skywalking.oap.server.core.dsldebug.ToJson; */ public final class LalPayloadDebugDump { + private static final String ANY_TYPE_NAME = Any.getDescriptor().getFullName(); + + private static final String VALUE_TYPE_NAME = Value.getDescriptor().getFullName(); + + /** + * Well-known type descriptors so {@link JsonFormat} can resolve and print + * {@code google.protobuf.Any}-typed fields (e.g. Envoy ALS + * {@code filter_state_objects}, which is {@code map<string, Any>}). Without + * a registry a single unresolvable {@code Any} makes the whole message + * un-printable. Anything outside this set is handled by + * {@link #sanitizeUnresolvableAny(Message)} below. + */ + private static final JsonFormat.TypeRegistry TYPE_REGISTRY = + JsonFormat.TypeRegistry.newBuilder() + .add(BytesValue.getDescriptor()) + .add(StringValue.getDescriptor()) + .add(BoolValue.getDescriptor()) + .add(Int32Value.getDescriptor()) + .add(Int64Value.getDescriptor()) + .add(UInt32Value.getDescriptor()) + .add(UInt64Value.getDescriptor()) + .add(FloatValue.getDescriptor()) + .add(DoubleValue.getDescriptor()) + .add(Struct.getDescriptor()) + .add(Value.getDescriptor()) + .add(ListValue.getDescriptor()) + .add(Timestamp.getDescriptor()) + .add(Duration.getDescriptor()) + .build(); + private static final JsonFormat.Printer JSON_PRINTER = - JsonFormat.printer().omittingInsignificantWhitespace(); + JsonFormat.printer() + .usingTypeRegistry(TYPE_REGISTRY) + .omittingInsignificantWhitespace(); private LalPayloadDebugDump() { } @@ -83,8 +134,40 @@ public final class LalPayloadDebugDump { return builder == null ? "null" : LogDataDebugDump.toJson(builder.build()); } - /** Generic protobuf renderer via {@link JsonFormat}. */ + /** + * Generic protobuf renderer via {@link JsonFormat}. A receiver plugin can + * override the rendering for its own input type via the + * {@link org.apache.skywalking.oap.log.analyzer.v2.spi.LalInputDebugRenderer} + * SPI (e.g. Envoy ALS decodes the Istio metadata-exchange peer instead of + * emitting opaque base64); when no renderer matches, the generic printer + * is used. + */ public static String toJson(final Message msg) { + if (msg == null) { + return "null"; + } + final String custom = LalInputDebugRenderers.render(msg); + if (custom != null) { + return custom; + } + return renderProto(msg); + } + + /** + * Robust generic protobuf render — the well-known {@link #TYPE_REGISTRY} + * plus the unresolvable-{@code Any} sanitizer — WITHOUT consulting the + * {@link org.apache.skywalking.oap.log.analyzer.v2.spi.LalInputDebugRenderer} + * SPI. A receiver renderer that decodes its own sub-fields (e.g. the Envoy + * ALS metadata-exchange peer) calls this to serialize the decoded message + * and inherit the same robustness: the decoded fields stay readable AND any + * unrelated unregistered {@code Any} elsewhere in the same message degrades + * to a placeholder instead of failing the whole render. Never re-enters the + * SPI, so a renderer calling it cannot recurse. + * + * @param msg the message to render. + * @return the proto3 JSON string; never {@code null}. + */ + public static String renderProto(final Message msg) { if (msg == null) { return "null"; } @@ -135,7 +218,8 @@ public final class LalPayloadDebugDump { private static String printProtoOrError(final MessageOrBuilder msg, final String typeLabel) { try { - return JSON_PRINTER.print(msg); + final Message message = toMessage(msg); + return JSON_PRINTER.print(message == null ? msg : sanitizeUnresolvableAny(message)); } catch (final Exception e) { final JsonObject obj = new JsonObject(); obj.addProperty("type", typeLabel); @@ -144,4 +228,159 @@ public final class LalPayloadDebugDump { return obj.toString(); } } + + private static Message toMessage(final MessageOrBuilder msg) { + if (msg instanceof Message) { + return (Message) msg; + } + if (msg instanceof Message.Builder) { + return ((Message.Builder) msg).build(); + } + return null; + } + + /** + * Recursively replace every {@code google.protobuf.Any} whose + * {@code type_url} the {@link #TYPE_REGISTRY} cannot resolve with a + * placeholder that preserves the type URL and the raw bytes. Because + * {@code Any}-valued fields (e.g. Envoy ALS {@code filter_state_objects}) + * are open by design — any filter or operator config can inject arbitrary + * messages — a single unresolvable {@code Any} must not blank the whole + * input. The placeholder is itself an {@code Any} wrapping a + * {@code Struct} so it stays canonical proto3 JSON and prints natively. + * Returns the original message unchanged when nothing needs replacing. + */ + private static Message sanitizeUnresolvableAny(final Message message) { + final Descriptor descriptor = message.getDescriptorForType(); + Message.Builder builder = null; + for (final FieldDescriptor field : descriptor.getFields()) { + if (field.getJavaType() != FieldDescriptor.JavaType.MESSAGE) { + continue; + } + if (field.isMapField()) { + final FieldDescriptor valueField = field.getMessageType().findFieldByName("value"); + if (valueField == null || valueField.getJavaType() != FieldDescriptor.JavaType.MESSAGE) { + continue; + } + final int count = message.getRepeatedFieldCount(field); + for (int i = 0; i < count; i++) { + final Message entry = (Message) message.getRepeatedField(field, i); + final Message value = (Message) entry.getField(valueField); + final Message fixed = fixMessage(value); + if (fixed != value) { + if (builder == null) { + builder = message.toBuilder(); + } + builder.setRepeatedField(field, i, + entry.toBuilder().setField(valueField, fixed).build()); + } + } + } else if (field.isRepeated()) { + final int count = message.getRepeatedFieldCount(field); + for (int i = 0; i < count; i++) { + final Message element = (Message) message.getRepeatedField(field, i); + final Message fixed = fixMessage(element); + if (fixed != element) { + if (builder == null) { + builder = message.toBuilder(); + } + builder.setRepeatedField(field, i, fixed); + } + } + } else if (message.hasField(field)) { + final Message child = (Message) message.getField(field); + final Message fixed = fixMessage(child); + if (fixed != child) { + if (builder == null) { + builder = message.toBuilder(); + } + builder.setField(field, fixed); + } + } + } + return builder == null ? message : builder.build(); + } + + private static Message fixMessage(final Message message) { + final String fullName = message.getDescriptorForType().getFullName(); + if (ANY_TYPE_NAME.equals(fullName)) { + return fixAny(message); + } + if (VALUE_TYPE_NAME.equals(fullName)) { + return fixValue(message); + } + return sanitizeUnresolvableAny(message); + } + + /** + * Neutralize an {@code Any} that {@link JsonFormat} cannot print: an + * unresolvable {@code type_url}, a no-slash {@code type_url} (the printer + * only accepts {@code .../type} form and throws otherwise — even for a + * well-known type), or a resolvable type whose packed bytes are corrupt. + * Each degrades to the {@code @unresolved} placeholder. A resolvable, well- + * formed payload is recursively sanitized and re-packed under its original + * {@code type_url} so a bad value nested inside it (e.g. a non-finite + * {@code Value}) can't blank the whole message either. + */ + private static Message fixAny(final Message message) { + final Any any; + try { + any = message instanceof Any ? (Any) message : Any.parseFrom(message.toByteString()); + } catch (final Exception e) { + return message; + } + final String typeUrl = any.getTypeUrl(); + if (typeUrl.indexOf('/') < 0) { + return Any.pack(unresolvedPlaceholder(any)); + } + final Descriptor descriptor = TYPE_REGISTRY.find(stripTypeUrlPrefix(typeUrl)); + if (descriptor == null) { + return Any.pack(unresolvedPlaceholder(any)); + } + try { + final Message packed = sanitizeUnresolvableAny(DynamicMessage.parseFrom(descriptor, any.getValue())); + return Any.newBuilder().setTypeUrl(typeUrl).setValue(packed.toByteString()).build(); + } catch (final Exception e) { + return Any.pack(unresolvedPlaceholder(any)); + } + } + + /** + * Replace a {@code Value} carrying a non-finite {@code number_value} + * (NaN / ±Infinity) — which {@link JsonFormat} refuses to encode and would + * otherwise blank the whole message — with the same number rendered as a + * string. Other {@code Value} kinds (struct / list) are recursed into so a + * non-finite double nested anywhere inside is scrubbed too. + */ + private static Message fixValue(final Message message) { + final Value value; + try { + value = message instanceof Value ? (Value) message : Value.parseFrom(message.toByteString()); + } catch (final Exception e) { + return message; + } + if (value.getKindCase() == Value.KindCase.NUMBER_VALUE) { + final double number = value.getNumberValue(); + if (Double.isNaN(number) || Double.isInfinite(number)) { + return Value.newBuilder().setStringValue(Double.toString(number)).build(); + } + return message; + } + return sanitizeUnresolvableAny(message); + } + + private static Struct unresolvedPlaceholder(final Any any) { + return Struct.newBuilder() + .putFields("@type", Value.newBuilder().setStringValue(any.getTypeUrl()).build()) + .putFields("@unresolved", Value.newBuilder().setBoolValue(true).build()) + .putFields("value", Value.newBuilder() + .setStringValue(Base64.getEncoder().encodeToString(any.getValue().toByteArray())) + .build()) + .build(); + } + + private static String stripTypeUrlPrefix(final String typeUrl) { + final int slash = typeUrl.lastIndexOf('/'); + return slash < 0 ? typeUrl : typeUrl.substring(slash + 1); + } } diff --git a/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/spi/LalInputDebugRenderer.java b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/spi/LalInputDebugRenderer.java new file mode 100644 index 0000000000..6e7b56973a --- /dev/null +++ b/oap-server/analyzer/log-analyzer/src/main/java/org/apache/skywalking/oap/log/analyzer/v2/spi/LalInputDebugRenderer.java @@ -0,0 +1,63 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.log.analyzer.v2.spi; + +import com.google.protobuf.Message; + +/** + * SPI for receiver plugins to render an operator-readable debug JSON for a + * typed LAL input that the generic + * {@link com.google.protobuf.util.JsonFormat} printer cannot render + * meaningfully. + * + * <p>The motivating case is Envoy ALS: its + * {@code common_properties.filter_state_objects} carry the Istio + * metadata-exchange peer as a FlatBuffer wrapped in a + * {@code google.protobuf.BytesValue} (legacy Wasm form) or a + * {@code google.protobuf.Struct} (modern native form). The generic printer + * can at best emit opaque base64 for the FlatBuffer; a renderer decodes it + * into the readable peer metadata, reusing the receiver's own decoders. + * + * <p>Why an SPI rather than a direct call: the dispatch happens in + * {@code log-analyzer}, which the receiver plugins depend on (never the + * reverse), so {@code log-analyzer} cannot import a receiver's decoder. The + * receiver registers an implementation here, discovered via + * {@link java.util.ServiceLoader}, mirroring the existing + * {@link LALSourceTypeProvider} seam. Implementations register in + * {@code META-INF/services/org.apache.skywalking.oap.log.analyzer.v2.spi.LalInputDebugRenderer}. + */ +public interface LalInputDebugRenderer { + + /** + * The proto input type this renderer handles, e.g. + * {@code HTTPAccessLogEntry}. Matched against the input's runtime class. + * + * @return the concrete {@link Message} subtype this renderer renders. + */ + Class<? extends Message> inputType(); + + /** + * Render an operator-readable JSON object string for {@code input}. + * Must not throw — return {@code null} on any failure so the framework + * falls back to its generic protobuf printer. + * + * @param input the LAL input message to render. + * @return a JSON object string, or {@code null} to fall back. + */ + String render(Message input); +} diff --git a/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/v2/dsldebug/LalPayloadDebugDumpTest.java b/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/v2/dsldebug/LalPayloadDebugDumpTest.java new file mode 100644 index 0000000000..25eca4fb09 --- /dev/null +++ b/oap-server/analyzer/log-analyzer/src/test/java/org/apache/skywalking/oap/log/analyzer/v2/dsldebug/LalPayloadDebugDumpTest.java @@ -0,0 +1,154 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.log.analyzer.v2.dsldebug; + +import com.google.protobuf.Any; +import com.google.protobuf.ByteString; +import com.google.protobuf.BytesValue; +import com.google.protobuf.Struct; +import com.google.protobuf.UInt32Value; +import com.google.protobuf.Value; +import io.envoyproxy.envoy.config.core.v3.Metadata; +import io.envoyproxy.envoy.data.accesslog.v3.AccessLogCommon; +import io.envoyproxy.envoy.data.accesslog.v3.HTTPAccessLogEntry; +import io.envoyproxy.envoy.data.accesslog.v3.HTTPResponseProperties; +import java.util.Base64; +import org.junit.jupiter.api.Test; + +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** + * Verifies the generic protobuf input renderer no longer blanks the whole + * message on an {@code Any}-typed field. The Envoy ALS shape + * ({@code filter_state_objects} is {@code map<string, google.protobuf.Any>}) + * is used because it is the reported failure. No Envoy receiver SPI is on this + * module's test classpath, so these exercise the generic path: + * {@code TypeRegistry} (well-known types) plus the unresolvable-{@code Any} + * sanitizer — not a {@code LalInputDebugRenderer} (tested in the envoy module). + */ +class LalPayloadDebugDumpTest { + + @Test + void wellKnownAnyRendersAsBase64InsteadOfFailing() { + final HTTPAccessLogEntry entry = HTTPAccessLogEntry.newBuilder() + .setResponse(HTTPResponseProperties.newBuilder().setResponseCode(UInt32Value.of(503))) + .setCommonProperties(AccessLogCommon.newBuilder() + .putFilterStateObjects( + "some.bytes_state", + Any.pack(BytesValue.newBuilder().setValue(ByteString.copyFromUtf8("hello")).build()))) + .build(); + + final String json = LalPayloadDebugDump.toJson(entry); + + // Regression: the bare printer used to throw on the BytesValue Any. + assertFalse(json.contains("jsonformat-failed"), json); + // BytesValue renders as base64 in proto3 JSON ("hello" -> "aGVsbG8="). + assertTrue(json.contains(Base64.getEncoder().encodeToString("hello".getBytes())), json); + // The rest of the entry still serialized. + assertTrue(json.contains("503"), json); + } + + @Test + void unresolvableInjectedAnyDegradesToPlaceholderAndKeepsRest() { + final Any injected = Any.newBuilder() + .setTypeUrl("type.googleapis.com/test.skywalking.CustomFilterState") + .setValue(ByteString.copyFromUtf8("opaque-bytes")) + .build(); + final HTTPAccessLogEntry entry = HTTPAccessLogEntry.newBuilder() + .setResponse(HTTPResponseProperties.newBuilder().setResponseCode(UInt32Value.of(200))) + .setCommonProperties(AccessLogCommon.newBuilder() + .putFilterStateObjects("test.skywalking.injected", injected)) + .build(); + + final String json = LalPayloadDebugDump.toJson(entry); + + // No whole-message blanking despite an unregistered injected type. + assertFalse(json.contains("jsonformat-failed"), json); + // The placeholder preserves the original type URL and flags it. + assertTrue(json.contains("@unresolved"), json); + assertTrue(json.contains("test.skywalking.CustomFilterState"), json); + assertTrue(json.contains(Base64.getEncoder().encodeToString("opaque-bytes".getBytes())), json); + // The rest of the entry still serialized. + assertTrue(json.contains("200"), json); + } + + @Test + void nonFiniteValueInStructDoesNotBlankEntry() { + // metadata.filter_metadata is map<string, Struct>; a NaN/Infinity in a + // Value makes JsonFormat throw, which used to blank the whole entry. + final Struct meta = Struct.newBuilder() + .putFields("ratio", Value.newBuilder().setNumberValue(Double.NaN).build()) + .putFields("ceiling", Value.newBuilder().setNumberValue(Double.POSITIVE_INFINITY).build()) + .build(); + final HTTPAccessLogEntry entry = HTTPAccessLogEntry.newBuilder() + .setResponse(HTTPResponseProperties.newBuilder().setResponseCode(UInt32Value.of(200))) + .setCommonProperties(AccessLogCommon.newBuilder() + .setMetadata(Metadata.newBuilder().putFilterMetadata("istio.authz", meta))) + .build(); + + final String json = LalPayloadDebugDump.toJson(entry); + + assertFalse(json.contains("jsonformat-failed"), json); + assertTrue(json.contains("NaN"), json); // non-finite scrubbed to a string + assertTrue(json.contains("Infinity"), json); + assertTrue(json.contains("200"), json); // rest of the entry survived + } + + @Test + void noSlashTypeUrlAnyDegradesToPlaceholder() { + // A no-slash type_url throws in JsonFormat.printAny even when the bare + // name resolves to a well-known type — must degrade, not blank. + final Any noSlash = Any.newBuilder() + .setTypeUrl("google.protobuf.BytesValue") + .setValue(ByteString.copyFromUtf8("x")) + .build(); + final HTTPAccessLogEntry entry = HTTPAccessLogEntry.newBuilder() + .setResponse(HTTPResponseProperties.newBuilder().setResponseCode(UInt32Value.of(200))) + .setCommonProperties(AccessLogCommon.newBuilder() + .putFilterStateObjects("no.slash", noSlash)) + .build(); + + final String json = LalPayloadDebugDump.toJson(entry); + + assertFalse(json.contains("jsonformat-failed"), json); + assertTrue(json.contains("@unresolved"), json); + assertTrue(json.contains("200"), json); + } + + @Test + void resolvableTypeUrlWithCorruptBytesDegradesToPlaceholder() { + // type_url resolves (Struct) but the packed bytes are garbage — the + // printer parses them and throws; must degrade, not blank. + final Any corrupt = Any.newBuilder() + .setTypeUrl("type.googleapis.com/google.protobuf.Struct") + .setValue(ByteString.copyFrom(new byte[] {(byte) 0xFF, (byte) 0xFF, (byte) 0xFF, (byte) 0xFF})) + .build(); + final HTTPAccessLogEntry entry = HTTPAccessLogEntry.newBuilder() + .setResponse(HTTPResponseProperties.newBuilder().setResponseCode(UInt32Value.of(200))) + .setCommonProperties(AccessLogCommon.newBuilder() + .putFilterStateObjects("corrupt", corrupt)) + .build(); + + final String json = LalPayloadDebugDump.toJson(entry); + + assertFalse(json.contains("jsonformat-failed"), json); + assertTrue(json.contains("@unresolved"), json); + assertTrue(json.contains("200"), json); + } +} diff --git a/oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/envoy/als/mx/EnvoyAlsHttpDebugRenderer.java b/oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/envoy/als/mx/EnvoyAlsHttpDebugRenderer.java new file mode 100644 index 0000000000..8a4295b0f2 --- /dev/null +++ b/oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/envoy/als/mx/EnvoyAlsHttpDebugRenderer.java @@ -0,0 +1,47 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.server.receiver.envoy.als.mx; + +import com.google.protobuf.Message; +import io.envoyproxy.envoy.data.accesslog.v3.HTTPAccessLogEntry; +import org.apache.skywalking.oap.log.analyzer.v2.spi.LalInputDebugRenderer; + +/** + * Bridges LAL live-debug input rendering for Envoy {@link HTTPAccessLogEntry} + * to {@link EnvoyAlsJsonUtils}, so the Istio metadata-exchange peer in + * {@code filter_state_objects} is shown decoded rather than as opaque base64. + * Registered via {@code META-INF/services}; discovered by the framework's + * {@link LalInputDebugRenderer} SPI. + */ +public class EnvoyAlsHttpDebugRenderer implements LalInputDebugRenderer { + + @Override + public Class<? extends Message> inputType() { + return HTTPAccessLogEntry.class; + } + + @Override + public String render(final Message input) { + try { + return EnvoyAlsJsonUtils.toJSON((HTTPAccessLogEntry) input); + } catch (final Exception e) { + return null; // fall back to the framework's generic protobuf printer + } + } +} diff --git a/oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/envoy/als/mx/EnvoyAlsJsonUtils.java b/oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/envoy/als/mx/EnvoyAlsJsonUtils.java new file mode 100644 index 0000000000..1a21c094da --- /dev/null +++ b/oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/envoy/als/mx/EnvoyAlsJsonUtils.java @@ -0,0 +1,157 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.server.receiver.envoy.als.mx; + +import Wasm.Common.FlatNode; +import com.google.protobuf.Any; +import com.google.protobuf.BytesValue; +import com.google.protobuf.Message; +import com.google.protobuf.Struct; +import io.envoyproxy.envoy.data.accesslog.v3.AccessLogCommon; +import io.envoyproxy.envoy.data.accesslog.v3.HTTPAccessLogEntry; +import io.envoyproxy.envoy.data.accesslog.v3.TCPAccessLogEntry; +import java.nio.ByteBuffer; +import java.util.Optional; +import java.util.Set; +import org.apache.skywalking.oap.log.analyzer.v2.dsldebug.LalPayloadDebugDump; + +/** + * Canonical "Envoy ALS entry to readable JSON" for SkyWalking, covering both + * the HTTP and TCP access-log shapes (both feed the LAL pipeline — see + * {@code LogsPersistence} / {@code TCPLogsPersistence}). The Istio + * metadata-exchange peer objects in {@code common_properties.filter_state_objects} + * are decoded into readable {@code Struct}s instead of opaque base64, then the + * entry is serialized. + * + * <p>Two encodings are handled, both reusing the same decoders the runtime + * analyzer uses ({@link ServiceMetaInfoAdapter} / {@link MetaExchangeALSHTTPAnalyzer}): + * <ul> + * <li>legacy Wasm ({@code wasm.upstream_peer} / {@code wasm.downstream_peer}): + * {@code Any{BytesValue}} wrapping a {@code FlatNode} FlatBuffer.</li> + * <li>modern native ({@code upstream_peer} / {@code downstream_peer}): + * {@code Any{Struct}}.</li> + * </ul> + * + * <p>A decoded peer is re-packed as {@code Any{Struct}} so the canonical + * proto3 JSON shape is preserved, then the whole entry is serialized by + * {@link LalPayloadDebugDump#renderProto(com.google.protobuf.Message)} — the + * framework's robust printer (well-known {@code TypeRegistry} plus an + * unresolvable-{@code Any} sanitizer). This matters because + * {@code filter_state_objects} is open by design: a non-peer entry can carry + * an arbitrary unregistered type, and the robust printer degrades just that + * one field to a placeholder while keeping the decoded peer readable — rather + * than failing the whole entry and losing the peer decode. + */ +public final class EnvoyAlsJsonUtils { + + private static final Set<String> PEER_KEYS = Set.of( + MetaExchangeALSHTTPAnalyzer.UPSTREAM_KEY, // "wasm.upstream_peer" (legacy FlatBuffer) + MetaExchangeALSHTTPAnalyzer.DOWNSTREAM_KEY, // "wasm.downstream_peer" + MetaExchangeALSHTTPAnalyzer.UPSTREAM_PEER, // "upstream_peer" (modern Struct) + MetaExchangeALSHTTPAnalyzer.DOWNSTREAM_PEER // "downstream_peer" + ); + + private EnvoyAlsJsonUtils() { + } + + /** + * Serialize an HTTP ALS entry to JSON with metadata-exchange peers decoded. + * Non-peer {@code filter_state_objects} of unregistered types do not fail + * the render — they degrade to a placeholder while the peer stays decoded. + * + * @param entry the HTTP access log entry to render. + * @return the entry as proto3 JSON, peers decoded where present. + */ + public static String toJSON(final HTTPAccessLogEntry entry) { + if (!entry.hasCommonProperties()) { + return LalPayloadDebugDump.renderProto(entry); + } + return LalPayloadDebugDump.renderProto( + entry.toBuilder().setCommonProperties(decodePeers(entry.getCommonProperties())).build()); + } + + /** + * Serialize a TCP ALS entry to JSON with metadata-exchange peers decoded. + * TCP shares {@code AccessLogCommon} with HTTP, so the same peer keys and + * encodings apply. + * + * @param entry the TCP access log entry to render. + * @return the entry as proto3 JSON, peers decoded where present. + */ + public static String toJSON(final TCPAccessLogEntry entry) { + if (!entry.hasCommonProperties()) { + return LalPayloadDebugDump.renderProto(entry); + } + return LalPayloadDebugDump.renderProto( + entry.toBuilder().setCommonProperties(decodePeers(entry.getCommonProperties())).build()); + } + + /** + * Dispatch overload for callers holding only a {@link Message} (e.g. the + * LAL output builder's {@code bindInput}). Routes ALS entries to the + * peer-decoding overloads above; any other message is rendered by the + * framework's robust printer so an unregistered {@code Any} still cannot + * throw. + * + * @param entry the message to render. + * @return proto3 JSON; peers decoded for ALS entries. + */ + public static String toJSON(final Message entry) { + if (entry instanceof HTTPAccessLogEntry) { + return toJSON((HTTPAccessLogEntry) entry); + } + if (entry instanceof TCPAccessLogEntry) { + return toJSON((TCPAccessLogEntry) entry); + } + return LalPayloadDebugDump.renderProto(entry); + } + + private static AccessLogCommon decodePeers(final AccessLogCommon common) { + if (common.getFilterStateObjectsCount() == 0) { + return common; + } + final AccessLogCommon.Builder builder = common.toBuilder(); + // iterate the original (immutable) map; mutate the builder + common.getFilterStateObjectsMap().forEach((key, value) -> { + if (PEER_KEYS.contains(key)) { + decodePeer(key, value).ifPresent(struct -> + builder.putFilterStateObjects(key, Any.pack(struct))); + } + }); + return builder.build(); + } + + private static Optional<Struct> decodePeer(final String key, final Any value) { + try { + if (key.equals(MetaExchangeALSHTTPAnalyzer.UPSTREAM_KEY) + || key.equals(MetaExchangeALSHTTPAnalyzer.DOWNSTREAM_KEY)) { + // legacy Wasm: Any{BytesValue} -> FlatBuffer -> Struct + final ByteBuffer buffer = ByteBuffer.wrap( + BytesValue.parseFrom(value.getValue()).getValue().toByteArray()); + return Optional.of( + ServiceMetaInfoAdapter.extractStructFromNodeFlatBuffer( + FlatNode.getRootAsFlatNode(buffer))); + } + // modern native: Any{Struct} + return Optional.of(value.unpack(Struct.class)); + } catch (final Exception e) { + return Optional.empty(); // leave the raw Any; the registry still renders it as base64 + } + } +} diff --git a/oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/envoy/als/mx/EnvoyAlsTcpDebugRenderer.java b/oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/envoy/als/mx/EnvoyAlsTcpDebugRenderer.java new file mode 100644 index 0000000000..d91d7acd35 --- /dev/null +++ b/oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/envoy/als/mx/EnvoyAlsTcpDebugRenderer.java @@ -0,0 +1,49 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.skywalking.oap.server.receiver.envoy.als.mx; + +import com.google.protobuf.Message; +import io.envoyproxy.envoy.data.accesslog.v3.TCPAccessLogEntry; +import org.apache.skywalking.oap.log.analyzer.v2.spi.LalInputDebugRenderer; + +/** + * Bridges LAL live-debug input rendering for Envoy {@link TCPAccessLogEntry} + * to {@link EnvoyAlsJsonUtils}. TCP access logs feed the LAL pipeline via + * {@code TCPLogsPersistence}, and share {@code AccessLogCommon} (hence the same + * metadata-exchange peer keys and encodings) with HTTP, so the peer in + * {@code filter_state_objects} is shown decoded rather than as opaque base64. + * Registered via {@code META-INF/services}; discovered by the framework's + * {@link LalInputDebugRenderer} SPI. + */ +public class EnvoyAlsTcpDebugRenderer implements LalInputDebugRenderer { + + @Override + public Class<? extends Message> inputType() { + return TCPAccessLogEntry.class; + } + + @Override + public String render(final Message input) { + try { + return EnvoyAlsJsonUtils.toJSON((TCPAccessLogEntry) input); + } catch (final Exception e) { + return null; // fall back to the framework's generic protobuf printer + } + } +} diff --git a/oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/envoy/persistence/EnvoyAccessLogBuilder.java b/oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/envoy/persistence/EnvoyAccessLogBuilder.java index 2b0efb7f5a..e1eee5ba8c 100644 --- a/oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/envoy/persistence/EnvoyAccessLogBuilder.java +++ b/oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/envoy/persistence/EnvoyAccessLogBuilder.java @@ -20,13 +20,12 @@ package org.apache.skywalking.oap.server.receiver.envoy.persistence; import com.google.gson.JsonObject; import com.google.protobuf.Message; -import lombok.SneakyThrows; import org.apache.skywalking.oap.server.core.query.type.ContentType; import org.apache.skywalking.oap.server.core.source.Log; import org.apache.skywalking.oap.server.core.source.LogBuilder; import org.apache.skywalking.oap.server.core.source.LogMetadata; import org.apache.skywalking.oap.server.library.module.ModuleManager; -import org.apache.skywalking.oap.server.library.util.ProtoBufJsonUtils; +import org.apache.skywalking.oap.server.receiver.envoy.als.mx.EnvoyAlsJsonUtils; /** * LAL output builder for envoy access logs (both HTTP and TCP). @@ -56,10 +55,15 @@ public class EnvoyAccessLogBuilder extends LogBuilder { } @Override - @SneakyThrows public void bindInput(final LogMetadata metadata, final Object input) { if (input != null) { - this.accessLogEntryJson = ProtoBufJsonUtils.toJSON((Message) input); + // Robust, peer-aware ALS serialization: decodes the Istio + // metadata-exchange peer in filter_state_objects and degrades any + // unregistered Any to a placeholder. Must not throw — bindInput + // runs eagerly in the generated execute() (before any debug + // capture), so a throw here would abort the whole rule and drop + // the log. + this.accessLogEntryJson = EnvoyAlsJsonUtils.toJSON((Message) input); } // Envoy ALS doesn't deliver LogData, so skip super.bindInput's LogData // cast branch and just run the metadata-derived field population. @@ -67,7 +71,6 @@ public class EnvoyAccessLogBuilder extends LogBuilder { } @Override - @SneakyThrows public void init(final LogMetadata metadata, final Object input, final ModuleManager moduleManager) { ensureInitialized(moduleManager); diff --git a/oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/src/main/resources/META-INF/services/org.apache.skywalking.oap.log.analyzer.v2.spi.LalInputDebugRenderer b/oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/src/main/resources/META-INF/services/org.apache.skywalking.oap.log.analyzer.v2.spi.LalInputDebugRenderer new file mode 100644 index 0000000000..0ddc3c96eb --- /dev/null +++ b/oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/src/main/resources/META-INF/services/org.apache.skywalking.oap.log.analyzer.v2.spi.LalInputDebugRenderer @@ -0,0 +1,20 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# + +org.apache.skywalking.oap.server.receiver.envoy.als.mx.EnvoyAlsHttpDebugRenderer +org.apache.skywalking.oap.server.receiver.envoy.als.mx.EnvoyAlsTcpDebugRenderer diff --git a/oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/src/test/java/org/apache/skywalking/oap/server/receiver/envoy/als/mx/EnvoyAlsDebugRendererTest.java b/oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/src/test/java/org/apache/skywalking/oap/server/receiver/envoy/als/mx/EnvoyAlsDebugRendererTest.java new file mode 100644 index 0000000000..489321f54e --- /dev/null +++ b/oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/src/test/java/org/apache/skywalking/oap/server/receiver/envoy/als/mx/EnvoyAlsDebugRendererTest.java @@ -0,0 +1,300 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.skywalking.oap.server.receiver.envoy.als.mx; + +import Wasm.Common.FlatNode; +import Wasm.Common.KeyVal; +import com.google.flatbuffers.FlatBufferBuilder; +import com.google.protobuf.Any; +import com.google.protobuf.ByteString; +import com.google.protobuf.BytesValue; +import com.google.protobuf.Struct; +import com.google.protobuf.UInt32Value; +import com.google.protobuf.Value; +import io.envoyproxy.envoy.config.core.v3.Metadata; +import io.envoyproxy.envoy.data.accesslog.v3.AccessLogCommon; +import io.envoyproxy.envoy.data.accesslog.v3.HTTPAccessLogEntry; +import io.envoyproxy.envoy.data.accesslog.v3.HTTPRequestProperties; +import io.envoyproxy.envoy.data.accesslog.v3.HTTPResponseProperties; +import io.envoyproxy.envoy.data.accesslog.v3.TCPAccessLogEntry; +import org.apache.skywalking.oap.log.analyzer.v2.dsldebug.LalPayloadDebugDump; +import org.junit.jupiter.api.Test; + +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; + +/** + * Verifies the Istio metadata-exchange peer injected into Envoy ALS + * {@code filter_state_objects} is decoded into readable metadata instead of + * opaque base64, for both encodings, and that the + * {@link EnvoyAlsHttpDebugRenderer} / {@link EnvoyAlsTcpDebugRenderer} SPI + * shims are wired into the framework dump path. + */ +class EnvoyAlsDebugRendererTest { + + private static HTTPAccessLogEntry entryWith(final String key, final Any peer) { + return HTTPAccessLogEntry.newBuilder() + .setResponse(HTTPResponseProperties.newBuilder().setResponseCode(UInt32Value.of(200))) + .setCommonProperties(AccessLogCommon.newBuilder().putFilterStateObjects(key, peer)) + .build(); + } + + private static TCPAccessLogEntry tcpEntryWith(final String key, final Any peer) { + return TCPAccessLogEntry.newBuilder() + .setCommonProperties(AccessLogCommon.newBuilder().putFilterStateObjects(key, peer)) + .build(); + } + + /** Legacy Wasm form: Any{BytesValue} wrapping a FlatNode FlatBuffer. */ + private static Any legacyWasmPeer(final String name, final String namespace, + final String labelKey, final String labelVal) { + final FlatBufferBuilder fbb = new FlatBufferBuilder(); + final int nameOff = fbb.createString(name); + final int nsOff = fbb.createString(namespace); + final int keyOff = fbb.createString(labelKey); + final int valOff = fbb.createString(labelVal); + final int kvOff = KeyVal.createKeyVal(fbb, keyOff, valOff); + final int labelsOff = FlatNode.createLabelsVector(fbb, new int[] {kvOff}); + final int node = FlatNode.createFlatNode(fbb, nameOff, nsOff, labelsOff, 0, 0, 0, 0, 0, 0, 0); + fbb.finish(node); + final BytesValue wrapped = BytesValue.newBuilder() + .setValue(ByteString.copyFrom(fbb.sizedByteArray())) + .build(); + return Any.pack(wrapped); + } + + /** Modern native form: Any{Struct}. */ + private static Any modernStructPeer(final String name, final String namespace) { + final Struct struct = Struct.newBuilder() + .putFields("NAME", Value.newBuilder().setStringValue(name).build()) + .putFields("NAMESPACE", Value.newBuilder().setStringValue(namespace).build()) + .build(); + return Any.pack(struct); + } + + @Test + void decodesLegacyWasmFlatBufferPeer() throws Exception { + final HTTPAccessLogEntry entry = entryWith( + MetaExchangeALSHTTPAnalyzer.DOWNSTREAM_KEY, + legacyWasmPeer("reviews-v3-pod", "bookinfo", "app", "reviews")); + + final String json = EnvoyAlsJsonUtils.toJSON(entry); + + assertFalse(json.contains("jsonformat-failed"), json); + assertTrue(json.contains("reviews-v3-pod"), json); + assertTrue(json.contains("bookinfo"), json); + assertTrue(json.contains("reviews"), json); + assertTrue(json.contains("200"), json); + } + + @Test + void decodesModernStructPeer() throws Exception { + final HTTPAccessLogEntry entry = entryWith( + MetaExchangeALSHTTPAnalyzer.DOWNSTREAM_PEER, + modernStructPeer("ratings-v1-pod", "bookinfo")); + + final String json = EnvoyAlsJsonUtils.toJSON(entry); + + assertFalse(json.contains("jsonformat-failed"), json); + assertTrue(json.contains("ratings-v1-pod"), json); + assertTrue(json.contains("bookinfo"), json); + } + + /** The renderer is discovered via ServiceLoader and used by the framework dump path. */ + @Test + void spiWiredIntoFrameworkDumpPath() { + final HTTPAccessLogEntry entry = entryWith( + MetaExchangeALSHTTPAnalyzer.DOWNSTREAM_KEY, + legacyWasmPeer("details-v1-pod", "bookinfo", "app", "details")); + + final String json = LalPayloadDebugDump.toJson(entry); + + assertFalse(json.contains("jsonformat-failed"), json); + assertTrue(json.contains("details-v1-pod"), json); + } + + @Test + void decodesBothUpstreamAndDownstreamLegacyPeers() { + final HTTPAccessLogEntry entry = HTTPAccessLogEntry.newBuilder() + .setCommonProperties(AccessLogCommon.newBuilder() + .putFilterStateObjects(MetaExchangeALSHTTPAnalyzer.UPSTREAM_KEY, + legacyWasmPeer("reviews-v3-pod", "bookinfo", "app", "reviews")) + .putFilterStateObjects(MetaExchangeALSHTTPAnalyzer.DOWNSTREAM_KEY, + legacyWasmPeer("productpage-v1-pod", "bookinfo", "app", "productpage"))) + .build(); + + final String json = EnvoyAlsJsonUtils.toJSON(entry); + + assertFalse(json.contains("jsonformat-failed"), json); + assertTrue(json.contains("reviews-v3-pod"), json); + assertTrue(json.contains("productpage-v1-pod"), json); + } + + @Test + void decodesBothUpstreamAndDownstreamModernPeers() { + final HTTPAccessLogEntry entry = HTTPAccessLogEntry.newBuilder() + .setCommonProperties(AccessLogCommon.newBuilder() + .putFilterStateObjects(MetaExchangeALSHTTPAnalyzer.UPSTREAM_PEER, + modernStructPeer("reviews-v3-pod", "bookinfo")) + .putFilterStateObjects(MetaExchangeALSHTTPAnalyzer.DOWNSTREAM_PEER, + modernStructPeer("productpage-v1-pod", "bookinfo"))) + .build(); + + final String json = EnvoyAlsJsonUtils.toJSON(entry); + + assertFalse(json.contains("jsonformat-failed"), json); + assertTrue(json.contains("reviews-v3-pod"), json); + assertTrue(json.contains("productpage-v1-pod"), json); + } + + /** + * Regression for the all-or-nothing bug: an unrelated unregistered + * {@code Any} in {@code filter_state_objects} must NOT make the decoded + * peer revert to opaque base64 — the peer stays decoded and the unknown + * type degrades to a placeholder, in the same output. + */ + @Test + void peerStaysDecodedAlongsideUnregisteredFilterState() { + final Any unknown = Any.newBuilder() + .setTypeUrl("type.googleapis.com/test.skywalking.CustomFilterState") + .setValue(ByteString.copyFromUtf8("opaque-bytes")) + .build(); + final HTTPAccessLogEntry entry = HTTPAccessLogEntry.newBuilder() + .setResponse(HTTPResponseProperties.newBuilder().setResponseCode(UInt32Value.of(200))) + .setCommonProperties(AccessLogCommon.newBuilder() + .putFilterStateObjects(MetaExchangeALSHTTPAnalyzer.DOWNSTREAM_KEY, + legacyWasmPeer("productpage-v1-pod", "bookinfo", "app", "productpage")) + .putFilterStateObjects("test.skywalking.injected", unknown)) + .build(); + + // Direct util. + final String json = EnvoyAlsJsonUtils.toJSON(entry); + assertFalse(json.contains("jsonformat-failed"), json); + assertTrue(json.contains("productpage-v1-pod"), json); // peer still decoded + assertTrue(json.contains("@unresolved"), json); // unknown degraded + assertTrue(json.contains("test.skywalking.CustomFilterState"), json); + assertTrue(json.contains("200"), json); // standard field intact + + // Same through the framework SPI dump path. + final String viaFramework = LalPayloadDebugDump.toJson(entry); + assertTrue(viaFramework.contains("productpage-v1-pod"), viaFramework); + assertTrue(viaFramework.contains("@unresolved"), viaFramework); + } + + /** A realistic full entry: standard scalar/wrapper fields, metadata Struct, and a decoded peer. */ + @Test + void rendersFullEntryWithPeerAndStandardFields() { + final HTTPAccessLogEntry entry = HTTPAccessLogEntry.newBuilder() + .setRequest(HTTPRequestProperties.newBuilder().setPath("/productpage").setAuthority("productpage:9080")) + .setResponse(HTTPResponseProperties.newBuilder().setResponseCode(UInt32Value.of(200))) + .setCommonProperties(AccessLogCommon.newBuilder() + .setMetadata(Metadata.newBuilder().putFilterMetadata("istio.authz", + Struct.newBuilder() + .putFields("policy", Value.newBuilder().setStringValue("allow").build()) + .build())) + .putFilterStateObjects(MetaExchangeALSHTTPAnalyzer.DOWNSTREAM_KEY, + legacyWasmPeer("productpage-v1-pod", "bookinfo", "version", "v1"))) + .build(); + + final String json = EnvoyAlsJsonUtils.toJSON(entry); + + assertFalse(json.contains("jsonformat-failed"), json); + assertTrue(json.contains("/productpage"), json); // scalar + assertTrue(json.contains("productpage:9080"), json); // scalar + assertTrue(json.contains("200"), json); // UInt32Value wrapper + assertTrue(json.contains("allow"), json); // metadata filter_metadata Struct + assertTrue(json.contains("productpage-v1-pod"), json); // decoded peer + assertTrue(json.contains("v1"), json); // decoded peer label + } + + // --- TCP access logs: TCPAccessLogEntry feeds LAL via TCPLogsPersistence, shares AccessLogCommon --- + + @Test + void decodesTcpLegacyWasmPeer() { + final TCPAccessLogEntry entry = tcpEntryWith( + MetaExchangeALSHTTPAnalyzer.DOWNSTREAM_KEY, + legacyWasmPeer("ratings-v1-pod", "bookinfo", "app", "ratings")); + + final String json = EnvoyAlsJsonUtils.toJSON(entry); + + assertFalse(json.contains("jsonformat-failed"), json); + assertTrue(json.contains("ratings-v1-pod"), json); + assertTrue(json.contains("bookinfo"), json); + } + + @Test + void decodesTcpModernStructPeer() { + final TCPAccessLogEntry entry = tcpEntryWith( + MetaExchangeALSHTTPAnalyzer.UPSTREAM_PEER, + modernStructPeer("mysql-pod", "bookinfo")); + + final String json = EnvoyAlsJsonUtils.toJSON(entry); + + assertFalse(json.contains("jsonformat-failed"), json); + assertTrue(json.contains("mysql-pod"), json); + } + + /** The TCP renderer is discovered via ServiceLoader and used by the framework dump path. */ + @Test + void tcpPeerWiredIntoFrameworkDumpPath() { + final TCPAccessLogEntry entry = tcpEntryWith( + MetaExchangeALSHTTPAnalyzer.DOWNSTREAM_KEY, + legacyWasmPeer("mongodb-pod", "bookinfo", "app", "mongodb")); + + final String json = LalPayloadDebugDump.toJson(entry); + + assertFalse(json.contains("jsonformat-failed"), json); + assertTrue(json.contains("mongodb-pod"), json); + } + + // --- decodePeer failure fallback: a malformed value under a real peer key must degrade, not blank --- + + @Test + void corruptLegacyWasmPeerDoesNotBlankEntry() { + // BytesValue parses, but the inner bytes are not a FlatNode FlatBuffer: + // decodePeer catches and keeps the raw Any; the entry must still render. + final Any corrupt = Any.pack(BytesValue.newBuilder() + .setValue(ByteString.copyFromUtf8("not-a-flatbuffer")).build()); + final HTTPAccessLogEntry entry = HTTPAccessLogEntry.newBuilder() + .setResponse(HTTPResponseProperties.newBuilder().setResponseCode(UInt32Value.of(200))) + .setCommonProperties(AccessLogCommon.newBuilder() + .putFilterStateObjects(MetaExchangeALSHTTPAnalyzer.DOWNSTREAM_KEY, corrupt)) + .build(); + + final String json = EnvoyAlsJsonUtils.toJSON(entry); + + assertFalse(json.contains("jsonformat-failed"), json); + assertTrue(json.contains("200"), json); // non-peer field survives regardless of peer outcome + } + + @Test + void modernPeerKeyWithNonStructValueDegradesGracefully() { + // A *_peer key whose value is not a Struct: value.unpack(Struct) throws, + // decodePeer catches, the raw (unregistered) Any degrades to @unresolved. + final Any notStruct = Any.newBuilder() + .setTypeUrl("type.googleapis.com/test.skywalking.NotAStruct") + .setValue(ByteString.copyFromUtf8("opaque")) + .build(); + final HTTPAccessLogEntry entry = entryWith(MetaExchangeALSHTTPAnalyzer.DOWNSTREAM_PEER, notStruct); + + final String json = EnvoyAlsJsonUtils.toJSON(entry); + + assertFalse(json.contains("jsonformat-failed"), json); + assertTrue(json.contains("@unresolved"), json); + } +} diff --git a/oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/src/test/java/org/apache/skywalking/oap/server/receiver/envoy/persistence/EnvoyAccessLogBuilderTest.java b/oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/src/test/java/org/apache/skywalking/oap/server/receiver/envoy/persistence/EnvoyAccessLogBuilderTest.java index 2c3b39536f..b7f71455f8 100644 --- a/oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/src/test/java/org/apache/skywalking/oap/server/receiver/envoy/persistence/EnvoyAccessLogBuilderTest.java +++ b/oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/src/test/java/org/apache/skywalking/oap/server/receiver/envoy/persistence/EnvoyAccessLogBuilderTest.java @@ -18,7 +18,11 @@ package org.apache.skywalking.oap.server.receiver.envoy.persistence; +import com.google.protobuf.Any; +import com.google.protobuf.ByteString; +import com.google.protobuf.Struct; import com.google.protobuf.UInt32Value; +import com.google.protobuf.Value; import io.envoyproxy.envoy.data.accesslog.v3.AccessLogCommon; import io.envoyproxy.envoy.data.accesslog.v3.HTTPAccessLogEntry; import io.envoyproxy.envoy.data.accesslog.v3.HTTPResponseProperties; @@ -38,13 +42,16 @@ import org.apache.skywalking.oap.server.core.config.group.EndpointNameGrouping; import org.apache.skywalking.oap.server.core.query.type.ContentType; import org.apache.skywalking.oap.server.core.source.Log; import org.apache.skywalking.oap.server.core.source.SourceReceiver; +import org.apache.skywalking.oap.server.receiver.envoy.als.mx.MetaExchangeALSHTTPAnalyzer; import org.apache.skywalking.oap.server.library.module.ModuleManager; import org.apache.skywalking.oap.server.library.module.ModuleProviderHolder; import org.apache.skywalking.oap.server.library.module.ModuleServiceHolder; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; +import static org.junit.jupiter.api.Assertions.assertDoesNotThrow; import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertNotNull; import static org.mockito.ArgumentMatchers.anyString; import static org.mockito.Mockito.mock; @@ -128,6 +135,69 @@ class EnvoyAccessLogBuilderTest { assertEquals("EnvoyAccessLog", new EnvoyAccessLogBuilder().name()); } + /** + * {@code bindInput} runs eagerly in the generated {@code execute()} (before + * any debug capture), so it must not throw on an unregistered {@code Any} + * in {@code filter_state_objects} — otherwise the whole rule aborts and the + * log is dropped. Regression for the all-or-nothing serialization failure. + */ + @Test + void unregisteredFilterStateAnyDoesNotAbortExecution() { + final EnvoyAccessLogBuilder builder = new EnvoyAccessLogBuilder(); + final Any unknown = Any.newBuilder() + .setTypeUrl("type.googleapis.com/test.skywalking.CustomFilterState") + .setValue(ByteString.copyFromUtf8("opaque-bytes")) + .build(); + final HTTPAccessLogEntry entry = HTTPAccessLogEntry.newBuilder() + .setResponse(HTTPResponseProperties.newBuilder().setResponseCode(UInt32Value.of(200))) + .setCommonProperties(AccessLogCommon.newBuilder() + .putFilterStateObjects("test.skywalking.injected", unknown)) + .build(); + builder.setService("mesh-svc"); + builder.setTimestamp(1L); + final LogMetadata metadata = LogMetadata.builder().service("mesh-svc").timestamp(1L).build(); + + // Before the fix this throws "Cannot find type for url: ...CustomFilterState". + final Log log = assertDoesNotThrow(() -> { + builder.init(metadata, entry, moduleManager); + return builder.toLog(); + }); + + final String c = log.getContent(); + assertFalse(c.contains("jsonformat-failed"), c); + assertTrue(c.contains("@unresolved"), c); // degraded, not fatal + assertTrue(c.contains("test.skywalking.CustomFilterState"), c); + assertTrue(c.contains("200"), c); // standard field intact + } + + /** + * End-to-end persistence path: a metadata-exchange peer in + * {@code filter_state_objects} is decoded into the readable peer metadata + * in the stored log {@code content}, not opaque base64. + */ + @Test + void persistedContentContainsDecodedPeer() { + final EnvoyAccessLogBuilder builder = new EnvoyAccessLogBuilder(); + final Any peer = Any.pack(Struct.newBuilder() + .putFields("NAME", Value.newBuilder().setStringValue("productpage-v1-pod").build()) + .putFields("NAMESPACE", Value.newBuilder().setStringValue("bookinfo").build()) + .build()); + final HTTPAccessLogEntry entry = HTTPAccessLogEntry.newBuilder() + .setCommonProperties(AccessLogCommon.newBuilder() + .putFilterStateObjects(MetaExchangeALSHTTPAnalyzer.DOWNSTREAM_PEER, peer)) + .build(); + builder.setService("mesh-svc"); + builder.setTimestamp(1L); + final LogMetadata metadata = LogMetadata.builder().service("mesh-svc").timestamp(1L).build(); + + builder.init(metadata, entry, moduleManager); + final Log log = builder.toLog(); + + assertNotNull(log.getContent()); + assertTrue(log.getContent().contains("productpage-v1-pod"), log.getContent()); + assertTrue(log.getContent().contains("bookinfo"), log.getContent()); + } + /** * Compiles and executes a LAL script that accesses both proto fields from * the HTTPAccessLogEntry (via {@code parsed.*}) and entity fields from
