This is an automated email from the ASF dual-hosted git repository.
wu-sheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/skywalking.git
The following commit(s) were added to refs/heads/master by this push:
new fe55b90bcf Extend `GET /inspect/entities` to inspect foreign metrics
persisted by any OAP (#13926)
fe55b90bcf is described below
commit fe55b90bcf12b7962017ea64086121b8a07c7368
Author: 吴晟 Wu Sheng <[email protected]>
AuthorDate: Wed Jun 24 09:19:23 2026 +0800
Extend `GET /inspect/entities` to inspect foreign metrics persisted by any
OAP (#13926)
### Extend `GET /inspect/entities` to inspect a metric persisted by any OAP
(foreign metric)
`/inspect/entities` previously enumerated entities only for metrics
**defined on the queried OAP**. This extends it so a metric persisted by *any*
OAP — one this node never loaded the OAL/MAL/runtime-rule for — can be
inspected when the caller supplies the metric's `valueColumn` + `valueType`.
- The storage backend resolves the physical index/table/group from its
**own running config** — no DB schema / table-metadata read:
- **Elasticsearch** — merged `metrics-all` index + `metric_table`
discriminator.
- **JDBC** — probes the node's function tables by the `table_name`
discriminator.
- **BanyanDB** — synthesizes a read-only measure schema.
- `entity_id` is decoded **structurally** (service / 2nd-level / relations)
with a generic `name` leaf; scope is no longer required for the foreign path.
- Locally-defined metrics keep the **exact** field names, scope, and
`mqeEntity` (behavior unchanged).
- Reads are optimistic — errors surface to the caller; a metric unknown
locally without `valueColumn`/`valueType` returns a clear `400` steering the
caller to the foreign path.
`swctl admin inspect entities` gains `--value-column` / `--value-type`
(apache/skywalking-cli#230); `SW_CTL_COMMIT` is bumped to the cli commit that
ships them, and a 2-OAP inspect e2e (aware + foreign) is wired across BanyanDB
/ ES / PostgreSQL.
---
.github/workflows/skywalking.yaml | 8 ++
docs/en/changes/changes.md | 1 +
docs/en/setup/backend/admin-api/inspect.md | 87 ++++++++++++--
.../admin/inspect/decoder/EntityDecoder.java | 97 ++++++++++++++++
.../admin/inspect/handler/InspectRestHandler.java | 126 ++++++++++++++++++++-
.../admin/inspect/decoder/EntityDecoderTest.java | 97 ++++++++++++++++
.../core/storage/query/IMetricsQueryDAO.java | 41 +++++--
.../storage/plugin/banyandb/MetadataRegistry.java | 52 +++++++++
.../banyandb/measure/BanyanDBMetricsQueryDAO.java | 20 +++-
.../plugin/elasticsearch/base/IndexController.java | 9 +-
.../elasticsearch/query/MetricsQueryEsDAO.java | 30 ++++-
.../server/storage/plugin/jdbc/TableMetaInfo.java | 10 ++
.../storage/plugin/jdbc/common/TableHelper.java | 47 ++++++++
.../jdbc/common/dao/JDBCMetricsQueryDAO.java | 24 +++-
test/e2e-v2/cases/inspect/README.md | 30 +++++
.../cases/inspect/banyandb/docker-compose.yml | 79 +++++++++++++
test/e2e-v2/cases/inspect/banyandb/e2e.yaml | 59 ++++++++++
.../cases/inspect/elasticsearch/docker-compose.yml | 92 +++++++++++++++
test/e2e-v2/cases/inspect/elasticsearch/e2e.yaml | 59 ++++++++++
test/e2e-v2/cases/inspect/expected/ok.txt | 1 +
test/e2e-v2/cases/inspect/inspect-foreign-flow.sh | 106 +++++++++++++++++
.../otel-rules/inspect-e2e.yaml} | 14 ++-
.../cases/inspect/postgresql/docker-compose.yml | 92 +++++++++++++++
test/e2e-v2/cases/inspect/postgresql/e2e.yaml | 59 ++++++++++
.../expected/inspect-entities-unknown-metric.yml | 7 +-
test/e2e-v2/cases/storage/storage-cases.yaml | 9 +-
test/e2e-v2/script/env | 2 +-
27 files changed, 1215 insertions(+), 43 deletions(-)
diff --git a/.github/workflows/skywalking.yaml
b/.github/workflows/skywalking.yaml
index 221de0f4ae..ae0e30d2b5 100644
--- a/.github/workflows/skywalking.yaml
+++ b/.github/workflows/skywalking.yaml
@@ -390,6 +390,14 @@ jobs:
config: test/e2e-v2/cases/storage/es/es-sharding/e2e.yaml
env: ES_VERSION=8.18.8
+ - name: Inspect API BanyanDB
+ config: test/e2e-v2/cases/inspect/banyandb/e2e.yaml
+ - name: Inspect API PostgreSQL
+ config: test/e2e-v2/cases/inspect/postgresql/e2e.yaml
+ - name: Inspect API Elasticsearch 8.18.8
+ config: test/e2e-v2/cases/inspect/elasticsearch/e2e.yaml
+ env: ES_VERSION=8.18.8
+
- name: Runtime Rule MAL Storage BanyanDB
config:
test/e2e-v2/cases/runtime-rule/mal-storage/banyandb/e2e.yaml
- name: Runtime Rule MAL Storage PostgreSQL
diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md
index b046523837..fde18400e2 100644
--- a/docs/en/changes/changes.md
+++ b/docs/en/changes/changes.md
@@ -2,6 +2,7 @@
#### Project
+* Extend the `GET /inspect/entities` admin API to inspect a metric persisted
by **any** OAP, even one this node does not define locally. When the metric is
unknown to the local registry, the caller supplies `valueColumn` + `valueType`
and the storage backend resolves the physical index/table/group from its own
running config (no DB schema/table-metadata read): ES uses the merged
`metrics-all` index + `metric_table` discriminator, JDBC probes the node's
function tables by the `table_name` [...]
* Remove the always-on alarm-to-event conversion (`EventHookCallback`). A
triggered alarm is no longer synthesized into the events pipeline as an
`Alarm`/`AlarmRecovery` event; events now originate only from real event
sources (agents, SkyWalking CLI, Kubernetes Event Exporter). Alarms remain
available through the alarm store (`getAlarm`/`queryAlarms`) and the configured
alarm hooks. This drops a documented "Known Event" and removes 1-2 synthetic
event records per alarm fire.
* **New `queryAlarms` GraphQL query — entity / layer / rule filters for
alarms.** Adds
a comprehensive alarm query API alongside the legacy `getAlarm`. The new
diff --git a/docs/en/setup/backend/admin-api/inspect.md
b/docs/en/setup/backend/admin-api/inspect.md
index 9163d57e47..c43dea12c9 100644
--- a/docs/en/setup/backend/admin-api/inspect.md
+++ b/docs/en/setup/backend/admin-api/inspect.md
@@ -6,9 +6,12 @@ let operators answer two questions without writing exploratory
MQE:
1. *Which metrics has OAP registered, and at what downsampling?*
2. *For metric `X` in time range `T`, which entities currently hold values?*
-The output of (2) carries a ready-to-paste `mqeEntity` payload, so the
-follow-up MQE call against the public GraphQL `execExpression` mutation is
-copy-paste from the inspect response.
+For a locally-defined metric, the output of (2) carries a ready-to-paste
+`mqeEntity` payload, so the follow-up MQE call against the public GraphQL
+`execExpression` mutation is copy-paste from the inspect response. A metric
+persisted by **another OAP** that this node does not define can also be
+inspected with caller-supplied metadata (see
+[Foreign metrics](#foreign-metrics-not-defined-on-this-oap)).
## Enabling
@@ -83,8 +86,12 @@ curl
'http://oap-admin:17128/inspect/metrics?regex=service_cpm'
### `GET /inspect/entities`
-For a metric + time range + step, returns the entities holding values, each
-decoded into a human-readable shape and an MQE-ready `mqeEntity` payload.
+For a metric + time range + step, returns the entities holding values. For a
+metric this OAP defines locally, each row is decoded into a human-readable
+shape and an MQE-ready `mqeEntity` payload. A metric persisted by **another
+OAP** that this node does not define can also be inspected by additionally
+supplying `valueColumn` + `valueType` — see
+[Foreign metrics](#foreign-metrics-not-defined-on-this-oap).
Restricted to `REGULAR_VALUE` and `LABELED_VALUE` metrics. The non-MQE
metric types (`HEATMAP` / `SAMPLED_RECORD`) and the out-of-scope scopes
@@ -94,11 +101,13 @@ Query parameters:
| Name | Required | Description |
|------|----------|-------------|
-| `metric` | yes | Metric name. Must resolve in `ValueColumnMetadata`. |
+| `metric` | yes | Metric name. If unknown to this OAP's local registry, also
supply `valueColumn` + `valueType` (see [Foreign
metrics](#foreign-metrics-not-defined-on-this-oap)). |
| `start` | yes | Time-range start. Same format as MQE `Duration.start`:
`yyyy-MM-dd` (DAY), `yyyy-MM-dd HH` (HOUR), `yyyy-MM-dd HHmm` (MINUTE),
`yyyy-MM-dd HHmmss` (SECOND). Note `HHmm` is no-separator — use `1230`, not
`12:30`. |
| `end` | yes | Time-range end. Format mirrors `start`. |
-| `step` | yes | One of `MINUTE` / `HOUR` / `DAY`. Must be one of the metric's
`downsamplings`. |
+| `step` | yes | One of `MINUTE` / `HOUR` / `DAY`. For a locally-defined
metric, must be one of the metric's `downsamplings`; for a foreign metric the
requested step is trusted as-is. |
| `limit` | no | Server-side cap. Default 300, hard-capped at 300. |
+| `valueColumn` | conditional | **Required when `metric` is not defined on
this OAP.** The metric's value column (post-override physical name, e.g.
`value`, `value_`, `double_value`, `datatable_value`, `dataset`). Ignored for a
locally-defined metric. |
+| `valueType` | conditional | **Required when `metric` is not defined on this
OAP.** One of `LONG` / `INT` / `DOUBLE` / `LABELED`. Ignored for a
locally-defined metric. |
The `limit` is applied as `LIMIT N` at the storage layer — it bounds the
total rows scanned (300 ≈ 10 buckets × 30 entities), not 300 distinct
@@ -159,6 +168,65 @@ curl
'http://oap-admin:17128/inspect/entities?metric=service_cpm&start=2026-05-1
}
```
+### Foreign metrics (not defined on this OAP)
+
+A metric persisted by **another OAP** — a different OAL/MAL/runtime-rule set —
+is absent from this node's local registry, so its value column, type, and scope
+cannot be recovered from the metric name alone (there is no OAL/MAL text here
to
+read). Supply `valueColumn` + `valueType` on the request and the backend
resolves
+the physical index/table/group from its own running configuration (the
+deterministic metric → storage mapping that merging has used for years), with
+**no storage schema / table-metadata read**:
+
+* **ES** — the merged `metrics-all` index, filtered by the `metric_table`
+ discriminator. Not supported under `logicSharding=true`, where the physical
+ index is derived from the metric's stream class (returns `500`).
+* **JDBC** — probes the node's aggregation-function metric tables
+ (`metrics_<fn>` / `meter_<fn>`) by the `table_name` discriminator.
+* **BanyanDB** — synthesizes a read-only measure schema from the deterministic
+ measure/group mapping.
+
+Because the scope is unknown, the response degrades gracefully:
+
+* `scope` is `null` (the structural kind is per-row in `decoded`).
+* `entity_id` is decoded **structurally**: a single entity yields `serviceName`
+ (plus a generic `name` leaf for a 2nd-level instance/endpoint — the two are
+ byte-identical and not distinguishable without the scope); a relation yields
a
+ `source` / `destination` pair.
+* **No `mqeEntity`** is produced — MQE needs the exact scope, and a foreign
+ metric is not MQE-queryable on this node anyway.
+
+Existence is decided by the data probe itself, so an **empty result means "no
+rows in range", not "metric absent"**. Nothing is validated against metadata up
+front: a wrong `valueColumn` / `valueType` surfaces as a storage error (`500`)
+or an empty result.
+
+Tip: query the writing OAP's own `/inspect/metrics?regex=<metric>` to read the
+exact `valueColumnName`, then pass that as `valueColumn`.
+
+Example — `meter_custom_x`, defined on another OAP, inspected here:
+
+```bash
+curl
'http://oap-admin:17128/inspect/entities?metric=meter_custom_x&valueColumn=value&valueType=LONG&start=2026-05-10%201230&end=2026-05-10%201240&step=MINUTE'
+```
+
+```json
+{
+ "metric": "meter_custom_x",
+ "scope": null,
+ "step": "MINUTE",
+ "start": "2026-05-10 1230",
+ "end": "2026-05-10 1240",
+ "rows": [
+ {
+ "entityId": "cGF5bWVudA==.1",
+ "decoded": { "serviceName": "payment", "isReal": true },
+ "layer": "GENERAL"
+ }
+ ]
+}
+```
+
## Discovering the OAP REST URL for the MQE follow-up
To keep the surface minimal, the inspect API does not introduce a separate
@@ -173,8 +241,9 @@ session start is enough.
| Status | Body | Cause |
|--------|------|-------|
-| 400 | `{"error":"unknown metric: foo"}` | Metric not in
`ValueColumnMetadata`. |
-| 400 | `{"error":"step DAY not supported by metric foo (MINUTE,HOUR)"}` |
Metric not materialised at the requested downsampling. |
+| 400 | `{"error":"metric unknown locally: foo — provide valueColumn and
valueType to inspect a metric persisted by another OAP"}` | Metric not defined
on this OAP, and the `valueColumn` / `valueType` pair was not supplied. See
[Foreign metrics](#foreign-metrics-not-defined-on-this-oap). |
+| 400 | `{"error":"valueType must be one of LONG / INT / DOUBLE / LABELED (got
X)"}` | Invalid `valueType` on the foreign-metric path. |
+| 400 | `{"error":"step DAY not supported by metric foo (MINUTE,HOUR)"}` |
Metric not materialised at the requested downsampling (locally-defined metric
only). |
| 400 | `{"error":"metric type HEATMAP is not MQE-queryable; /inspect/entities
only accepts REGULAR_VALUE and LABELED_VALUE"}` | Metric is `HEATMAP`
(`HISTOGRAM` `dataType`). |
| 400 | `{"error":"metric type SAMPLED_RECORD is out of scope for
/inspect/entities"}` | Metric is `SAMPLED_RECORD`. |
| 400 | `{"error":"process scope is out of scope"}` | Scope is `Process` /
`ProcessRelation`. |
diff --git
a/oap-server/server-admin/inspect/src/main/java/org/apache/skywalking/oap/server/admin/inspect/decoder/EntityDecoder.java
b/oap-server/server-admin/inspect/src/main/java/org/apache/skywalking/oap/server/admin/inspect/decoder/EntityDecoder.java
index db8ea609e9..01d0b1abf1 100644
---
a/oap-server/server-admin/inspect/src/main/java/org/apache/skywalking/oap/server/admin/inspect/decoder/EntityDecoder.java
+++
b/oap-server/server-admin/inspect/src/main/java/org/apache/skywalking/oap/server/admin/inspect/decoder/EntityDecoder.java
@@ -21,6 +21,7 @@ package
org.apache.skywalking.oap.server.admin.inspect.decoder;
import java.util.LinkedHashMap;
import java.util.Map;
import org.apache.skywalking.oap.server.admin.inspect.response.MqeEntity;
+import org.apache.skywalking.oap.server.core.Const;
import org.apache.skywalking.oap.server.core.analysis.IDManager;
import org.apache.skywalking.oap.server.core.query.enumeration.Scope;
@@ -92,6 +93,102 @@ public final class EntityDecoder {
}
}
+ /**
+ * Structural, scope-free decode for a metric this OAP does not define (no
{@link Scope}
+ * available). The stored {@code entity_id} self-encodes the names with
standard base64 plus
+ * the {@code .} / {@code _} / {@code -} delimiters (none of which appear
in base64 output), so
+ * the entity kind is recoverable from delimiter structure alone:
+ * <ul>
+ * <li>no {@code -}, no {@code _} → service</li>
+ * <li>no {@code -}, one {@code _} → 2nd-level entity (service instance
OR endpoint —
+ * byte-identical encoding, emitted as a generic {@code name})</li>
+ * <li>one {@code -} (2 parts) → service relation, or 2nd-level relation
when each side has a
+ * {@code _}</li>
+ * <li>three {@code -} (4 parts) → endpoint relation</li>
+ * </ul>
+ * The only thing not recoverable is the instance-vs-endpoint label, so
the leaf is reported as
+ * {@code name} and no {@link MqeEntity} is produced — MQE re-query needs
the exact scope, and a
+ * foreign metric is not MQE-queryable on this node anyway.
+ */
+ public static Decoded decodeUnknownScope(final String entityId) {
+ final String[] relationParts =
entityId.split(Const.RELATION_ID_PARSER_SPLIT);
+ switch (relationParts.length) {
+ case 1:
+ return entityId.contains(Const.ID_CONNECTOR)
+ ? decodeLevel2Generic(entityId)
+ : decodeServiceGeneric(entityId);
+ case 2:
+ return relationParts[0].contains(Const.ID_CONNECTOR)
+ ? decodeLevel2RelationGeneric(entityId)
+ : decodeServiceRelationGeneric(entityId);
+ case 4:
+ return decodeEndpointRelationGeneric(entityId);
+ default:
+ throw new IllegalArgumentException(
+ "cannot structurally decode entity_id without scope: " +
entityId);
+ }
+ }
+
+ private static Decoded decodeServiceGeneric(final String entityId) {
+ final IDManager.ServiceID.ServiceIDDefinition def =
IDManager.ServiceID.analysisId(entityId);
+ final Map<String, Object> decoded = new LinkedHashMap<>();
+ decoded.put("serviceName", def.getName());
+ decoded.put("isReal", def.isReal());
+ return new Decoded(decoded, null, entityId);
+ }
+
+ private static Decoded decodeLevel2Generic(final String entityId) {
+ final IDManager.ServiceInstanceID.InstanceIDDefinition def =
+ IDManager.ServiceInstanceID.analysisId(entityId);
+ final IDManager.ServiceID.ServiceIDDefinition svc =
+ IDManager.ServiceID.analysisId(def.getServiceId());
+ return new Decoded(toLevel2Map(svc, def.getName()), null,
def.getServiceId());
+ }
+
+ private static Decoded decodeServiceRelationGeneric(final String entityId)
{
+ final IDManager.ServiceID.ServiceRelationDefine rel =
+ IDManager.ServiceID.analysisRelationId(entityId);
+ final IDManager.ServiceID.ServiceIDDefinition src =
IDManager.ServiceID.analysisId(rel.getSourceId());
+ final IDManager.ServiceID.ServiceIDDefinition dst =
IDManager.ServiceID.analysisId(rel.getDestId());
+ final Map<String, Object> decoded = new LinkedHashMap<>();
+ decoded.put("source", toServiceMap(src));
+ decoded.put("destination", toServiceMap(dst));
+ return new Decoded(decoded, null, rel.getSourceId());
+ }
+
+ private static Decoded decodeLevel2RelationGeneric(final String entityId) {
+ final IDManager.ServiceInstanceID.ServiceInstanceRelationDefine rel =
+ IDManager.ServiceInstanceID.analysisRelationId(entityId);
+ final IDManager.ServiceInstanceID.InstanceIDDefinition srcInst =
+ IDManager.ServiceInstanceID.analysisId(rel.getSourceId());
+ final IDManager.ServiceInstanceID.InstanceIDDefinition dstInst =
+ IDManager.ServiceInstanceID.analysisId(rel.getDestId());
+ final Map<String, Object> decoded = new LinkedHashMap<>();
+ decoded.put("source",
toLevel2Map(IDManager.ServiceID.analysisId(srcInst.getServiceId()),
srcInst.getName()));
+ decoded.put("destination",
toLevel2Map(IDManager.ServiceID.analysisId(dstInst.getServiceId()),
dstInst.getName()));
+ return new Decoded(decoded, null, srcInst.getServiceId());
+ }
+
+ private static Decoded decodeEndpointRelationGeneric(final String
entityId) {
+ final IDManager.EndpointID.EndpointRelationDefine rel =
+ IDManager.EndpointID.analysisRelationId(entityId);
+ final IDManager.ServiceID.ServiceIDDefinition srcSvc =
+ IDManager.ServiceID.analysisId(rel.getSourceServiceId());
+ final IDManager.ServiceID.ServiceIDDefinition dstSvc =
+ IDManager.ServiceID.analysisId(rel.getDestServiceId());
+ final Map<String, Object> decoded = new LinkedHashMap<>();
+ decoded.put("source", toLevel2Map(srcSvc, rel.getSource()));
+ decoded.put("destination", toLevel2Map(dstSvc, rel.getDest()));
+ return new Decoded(decoded, null, rel.getSourceServiceId());
+ }
+
+ private static Map<String, Object> toLevel2Map(final
IDManager.ServiceID.ServiceIDDefinition svc,
+ final String leafName) {
+ final Map<String, Object> map = toServiceMap(svc);
+ map.put("name", leafName);
+ return map;
+ }
+
private static Decoded decodeService(final String entityId) {
final IDManager.ServiceID.ServiceIDDefinition def =
IDManager.ServiceID.analysisId(entityId);
final Map<String, Object> decoded = new LinkedHashMap<>();
diff --git
a/oap-server/server-admin/inspect/src/main/java/org/apache/skywalking/oap/server/admin/inspect/handler/InspectRestHandler.java
b/oap-server/server-admin/inspect/src/main/java/org/apache/skywalking/oap/server/admin/inspect/handler/InspectRestHandler.java
index 48f29c3737..1d358542ad 100644
---
a/oap-server/server-admin/inspect/src/main/java/org/apache/skywalking/oap/server/admin/inspect/handler/InspectRestHandler.java
+++
b/oap-server/server-admin/inspect/src/main/java/org/apache/skywalking/oap/server/admin/inspect/handler/InspectRestHandler.java
@@ -74,6 +74,9 @@ public class InspectRestHandler {
private static final int LIMIT_DEFAULT = 300;
private static final int LIMIT_MAX = 300;
+ /** Value types a caller may declare for a foreign (locally-undefined)
metric. */
+ private static final Set<String> ACCEPTED_FOREIGN_VALUE_TYPES =
+ Set.of("LONG", "INT", "DOUBLE", "LABELED");
private final ModuleManager moduleManager;
@@ -161,17 +164,48 @@ public class InspectRestHandler {
return HttpResponse.ofJson(MediaType.JSON_UTF_8, new
MetricsResponse(rows));
}
+ /**
+ * Enumerate the entities holding values for a metric in a time range.
+ *
+ * <p>For a metric defined on this OAP, only {@code metric} + time params
are needed; metadata
+ * is read from the local registry and the response carries exact field
names, scope, and a
+ * re-queryable {@code mqeEntity}.
+ *
+ * <p>For a metric persisted by ANOTHER OAP that this node does not define
(no local registry
+ * entry, no OAL/MAL text to recover it from), the caller MUST also supply
the metric's storage
+ * metadata, which cannot be inferred from the name:
+ * <ul>
+ * <li>{@code valueColumn} — the metric's value column, a property of
its aggregation FUNCTION:
+ * one of {@code value} (common scalar), {@code double_value},
{@code int_value},
+ * {@code percentage}, {@code datatable_value} (labeled), {@code
dataset} (histogram). On
+ * MySQL / PostgreSQL pass the reserved-word-overridden physical
name ({@code value_}).</li>
+ * <li>{@code valueType} — how to read/decode the value: {@code LONG} /
{@code INT} /
+ * {@code DOUBLE} (scalar) or {@code LABELED} (DataTable).
HISTOGRAM/heatmap and
+ * SAMPLED_RECORD are out of scope for this endpoint.</li>
+ * </ul>
+ */
@Get("/inspect/entities")
public HttpResponse listEntities(@Param("metric") final String metric,
@Param("start") final String start,
@Param("end") final String end,
@Param("step") final String stepStr,
- @Param("limit") final Optional<Integer>
limitOpt) {
+ @Param("limit") final Optional<Integer>
limitOpt,
+ @Param("valueColumn") final
Optional<String> valueColumnOpt,
+ @Param("valueType") final
Optional<String> valueTypeOpt) {
// Resolve metadata.
final Optional<ValueColumnMetadata.ValueColumn> vcOpt =
ValueColumnMetadata.INSTANCE.readValueColumnDefinition(metric);
if (vcOpt.isEmpty()) {
- return error(HttpStatus.BAD_REQUEST, "unknown metric: " + metric);
+ // Foreign metric: not defined on this OAP. There is no OAL/MAL
text or local model to
+ // recover its value column / type / scope from, so the caller
must supply them. Without
+ // both, fall back to the original "unknown metric" rejection.
+ if (valueColumnOpt.isEmpty() || valueTypeOpt.isEmpty()) {
+ return error(HttpStatus.BAD_REQUEST,
+ "metric unknown locally: " + metric + " — provide
valueColumn and valueType to "
+ + "inspect a metric persisted by another OAP");
+ }
+ return listForeignEntities(metric, valueColumnOpt.get(),
valueTypeOpt.get(),
+ start, end, stepStr, limitOpt);
}
final ValueColumnMetadata.ValueColumn vc = vcOpt.get();
@@ -247,7 +281,7 @@ public class InspectRestHandler {
final List<String> entityIds;
try {
entityIds = metricsQueryDAO()
- .listEntityIdsInRange(metric, vc.getValueCName(), duration,
limit);
+ .listEntityIdsInRange(metric, vc.getValueCName(), null,
duration, limit);
} catch (IOException e) {
log.warn("listEntityIdsInRange failed for metric={} step={}",
metric, step, e);
return error(HttpStatus.INTERNAL_SERVER_ERROR, e.getMessage());
@@ -278,6 +312,92 @@ public class InspectRestHandler {
return HttpResponse.ofJson(MediaType.JSON_UTF_8, body);
}
+ /**
+ * Foreign-metric path: the metric is not defined on this OAP, so the
caller supplied
+ * {@code valueColumn} + {@code valueType}. Nothing is resolved from the
local registry — the
+ * storage DAO derives the physical target from its own running config —
and each entity_id is
+ * decoded structurally (scope-free), emitting a generic {@code name} leaf
and no
+ * {@code mqeEntity}. Errors and empty results flow straight back to the
caller; an empty result
+ * means "no rows in range", not a reliable "metric absent".
+ */
+ private HttpResponse listForeignEntities(final String metric,
+ final String valueColumn,
+ final String valueType,
+ final String start,
+ final String end,
+ final String stepStr,
+ final Optional<Integer> limitOpt)
{
+ final String type = valueType.toUpperCase();
+ if (!ACCEPTED_FOREIGN_VALUE_TYPES.contains(type)) {
+ return error(HttpStatus.BAD_REQUEST,
+ "valueType must be one of LONG / INT / DOUBLE / LABELED (got "
+ valueType + ")");
+ }
+
+ final Step step;
+ try {
+ step = Step.valueOf(stepStr.toUpperCase());
+ } catch (Exception e) {
+ return error(HttpStatus.BAD_REQUEST,
+ "step must be one of MINUTE / HOUR / DAY (got " + stepStr +
")");
+ }
+ if (step == Step.SECOND) {
+ return error(HttpStatus.BAD_REQUEST,
+ "step must be one of MINUTE / HOUR / DAY (got SECOND)");
+ }
+
+ final int limit = limitOpt.orElse(LIMIT_DEFAULT);
+ if (limit < 1 || limit > LIMIT_MAX) {
+ return error(HttpStatus.BAD_REQUEST, "limit must be between 1 and
" + LIMIT_MAX);
+ }
+
+ final Duration duration = new Duration();
+ duration.setStart(start);
+ duration.setEnd(end);
+ duration.setStep(step);
+ try {
+ duration.getStartTimeBucket();
+ duration.getEndTimeBucket();
+ } catch (IllegalArgumentException | UnexpectedException e) {
+ return error(HttpStatus.BAD_REQUEST,
+ "start / end must follow the step's date format (DAY:
yyyy-MM-dd, HOUR: "
+ + "yyyy-MM-dd HH, MINUTE: yyyy-MM-dd HHmm): " +
e.getMessage());
+ }
+
+ final List<String> entityIds;
+ try {
+ entityIds = metricsQueryDAO().listEntityIdsInRange(metric,
valueColumn, type, duration, limit);
+ } catch (Exception e) {
+ // Optimistic read: surface the storage error directly. A wrong
valueColumn/valueType, an
+ // unsupported storage mode (e.g. ES logicSharding), or a missing
table lands here.
+ log.warn("foreign-metric listEntityIdsInRange failed for metric={}
step={}", metric, step, e);
+ return error(HttpStatus.INTERNAL_SERVER_ERROR, e.getMessage());
+ }
+
+ final List<EntityRow> rows = new ArrayList<>();
+ for (final String entityId : entityIds) {
+ final EntityDecoder.Decoded decoded;
+ try {
+ decoded = EntityDecoder.decodeUnknownScope(entityId);
+ } catch (Exception e) {
+ log.warn("Failed to structurally decode entity_id={}",
entityId, e);
+ continue;
+ }
+ final List<String> layers =
lookupLayers(decoded.serviceIdForLayer);
+ if (layers.isEmpty()) {
+ rows.add(new EntityRow(entityId, decoded.decodedFields, null,
decoded.mqeEntity));
+ } else {
+ for (final String layer : layers) {
+ rows.add(new EntityRow(entityId, decoded.decodedFields,
layer, decoded.mqeEntity));
+ }
+ }
+ }
+
+ // scope is null: a foreign metric's structural kind is per-row in
`decoded`, and a single
+ // metric's entities all share one structure anyway.
+ final EntitiesResponse body = new EntitiesResponse(metric, null,
step.name(), start, end, rows);
+ return HttpResponse.ofJson(MediaType.JSON_UTF_8, body);
+ }
+
/**
* Mirror of the {@code /inspect/entities} type acceptance set. Kept in
one place so
* the {@code mqeQueryable=true} filter on {@code /inspect/metrics} and
the actual
diff --git
a/oap-server/server-admin/inspect/src/test/java/org/apache/skywalking/oap/server/admin/inspect/decoder/EntityDecoderTest.java
b/oap-server/server-admin/inspect/src/test/java/org/apache/skywalking/oap/server/admin/inspect/decoder/EntityDecoderTest.java
index 890c8bafba..84486f1e17 100644
---
a/oap-server/server-admin/inspect/src/test/java/org/apache/skywalking/oap/server/admin/inspect/decoder/EntityDecoderTest.java
+++
b/oap-server/server-admin/inspect/src/test/java/org/apache/skywalking/oap/server/admin/inspect/decoder/EntityDecoderTest.java
@@ -186,4 +186,101 @@ class EntityDecoderTest {
assertNull(d.mqeEntity.getEndpointName());
assertTrue(d.mqeEntity.getNormal());
}
+
+ // ==================== scope-free (foreign metric) decode
====================
+
+ @Test
+ void unknownScopeService() {
+ final String id = IDManager.ServiceID.buildId("payment", true);
+ final EntityDecoder.Decoded d = EntityDecoder.decodeUnknownScope(id);
+ assertEquals("payment", d.decodedFields.get("serviceName"));
+ assertEquals(Boolean.TRUE, d.decodedFields.get("isReal"));
+ assertNull(d.mqeEntity);
+ assertEquals(id, d.serviceIdForLayer);
+ }
+
+ @Test
+ void unknownScopeServiceConjectured() {
+ final String id = IDManager.ServiceID.buildId("mysql", false);
+ final EntityDecoder.Decoded d = EntityDecoder.decodeUnknownScope(id);
+ assertEquals("mysql", d.decodedFields.get("serviceName"));
+ assertEquals(Boolean.FALSE, d.decodedFields.get("isReal"));
+ }
+
+ @Test
+ void unknownScopeLevel2InstanceAndEndpointDecodeIdentically() {
+ // Instance and endpoint encode byte-identically (serviceId + "_" +
base64(name)), so the
+ // scope-free decode yields the same shape with a generic "name" leaf
for both.
+ final String svcId = IDManager.ServiceID.buildId("payment", true);
+ final String instId = IDManager.ServiceInstanceID.buildId(svcId,
"pod-01");
+ final String epId = IDManager.EndpointID.buildId(svcId,
"POST:/charge");
+
+ final EntityDecoder.Decoded inst =
EntityDecoder.decodeUnknownScope(instId);
+ assertEquals("payment", inst.decodedFields.get("serviceName"));
+ assertEquals("pod-01", inst.decodedFields.get("name"));
+ assertNull(inst.decodedFields.get("serviceInstanceName"));
+ assertNull(inst.mqeEntity);
+ assertEquals(svcId, inst.serviceIdForLayer);
+
+ final EntityDecoder.Decoded ep =
EntityDecoder.decodeUnknownScope(epId);
+ assertEquals("payment", ep.decodedFields.get("serviceName"));
+ assertEquals("POST:/charge", ep.decodedFields.get("name"));
+ assertNull(ep.decodedFields.get("endpointName"));
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ void unknownScopeServiceRelation() {
+ final String src = IDManager.ServiceID.buildId("checkout", true);
+ final String dst = IDManager.ServiceID.buildId("payment", true);
+ final String id = IDManager.ServiceID.buildRelationId(
+ new IDManager.ServiceID.ServiceRelationDefine(src, dst));
+
+ final EntityDecoder.Decoded d = EntityDecoder.decodeUnknownScope(id);
+ final Map<String, Object> source = (Map<String, Object>)
d.decodedFields.get("source");
+ final Map<String, Object> dest = (Map<String, Object>)
d.decodedFields.get("destination");
+ assertEquals("checkout", source.get("serviceName"));
+ assertEquals("payment", dest.get("serviceName"));
+ assertNull(source.get("name"));
+ assertNull(d.mqeEntity);
+ assertEquals(src, d.serviceIdForLayer);
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ void unknownScopeLevel2Relation() {
+ final String srcSvc = IDManager.ServiceID.buildId("consumer", true);
+ final String dstSvc = IDManager.ServiceID.buildId("provider", true);
+ final String srcInst = IDManager.ServiceInstanceID.buildId(srcSvc,
"pod-a");
+ final String dstInst = IDManager.ServiceInstanceID.buildId(dstSvc,
"pod-b");
+ final String id = IDManager.ServiceInstanceID.buildRelationId(
+ new
IDManager.ServiceInstanceID.ServiceInstanceRelationDefine(srcInst, dstInst));
+
+ final EntityDecoder.Decoded d = EntityDecoder.decodeUnknownScope(id);
+ final Map<String, Object> source = (Map<String, Object>)
d.decodedFields.get("source");
+ final Map<String, Object> dest = (Map<String, Object>)
d.decodedFields.get("destination");
+ assertEquals("consumer", source.get("serviceName"));
+ assertEquals("pod-a", source.get("name"));
+ assertEquals("provider", dest.get("serviceName"));
+ assertEquals("pod-b", dest.get("name"));
+ assertEquals(srcSvc, d.serviceIdForLayer);
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ void unknownScopeEndpointRelation() {
+ final String srcSvc = IDManager.ServiceID.buildId("consumer", true);
+ final String dstSvc = IDManager.ServiceID.buildId("provider", true);
+ final String id = IDManager.EndpointID.buildRelationId(
+ new IDManager.EndpointID.EndpointRelationDefine(srcSvc, "/order",
dstSvc, "/charge"));
+
+ final EntityDecoder.Decoded d = EntityDecoder.decodeUnknownScope(id);
+ final Map<String, Object> source = (Map<String, Object>)
d.decodedFields.get("source");
+ final Map<String, Object> dest = (Map<String, Object>)
d.decodedFields.get("destination");
+ assertEquals("consumer", source.get("serviceName"));
+ assertEquals("/order", source.get("name"));
+ assertEquals("provider", dest.get("serviceName"));
+ assertEquals("/charge", dest.get("name"));
+ assertEquals(srcSvc, d.serviceIdForLayer);
+ }
}
diff --git
a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/query/IMetricsQueryDAO.java
b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/query/IMetricsQueryDAO.java
index 6b5b49426a..3b0061e45b 100644
---
a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/query/IMetricsQueryDAO.java
+++
b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/storage/query/IMetricsQueryDAO.java
@@ -118,26 +118,51 @@ public interface IMetricsQueryDAO extends DAO {
/**
* List distinct {@code entity_id}s that have at least one row for the
given metric in the
* given time range, capped at {@code limit}. Used by the {@code
/inspect/entities}
- * admin-server endpoint to enumerate the entities currently emitting
values for a metric —
- * feeds the decoded {@code mqeEntity} payload the inspect API hands back
to operators so
- * they can follow up with a public-GraphQL MQE query.
+ * admin-server endpoint to enumerate the entities currently emitting
values for a metric.
*
* <p>Order: most recent timestamp first within the range so callers see
live entities ahead
* of stale ones. Backends dedup by {@code entity_id} before returning.
The {@code limit}
* argument is a server-side cap on the rows scanned, not a guarantee on
distinct entities
* (300 rows ≈ 10 buckets × 30 entities).
*
+ * <p>Handles two cases through one path, mirroring the single {@code
/inspect/entities}
+ * endpoint:
+ * <ul>
+ * <li><b>Locally-defined metric</b> — the model / {@code
ValueColumnMetadata} entry exists,
+ * so the backend resolves the physical index/table/group from its
registry as before and
+ * the {@code valueType} hint is unused.</li>
+ * <li><b>Foreign metric</b> — persisted by another OAP whose
OAL/MAL/runtime-rule set this
+ * node never loaded, so there is no local model. The backend
resolves the physical target
+ * from its OWN running configuration (the deterministic metric →
storage mapping that
+ * merging has used for years) WITHOUT reading any storage
schema/table metadata, using
+ * the caller-supplied {@code valueColumnName} + {@code valueType}.
Existence is decided by
+ * the data probe itself (the merged-table discriminator {@code
metric_table} /
+ * {@code table_name} on ES / JDBC, the synthesized measure on
BanyanDB), so an empty
+ * result means "no rows in range", never a reliable "metric
absent".</li>
+ * </ul>
+ *
* <p>Abstract on purpose — any 3rd party storage backend that implements
- * {@code IMetricsQueryDAO} after 10.5.0 MUST provide this override. A
default
- * (empty list or thrown exception) would let a missing override slip
through
- * compilation and surface as a runtime "no entities" or 500 the first
time the
- * inspect API hit that backend; the breaking-at-compile signal is the
safer
- * contract for the inspect storage path.
+ * {@code IMetricsQueryDAO} MUST provide this override. A default (empty
list or thrown
+ * exception) would let a missing override slip through compilation and
surface as a runtime
+ * "no entities" or 500 the first time the inspect API hit that backend;
the breaking-at-compile
+ * signal is the safer contract for the inspect storage path.
*
+ * @param metricName metric (model) name; also the merged-table
discriminator value.
+ * @param valueColumnName the metric's value column (post-override
physical name). Required for
+ * the foreign-metric path (BanyanDB
projects/defines the field with it);
+ * ES / JDBC entity enumeration is
value-column-agnostic.
+ * @param valueType value data type for a foreign metric — one of
{@code LONG} /
+ * {@code INT} / {@code DOUBLE} / {@code LABELED};
drives BanyanDB
+ * field-type synthesis. {@code null} for a
locally-defined metric (the
+ * backend reads the type from its local model).
+ * @param duration query time range + step.
+ * @param limit server-side row cap.
+ * @return distinct entity ids holding values for the metric in range,
most-recent first.
* @since 10.5.0
*/
List<String> listEntityIdsInRange(String metricName,
String valueColumnName,
+ String valueType,
Duration duration,
int limit) throws IOException;
diff --git
a/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/MetadataRegistry.java
b/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/MetadataRegistry.java
index a8591e5ee9..415e183a88 100644
---
a/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/MetadataRegistry.java
+++
b/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/MetadataRegistry.java
@@ -425,6 +425,58 @@ public enum MetadataRegistry {
return this.registry.get(SchemaMetadata.formatName(modelName,
downSampling));
}
+ /**
+ * Synthesize a read-only measure {@link Schema} for a metric this OAP
does not define locally
+ * (persisted by another OAP). No storage schema read is performed: the
measure name / group /
+ * entity_id tag are derived from the deterministic metric → measure
mapping, and the value field
+ * from the caller-supplied {@code valueColumn} / {@code valueType}. The
node-global namespace is
+ * borrowed from any registered measure (the node always has its own
metrics). BanyanDB validates
+ * the projection server-side, so a wrong value column surfaces as a query
error.
+ *
+ * @return the synthesized schema, or {@code null} if this node has no
registered measure to
+ * borrow the namespace from (foreign-metric inspect is then
unavailable here).
+ */
+ public Schema synthesizeForeignMetricSchema(final String metricName,
+ final Step step,
+ final String valueColumn,
+ final String valueType) {
+ final DownSampling downSampling = deriveFromStep(step);
+ final String rawGroup;
+ switch (downSampling) {
+ case Minute:
+ rawGroup = BanyanDB.MeasureGroup.METRICS_MINUTE.getName();
+ break;
+ case Hour:
+ rawGroup = BanyanDB.MeasureGroup.METRICS_HOUR.getName();
+ break;
+ case Day:
+ rawGroup = BanyanDB.MeasureGroup.METRICS_DAY.getName();
+ break;
+ default:
+ throw new IllegalArgumentException(
+ "foreign-metric inspect supports step MINUTE / HOUR / DAY
only, got " + step);
+ }
+ // namespace is node-global; borrow it from any registered measure.
convertGroupName treats a
+ // null/empty namespace as "no prefix".
+ String namespace = null;
+ for (final Schema registered : this.registry.values()) {
+ if (registered.getMetadata().getKind() == Kind.MEASURE) {
+ namespace = registered.getMetadata().getNamespace();
+ break;
+ }
+ }
+ final SchemaMetadata metadata = new SchemaMetadata(
+ namespace, rawGroup, metricName, Kind.MEASURE, downSampling, null);
+ final Class<?> fieldClass = "DOUBLE".equalsIgnoreCase(valueType) ?
double.class : long.class;
+ return Schema.builder()
+ .metadata(metadata)
+ .tag(Metrics.ENTITY_ID, metadata.indexFamily())
+ .field(valueColumn)
+ .spec(Metrics.ENTITY_ID, new ColumnSpec(ColumnType.TAG,
String.class))
+ .spec(valueColumn, new ColumnSpec(ColumnType.FIELD,
fieldClass))
+ .build();
+ }
+
public Schema findRecordMetadata(final String modelName) {
return this.registry.get(modelName);
}
diff --git
a/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/measure/BanyanDBMetricsQueryDAO.java
b/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/measure/BanyanDBMetricsQueryDAO.java
index 1c139df0be..4139141774 100644
---
a/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/measure/BanyanDBMetricsQueryDAO.java
+++
b/oap-server/server-storage-plugin/storage-banyandb-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/banyandb/measure/BanyanDBMetricsQueryDAO.java
@@ -169,12 +169,26 @@ public class BanyanDBMetricsQueryDAO extends
AbstractBanyanDBDAO implements IMet
@Override
public List<String> listEntityIdsInRange(final String metricName,
final String valueColumnName,
+ final String valueType,
final Duration duration,
final int limit) throws
IOException {
final boolean isColdStage = duration != null && duration.isColdStage();
- final MetadataRegistry.Schema schema =
MetadataRegistry.INSTANCE.findMetricMetadata(metricName, duration.getStep());
- if (schema == null) {
- throw new IOException("schema is not registered");
+ final MetadataRegistry.Schema schema;
+ if (valueType != null) {
+ // Foreign metric: no local schema. Synthesize a read-only measure
schema from the
+ // deterministic name → measure/group mapping plus the caller's
value column / type.
+ schema = MetadataRegistry.INSTANCE.synthesizeForeignMetricSchema(
+ metricName, duration.getStep(), valueColumnName, valueType);
+ if (schema == null) {
+ throw new IOException(
+ "cannot inspect foreign metric " + metricName + " on
BanyanDB: this node has no "
+ + "registered measure to resolve the namespace/group
from");
+ }
+ } else {
+ schema = MetadataRegistry.INSTANCE.findMetricMetadata(metricName,
duration.getStep());
+ if (schema == null) {
+ throw new IOException("schema is not registered");
+ }
}
final MeasureQueryResponse resp = query(
isColdStage, schema,
diff --git
a/oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/base/IndexController.java
b/oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/base/IndexController.java
index 29f4211d9d..d41ebf7554 100644
---
a/oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/base/IndexController.java
+++
b/oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/base/IndexController.java
@@ -53,12 +53,19 @@ public enum IndexController {
@Getter
private boolean enableCustomRouting = false;
+ /**
+ * The single physical index every metric model merges into when {@link
#logicSharding} is off
+ * (the default for years). Used at install/write time and by the inspect
read path for a
+ * foreign metric, whose physical index cannot be resolved through the
(absent) local model.
+ */
+ public static final String METRICS_LOGIC_TABLE_NAME = "metrics-all";
+
public String getTableName(Model model) {
if (!model.isTimeSeries()) {
return "management";
}
if (!logicSharding) {
- return model.isMetric() ? "metrics-all" :
+ return model.isMetric() ? METRICS_LOGIC_TABLE_NAME :
(model.isRecord() && !model.isSuperDataset() ? "records-all" :
model.getName());
}
String aggFuncName =
FunctionCategory.uniqueFunctionName(model.getStreamClass());
diff --git
a/oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/MetricsQueryEsDAO.java
b/oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/MetricsQueryEsDAO.java
index 82123c83c1..b4c44e0be2 100644
---
a/oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/MetricsQueryEsDAO.java
+++
b/oap-server/server-storage-plugin/storage-elasticsearch-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/elasticsearch/query/MetricsQueryEsDAO.java
@@ -18,6 +18,7 @@
package org.apache.skywalking.oap.server.storage.plugin.elasticsearch.query;
+import java.io.IOException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.LinkedHashSet;
@@ -197,8 +198,31 @@ public class MetricsQueryEsDAO extends EsDAO implements
IMetricsQueryDAO {
@Override
public List<String> listEntityIdsInRange(final String metricName,
final String valueColumnName,
+ final String valueType,
final Duration duration,
- final int limit) {
+ final int limit) throws
IOException {
+ // valueType != null signals a foreign metric (not defined on this
OAP). The value column
+ // is unused by ES entity enumeration; only the physical index +
discriminator differ.
+ final boolean foreign = valueType != null;
+ final String physicalIndex;
+ final boolean filterByDiscriminator;
+ if (foreign) {
+ // Resolve from running config, not the local registry. In the
default merged mode
+ // every metric lives in the single METRICS_LOGIC_TABLE_NAME index
and carries the
+ // metric_table discriminator. Under logicSharding the physical
index is derived from
+ // the (absent) stream class, so it cannot be resolved without the
local model.
+ if (IndexController.INSTANCE.isLogicSharding()) {
+ throw new IOException(
+ "inspecting a foreign metric is unsupported under ES
logicSharding=true: the "
+ + "physical index is derived from the metric's stream
class, which this OAP "
+ + "does not have for " + metricName);
+ }
+ physicalIndex = IndexController.METRICS_LOGIC_TABLE_NAME;
+ filterByDiscriminator = true;
+ } else {
+ physicalIndex =
IndexController.LogicIndicesRegister.getPhysicalTableName(metricName);
+ filterByDiscriminator =
IndexController.LogicIndicesRegister.isMergedTable(metricName);
+ }
final SearchBuilder search = Search.builder().size(limit);
// Most-recent-first ordering must be explicit — without sort the hit
set is
// score / index-internal ordered, so a hot entity that ingested late
can be dropped
@@ -207,7 +231,7 @@ public class MetricsQueryEsDAO extends EsDAO implements
IMetricsQueryDAO {
final BoolQueryBuilder query =
Query.bool().must(Query.range(Metrics.TIME_BUCKET)
.lte(duration.getEndTimeBucket())
.gte(duration.getStartTimeBucket()));
- if (IndexController.LogicIndicesRegister.isMergedTable(metricName)) {
+ if (filterByDiscriminator) {
query.must(Query.term(
IndexController.LogicIndicesRegister.METRIC_TABLE_NAME,
metricName
@@ -215,7 +239,7 @@ public class MetricsQueryEsDAO extends EsDAO implements
IMetricsQueryDAO {
}
search.query(query);
final SearchResponse response = getClient().search(new
TimeRangeIndexNameGenerator(
-
IndexController.LogicIndicesRegister.getPhysicalTableName(metricName),
+ physicalIndex,
duration.getStartTimeBucketInSec(),
duration.getEndTimeBucketInSec()), search.build());
// Top-N hits across the time range, dedup client-side on entity_id.
LinkedHashSet
diff --git
a/oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/TableMetaInfo.java
b/oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/TableMetaInfo.java
index 559d7cf391..339eb8ac4b 100644
---
a/oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/TableMetaInfo.java
+++
b/oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/TableMetaInfo.java
@@ -20,6 +20,8 @@ package org.apache.skywalking.oap.server.storage.plugin.jdbc;
import org.apache.skywalking.oap.server.core.storage.model.Model;
+import java.util.Collection;
+import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
@@ -39,4 +41,12 @@ public class TableMetaInfo {
public static Model get(String moduleName) {
return TABLES.get(moduleName);
}
+
+ /**
+ * All locally-installed models. The inspect foreign-metric probe uses
this to enumerate the
+ * node's metric function tables without a per-metric {@link Model} lookup.
+ */
+ public static Collection<Model> getModels() {
+ return Collections.unmodifiableCollection(TABLES.values());
+ }
}
diff --git
a/oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/common/TableHelper.java
b/oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/common/TableHelper.java
index 95575b6da7..7d02b4afce 100644
---
a/oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/common/TableHelper.java
+++
b/oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/common/TableHelper.java
@@ -174,6 +174,53 @@ public class TableHelper {
.collect(toList());
}
+ /**
+ * Distinct physical (raw) table names of every aggregation-FUNCTION
metric model installed on
+ * this node — the closed set of {@code metrics_<fn>} / {@code meter_<fn>}
tables that a foreign
+ * metric (defined by another OAP) must also live in. Used by the inspect
probe.
+ *
+ * <p>Filtered to function metrics, NOT all {@code isMetric()} models:
metadata "metrics" such as
+ * {@code ServiceTraffic} / {@code InstanceTraffic} / {@code
EndpointTraffic} are {@code Metrics}
+ * subclasses (so {@code isMetric()} is true) but carry no aggregation
function, no
+ * {@code entity_id}, and no {@code table_name} discriminator column.
Probing them with
+ * {@code select entity_id ... where table_name = ?} would hit "column not
found" and 500. Only
+ * function metrics are merged into the shared {@code metrics_<fn>} tables
and always carry both
+ * columns.
+ */
+ public static List<String> getMetricRawTables() {
+ return TableMetaInfo.getModels().stream()
+ .filter(TableHelper::isFunctionMetric)
+ .map(TableHelper::getTableName)
+ .distinct()
+ .collect(toList());
+ }
+
+ /**
+ * Day-partitioned tables for a RAW physical table name (a metric function
table) within a
+ * time-bucket range, filtered to those that actually exist. Unlike
+ * {@link #getTablesForRead(String, long, long)} this needs no local
{@link Model}, so it backs
+ * the foreign-metric inspect probe across the node's known function
tables.
+ */
+ public List<String> getExistingDayTables(String rawTableName, long
timeBucketStart, long timeBucketEnd) {
+ final var timestampStart = TimeBucket.getTimestamp(timeBucketStart);
+ final var timestampEnd = TimeBucket.getTimestamp(timeBucketEnd);
+ final var timeBuckets = LongStream.builder();
+ for (var timestamp = timestampStart; timestamp <= timestampEnd;
timestamp += TimeUnit.DAYS.toMillis(1)) {
+ timeBuckets.add(TimeBucket.getTimeBucket(timestamp,
DownSampling.Day));
+ }
+ return timeBuckets.build()
+ .distinct()
+ .mapToObj(timeBucket -> getTable(rawTableName, timeBucket))
+ .filter(table -> {
+ try {
+ return tableExistence.get(table);
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ })
+ .collect(toList());
+ }
+
public static String generateId(Model model, String originalID) {
if (model.isRecord() && !model.isSuperDataset()) {
return generateId(model.getName(), originalID);
diff --git
a/oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/common/dao/JDBCMetricsQueryDAO.java
b/oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/common/dao/JDBCMetricsQueryDAO.java
index 2ded45d408..5aa18fc9e2 100644
---
a/oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/common/dao/JDBCMetricsQueryDAO.java
+++
b/oap-server/server-storage-plugin/storage-jdbc-hikaricp-plugin/src/main/java/org/apache/skywalking/oap/server/storage/plugin/jdbc/common/dao/JDBCMetricsQueryDAO.java
@@ -203,13 +203,27 @@ public class JDBCMetricsQueryDAO extends JDBCSQLExecutor
implements IMetricsQuer
@SneakyThrows
public List<String> listEntityIdsInRange(final String metricName,
final String valueColumnName,
+ final String valueType,
final Duration duration,
final int limit) {
- final var tables = tableHelper.getTablesForRead(
- metricName,
- duration.getStartTimeBucket(),
- duration.getEndTimeBucket()
- );
+ // valueType != null signals a foreign metric (not defined on this
OAP). Its physical table
+ // is per-function, derivable only from the absent model, so probe the
node's known metric
+ // function tables; the table_name = ? discriminator below keeps only
this metric's rows. A
+ // locally-defined metric resolves straight to its own table set.
+ final List<String> tables;
+ if (valueType != null) {
+ tables = new ArrayList<>();
+ for (final var rawTable : TableHelper.getMetricRawTables()) {
+ tables.addAll(tableHelper.getExistingDayTables(
+ rawTable, duration.getStartTimeBucket(),
duration.getEndTimeBucket()));
+ }
+ } else {
+ tables = tableHelper.getTablesForRead(
+ metricName,
+ duration.getStartTimeBucket(),
+ duration.getEndTimeBucket()
+ );
+ }
// For each entity_id, track the latest time_bucket seen across every
day-partitioned
// table the range touches. Per-table query shape is GROUP BY
entity_id with MAX(time_bucket)
// and ORDER BY that max — portable across H2 / MySQL / PostgreSQL
(Postgres rejects
diff --git a/test/e2e-v2/cases/inspect/README.md
b/test/e2e-v2/cases/inspect/README.md
new file mode 100644
index 0000000000..32ff07a740
--- /dev/null
+++ b/test/e2e-v2/cases/inspect/README.md
@@ -0,0 +1,30 @@
+# Inspect API e2e — aware (existing) + foreign-metric (new) paths
+
+Two independent OAPs share one storage backend (no cluster):
+
+- **oap-a** loads `otel-rules/inspect-e2e.yaml`, turning the OTLP emitter's
+ `e2e_rr_pool_size` into the Service metric `meter_inspect_e2e_pool`.
+- **oap-b** loads no such rule, so that metric is **foreign** to it — absent
from its
+ local registry but present in the shared storage.
+
+`inspect-foreign-flow.sh` drives both paths and asserts inline:
+
+| Path | OAP | Assertion |
+|------|-----|-----------|
+| aware (existing) | oap-a | `/inspect/metrics` lists it; `/inspect/entities`
returns `inspect-e2e-svc` with an `mqeEntity` |
+| — | oap-b | `/inspect/metrics` excludes it |
+| aware, no metadata | oap-b | `/inspect/entities` → `400 metric unknown
locally …` |
+| **foreign (new)** | oap-b | `/inspect/entities --value-column --value-type`
returns the same entity, `scope:null`, no `mqeEntity` |
+
+Covered storages: `banyandb/`, `elasticsearch/`, `postgresql/`.
+
+## CI wiring (gated on skywalking-cli)
+
+The foreign assertion calls `swctl admin inspect entities --value-column /
--value-type`,
+flags added in [skywalking-cli
#230](https://github.com/apache/skywalking-cli/pull/230).
+The e2e builds swctl from `SW_CTL_COMMIT` (`test/e2e-v2/script/env`), which is
pinned to a
+cli commit that includes those flags, and the three storage variants are wired
into the
+`e2e` matrix in `.github/workflows/skywalking.yaml`.
+
+To validate locally, build swctl from that commit (or newer) and run any
variant's
+`e2e.yaml` with `skywalking-infra-e2e`; all three storages pass.
diff --git a/test/e2e-v2/cases/inspect/banyandb/docker-compose.yml
b/test/e2e-v2/cases/inspect/banyandb/docker-compose.yml
new file mode 100644
index 0000000000..10ce114659
--- /dev/null
+++ b/test/e2e-v2/cases/inspect/banyandb/docker-compose.yml
@@ -0,0 +1,79 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Inspect foreign-metric e2e — BanyanDB. Two independent OAPs share one
BanyanDB
+# (no cluster). oap-a loads the inspect-e2e otel-rule (→
meter_inspect_e2e_pool);
+# oap-b does not, so that metric is foreign to it.
+services:
+ oap-a:
+ extends:
+ file: ../../../script/docker-compose/base-compose.yml
+ service: oap
+ volumes:
+ -
./../otel-rules/inspect-e2e.yaml:/skywalking/config/otel-rules/inspect-e2e.yaml
+ environment:
+ SW_ADMIN_SERVER: default
+ SW_INSPECT: default
+ JAVA_OPTS: "-Xms512m -Xmx1g"
+ SW_STORAGE: banyandb
+ SW_OTEL_RECEIVER: default
+ SW_OTEL_RECEIVER_ENABLED_OTEL_METRICS_RULES: "inspect-e2e"
+ ports:
+ - "17128:17128"
+ - "12800:12800"
+ depends_on:
+ banyandb:
+ condition: service_healthy
+
+ oap-b:
+ extends:
+ file: ../../../script/docker-compose/base-compose.yml
+ service: oap
+ environment:
+ SW_ADMIN_SERVER: default
+ SW_INSPECT: default
+ JAVA_OPTS: "-Xms512m -Xmx1g"
+ SW_STORAGE: banyandb
+ ports:
+ - "17129:17128"
+ - "12801:12800"
+ depends_on:
+ # Stagger boot: let oap-a initialize the shared storage schema first,
then
+ # oap-b joins an already-initialized backend. Two OAPs racing first-time
+ # schema init on a fresh shared DB conflict on ES/JDBC; this also
mirrors a
+ # real "second OAP joins the cluster" order.
+ oap-a:
+ condition: service_healthy
+
+ banyandb:
+ extends:
+ file: ../../../script/docker-compose/base-compose.yml
+ service: banyandb
+
+ otlp-emitter:
+ build:
+ context: ../../runtime-rule/mal-storage/otlp-emitter
+ networks:
+ - e2e
+ environment:
+ OTLP_ENDPOINT: http://oap-a:11800
+ EMITTER_SERVICE: inspect-e2e-svc
+ EMITTER_INSTANCE: inspect-e2e-i1
+ depends_on:
+ oap-a:
+ condition: service_healthy
+
+networks:
+ e2e:
diff --git a/test/e2e-v2/cases/inspect/banyandb/e2e.yaml
b/test/e2e-v2/cases/inspect/banyandb/e2e.yaml
new file mode 100644
index 0000000000..f49abbddf1
--- /dev/null
+++ b/test/e2e-v2/cases/inspect/banyandb/e2e.yaml
@@ -0,0 +1,59 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Inspect API e2e (existing aware path + new foreign-metric path) — BanyanDB.
+# Two OAPs share one BanyanDB; oap-a defines meter_inspect_e2e_pool, oap-b
does not.
+# The flow script asserts the aware path on oap-a (17128) and the foreign path
on
+# oap-b (17129, via --value-column / --value-type).
+
+setup:
+ env: compose
+ file: docker-compose.yml
+ timeout: 25m
+ init-system-environment: ../../../script/env
+ steps:
+ - name: set PATH
+ command: export PATH=/tmp/skywalking-infra-e2e/bin:$PATH
+ - name: install yq
+ command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh yq
+ - name: install swctl
+ command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh swctl
+ - name: install jq
+ command: |
+ if ! command -v jq >/dev/null 2>&1; then
+ curl -fsSL -o /tmp/skywalking-infra-e2e/bin/jq \
+
https://github.com/jqlang/jq/releases/download/jq-1.7.1/jq-linux-amd64
+ chmod +x /tmp/skywalking-infra-e2e/bin/jq
+ fi
+ - name: drive inspect aware + foreign flow
+ command: |
+ set -euo pipefail
+ export PATH=/tmp/skywalking-infra-e2e/bin:$PATH
+ export A_REST=http://127.0.0.1:17128
+ export B_REST=http://127.0.0.1:17129
+ bash test/e2e-v2/cases/inspect/inspect-foreign-flow.sh
+
+verify:
+ retry:
+ count: 1
+ interval: 1s
+ cases:
+ # The flow script drives every assertion inline; this trailing smoke check
+ # confirms the inspect port is still serving after the foreign-read phase.
+ - query: swctl --display json --admin-url=http://127.0.0.1:17128 admin
inspect metrics --regex meter_inspect_e2e_pool >/dev/null && echo ok
+ expected: ../expected/ok.txt
+
+cleanup:
+ on: always
diff --git a/test/e2e-v2/cases/inspect/elasticsearch/docker-compose.yml
b/test/e2e-v2/cases/inspect/elasticsearch/docker-compose.yml
new file mode 100644
index 0000000000..d408ac3928
--- /dev/null
+++ b/test/e2e-v2/cases/inspect/elasticsearch/docker-compose.yml
@@ -0,0 +1,92 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Inspect foreign-metric e2e — Elasticsearch. Two OAPs share one ES (no
cluster).
+# oap-a loads the inspect-e2e otel-rule (→ meter_inspect_e2e_pool); oap-b does
not,
+# so that metric is foreign to it (read via the merged metrics-all index).
+services:
+ es:
+ image: elastic/elasticsearch:${ES_VERSION}
+ expose:
+ - 9200
+ networks:
+ - e2e
+ environment:
+ - discovery.type=single-node
+ - cluster.routing.allocation.disk.threshold_enabled=false
+ - xpack.security.enabled=false
+ healthcheck:
+ test: ["CMD", "bash", "-c", "cat < /dev/null > /dev/tcp/127.0.0.1/9200"]
+ interval: 5s
+ timeout: 60s
+ retries: 120
+
+ oap-a:
+ extends:
+ file: ../../../script/docker-compose/base-compose.yml
+ service: oap
+ volumes:
+ -
./../otel-rules/inspect-e2e.yaml:/skywalking/config/otel-rules/inspect-e2e.yaml
+ environment:
+ SW_ADMIN_SERVER: default
+ SW_INSPECT: default
+ JAVA_OPTS: "-Xms512m -Xmx1g"
+ SW_STORAGE: elasticsearch
+ SW_STORAGE_ES_CLUSTER_NODES: es:9200
+ SW_OTEL_RECEIVER: default
+ SW_OTEL_RECEIVER_ENABLED_OTEL_METRICS_RULES: "inspect-e2e"
+ ports:
+ - "17128:17128"
+ - "12800:12800"
+ depends_on:
+ es:
+ condition: service_healthy
+
+ oap-b:
+ extends:
+ file: ../../../script/docker-compose/base-compose.yml
+ service: oap
+ environment:
+ SW_ADMIN_SERVER: default
+ SW_INSPECT: default
+ JAVA_OPTS: "-Xms512m -Xmx1g"
+ SW_STORAGE: elasticsearch
+ SW_STORAGE_ES_CLUSTER_NODES: es:9200
+ ports:
+ - "17129:17128"
+ - "12801:12800"
+ depends_on:
+ # Stagger boot: let oap-a initialize the shared storage schema first,
then
+ # oap-b joins an already-initialized backend. Two OAPs racing first-time
+ # index init on a fresh shared ES conflict; this also mirrors a real
+ # "second OAP joins the cluster" order.
+ oap-a:
+ condition: service_healthy
+
+ otlp-emitter:
+ build:
+ context: ../../runtime-rule/mal-storage/otlp-emitter
+ networks:
+ - e2e
+ environment:
+ OTLP_ENDPOINT: http://oap-a:11800
+ EMITTER_SERVICE: inspect-e2e-svc
+ EMITTER_INSTANCE: inspect-e2e-i1
+ depends_on:
+ oap-a:
+ condition: service_healthy
+
+networks:
+ e2e:
diff --git a/test/e2e-v2/cases/inspect/elasticsearch/e2e.yaml
b/test/e2e-v2/cases/inspect/elasticsearch/e2e.yaml
new file mode 100644
index 0000000000..4e12b3b052
--- /dev/null
+++ b/test/e2e-v2/cases/inspect/elasticsearch/e2e.yaml
@@ -0,0 +1,59 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Inspect API e2e (existing aware path + new foreign-metric path) —
Elasticsearch.
+# Two OAPs share one Elasticsearch; oap-a defines meter_inspect_e2e_pool,
oap-b does not.
+# The flow script asserts the aware path on oap-a (17128) and the foreign path
on
+# oap-b (17129, via --value-column / --value-type).
+
+setup:
+ env: compose
+ file: docker-compose.yml
+ timeout: 25m
+ init-system-environment: ../../../script/env
+ steps:
+ - name: set PATH
+ command: export PATH=/tmp/skywalking-infra-e2e/bin:$PATH
+ - name: install yq
+ command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh yq
+ - name: install swctl
+ command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh swctl
+ - name: install jq
+ command: |
+ if ! command -v jq >/dev/null 2>&1; then
+ curl -fsSL -o /tmp/skywalking-infra-e2e/bin/jq \
+
https://github.com/jqlang/jq/releases/download/jq-1.7.1/jq-linux-amd64
+ chmod +x /tmp/skywalking-infra-e2e/bin/jq
+ fi
+ - name: drive inspect aware + foreign flow
+ command: |
+ set -euo pipefail
+ export PATH=/tmp/skywalking-infra-e2e/bin:$PATH
+ export A_REST=http://127.0.0.1:17128
+ export B_REST=http://127.0.0.1:17129
+ bash test/e2e-v2/cases/inspect/inspect-foreign-flow.sh
+
+verify:
+ retry:
+ count: 1
+ interval: 1s
+ cases:
+ # The flow script drives every assertion inline; this trailing smoke check
+ # confirms the inspect port is still serving after the foreign-read phase.
+ - query: swctl --display json --admin-url=http://127.0.0.1:17128 admin
inspect metrics --regex meter_inspect_e2e_pool >/dev/null && echo ok
+ expected: ../expected/ok.txt
+
+cleanup:
+ on: always
diff --git a/test/e2e-v2/cases/inspect/expected/ok.txt
b/test/e2e-v2/cases/inspect/expected/ok.txt
new file mode 100644
index 0000000000..9766475a41
--- /dev/null
+++ b/test/e2e-v2/cases/inspect/expected/ok.txt
@@ -0,0 +1 @@
+ok
diff --git a/test/e2e-v2/cases/inspect/inspect-foreign-flow.sh
b/test/e2e-v2/cases/inspect/inspect-foreign-flow.sh
new file mode 100755
index 0000000000..f86a130f20
--- /dev/null
+++ b/test/e2e-v2/cases/inspect/inspect-foreign-flow.sh
@@ -0,0 +1,106 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Inspect API e2e — existing (OAP-aware) + new (foreign-metric) paths against
two
+# independent OAPs that SHARE one storage backend (no cluster).
+#
+# OAP-A loads otel-rules/inspect-e2e.yaml and turns the OTLP emitter's
+# e2e_rr_pool_size into the Service metric meter_inspect_e2e_pool.
+# OAP-B loads NO such rule, so meter_inspect_e2e_pool is "foreign" to it —
+# absent from its local registry but present in the shared storage.
+#
+# Asserts:
+# 1. OAP-A /inspect/metrics lists the metric, /inspect/entities (AWARE/old
path)
+# returns inspect-e2e-svc with an mqeEntity.
+# 2. OAP-B /inspect/metrics does NOT list it.
+# 3. OAP-B /inspect/entities WITHOUT valueColumn/valueType → "unknown
locally".
+# 4. OAP-B /inspect/entities WITH valueColumn/valueType (FOREIGN/new path)
returns
+# the SAME entity, scope=null, no mqeEntity.
+set -euo pipefail
+
+A_REST="${A_REST:-http://127.0.0.1:17128}"
+B_REST="${B_REST:-http://127.0.0.1:17129}"
+METRIC="meter_inspect_e2e_pool"
+SVC="inspect-e2e-svc"
+SETTLE="${SETTLE:-360}"
+
+log() { echo "[inspect-flow] $*" >&2; }
+fail() { echo "[inspect-flow] FAIL: $*" >&2; exit 1; }
+
+a_inspect() { swctl --display json --admin-url="${A_REST}" admin inspect "$@";
}
+b_inspect() { swctl --display json --admin-url="${B_REST}" admin inspect "$@";
}
+
+DAY="$(date -u +%Y-%m-%d)"
+
+log "wait for both OAP inspect ports"
+for _ in $(seq 1 90); do
+ a_inspect metrics --regex 'service_.*' >/dev/null 2>&1 && \
+ b_inspect metrics --regex 'service_.*' >/dev/null 2>&1 && break
+ sleep 2
+done
+
+# --- 1. OAP-A registers + persists the custom metric (AWARE / existing path)
---
+log "await ${METRIC} registered + producing entities on OAP-A (≤${SETTLE}s)"
+deadline=$(( $(date +%s) + SETTLE ))
+a_rows=""
+while :; do
+ if a_inspect metrics --regex "${METRIC}" | jq -e '.metrics[]? |
select(.name=="'"${METRIC}"'")' >/dev/null 2>&1; then
+ a_rows="$(a_inspect entities --metric "${METRIC}" --start "${DAY}" --end
"${DAY}" --step DAY 2>/dev/null || true)"
+ if echo "${a_rows}" | jq -e '.rows[]? |
select(.decoded.serviceName=="'"${SVC}"'")' >/dev/null 2>&1; then
+ break
+ fi
+ fi
+ (( $(date +%s) < deadline )) || fail "OAP-A never produced ${METRIC} entity
${SVC} within ${SETTLE}s"
+ sleep 5
+done
+echo "${a_rows}" | jq -e '.rows[] | select(.decoded.serviceName=="'"${SVC}"'")
| .mqeEntity.serviceName=="'"${SVC}"'"' >/dev/null \
+ || fail "OAP-A aware row missing mqeEntity for ${SVC}: ${a_rows}"
+log " ✓ OAP-A aware /inspect/entities returns ${SVC} with mqeEntity (old
path)"
+
+# Value column as OAP-A reports it (applies the per-engine override, e.g.
value -> value_ on jdbc).
+VC="$(a_inspect metrics --regex "${METRIC}" | jq -r '.metrics[] |
select(.name=="'"${METRIC}"'") | .valueColumnName')"
+[ -n "${VC}" ] && [ "${VC}" != "null" ] || fail "could not read
valueColumnName from OAP-A"
+log " OAP-A reports valueColumn=${VC}"
+
+# --- 2. OAP-B does not know the metric ---
+if b_inspect metrics --regex "${METRIC}" | jq -e '.metrics[]? |
select(.name=="'"${METRIC}"'")' >/dev/null 2>&1; then
+ fail "OAP-B unexpectedly knows ${METRIC} (should be foreign)"
+fi
+log " ✓ OAP-B /inspect/metrics excludes ${METRIC}"
+
+# --- 3. OAP-B aware path (no metadata) is rejected ---
+if out="$(b_inspect entities --metric "${METRIC}" --start "${DAY}" --end
"${DAY}" --step DAY 2>&1)"; then
+ fail "OAP-B aware path unexpectedly succeeded for foreign metric: ${out}"
+fi
+echo "${out}" | grep -qi "unknown locally" \
+ || fail "OAP-B aware path expected 'unknown locally', got: ${out}"
+log " ✓ OAP-B aware /inspect/entities (no metadata) → unknown locally"
+
+# --- 4. OAP-B foreign path (valueColumn + valueType) reads the shared-storage
rows ---
+b_rows="$(b_inspect entities --metric "${METRIC}" --value-column "${VC}"
--value-type LONG \
+ --start "${DAY}" --end "${DAY}" --step DAY)" \
+ || fail "OAP-B foreign path errored"
+echo "${b_rows}" | jq -e '.rows[]? |
select(.decoded.serviceName=="'"${SVC}"'")' >/dev/null \
+ || fail "OAP-B foreign path returned no ${SVC} row: ${b_rows}"
+# Foreign response degrades: scope null (the OAP emits null; swctl's typed
model renders it as
+# an empty string — accept either), no mqeEntity.
+[ -z "$(echo "${b_rows}" | jq -r '.scope // ""')" ] \
+ || fail "OAP-B foreign response should have empty/null scope: ${b_rows}"
+echo "${b_rows}" | jq -e '.rows[] | select(.decoded.serviceName=="'"${SVC}"'")
| .mqeEntity == null' >/dev/null \
+ || fail "OAP-B foreign row should carry no mqeEntity: ${b_rows}"
+log " ✓ OAP-B FOREIGN /inspect/entities returns ${SVC}, scope=null, no
mqeEntity (new path)"
+
+log "=== inspect-foreign-flow.sh PASSED ==="
diff --git
a/test/e2e-v2/cases/storage/expected/inspect-entities-unknown-metric.yml
b/test/e2e-v2/cases/inspect/otel-rules/inspect-e2e.yaml
similarity index 59%
copy from test/e2e-v2/cases/storage/expected/inspect-entities-unknown-metric.yml
copy to test/e2e-v2/cases/inspect/otel-rules/inspect-e2e.yaml
index 80d0b6f0e3..70477a3b6f 100644
--- a/test/e2e-v2/cases/storage/expected/inspect-entities-unknown-metric.yml
+++ b/test/e2e-v2/cases/inspect/otel-rules/inspect-e2e.yaml
@@ -12,6 +12,14 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-# /inspect/entities for a metric that is not in ValueColumnMetadata — 400
-# error JSON shape (curl -s strips status but the body is the ErrorResponse).
-error: 'unknown metric: nonexistent_metric_xyz'
+
+# Mounted on OAP-A only. Turns the shared OTLP emitter's `e2e_rr_pool_size`
gauge
+# into the Service-scoped metric `meter_inspect_e2e_pool`. OAP-B does NOT load
this
+# rule, so the metric is "foreign" to B — readable only through the new
+# valueColumn/valueType inspect path. (Reuses the runtime-rule otlp-emitter,
which
+# publishes e2e_rr_pool_size tagged with service_name.)
+expSuffix: service(['service_name'], Layer.GENERAL)
+metricPrefix: meter_inspect_e2e
+metricsRules:
+ - name: pool
+ exp: e2e_rr_pool_size.sum(['service_name'])
diff --git a/test/e2e-v2/cases/inspect/postgresql/docker-compose.yml
b/test/e2e-v2/cases/inspect/postgresql/docker-compose.yml
new file mode 100644
index 0000000000..e38a2c2a09
--- /dev/null
+++ b/test/e2e-v2/cases/inspect/postgresql/docker-compose.yml
@@ -0,0 +1,92 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Inspect foreign-metric e2e — PostgreSQL (JDBC). Two OAPs share one PostgreSQL
+# (no cluster). oap-a loads the inspect-e2e otel-rule (→
meter_inspect_e2e_pool);
+# oap-b does not, so that metric is foreign to it (read by probing the function
+# tables on the table_name discriminator).
+services:
+ postgres:
+ image: postgres:13
+ expose:
+ - 5432
+ networks:
+ - e2e
+ environment:
+ - POSTGRES_PASSWORD=123456
+ - POSTGRES_DB=skywalking
+ healthcheck:
+ test: ["CMD", "bash", "-c", "cat < /dev/null > /dev/tcp/127.0.0.1/5432"]
+ interval: 5s
+ timeout: 60s
+ retries: 120
+
+ oap-a:
+ extends:
+ file: ../../../script/docker-compose/base-compose.yml
+ service: oap
+ volumes:
+ -
./../otel-rules/inspect-e2e.yaml:/skywalking/config/otel-rules/inspect-e2e.yaml
+ environment:
+ SW_ADMIN_SERVER: default
+ SW_INSPECT: default
+ JAVA_OPTS: "-Xms512m -Xmx1g"
+ SW_STORAGE: postgresql
+ SW_JDBC_URL: "jdbc:postgresql://postgres:5432/skywalking"
+ SW_OTEL_RECEIVER: default
+ SW_OTEL_RECEIVER_ENABLED_OTEL_METRICS_RULES: "inspect-e2e"
+ ports:
+ - "17128:17128"
+ - "12800:12800"
+ depends_on:
+ postgres:
+ condition: service_healthy
+
+ oap-b:
+ extends:
+ file: ../../../script/docker-compose/base-compose.yml
+ service: oap
+ environment:
+ SW_ADMIN_SERVER: default
+ SW_INSPECT: default
+ JAVA_OPTS: "-Xms512m -Xmx1g"
+ SW_STORAGE: postgresql
+ SW_JDBC_URL: "jdbc:postgresql://postgres:5432/skywalking"
+ ports:
+ - "17129:17128"
+ - "12801:12800"
+ depends_on:
+ # Stagger boot: let oap-a initialize the shared storage schema first,
then
+ # oap-b joins an already-initialized backend. Two OAPs racing first-time
+ # DDL on a fresh shared PostgreSQL conflict; this also mirrors a real
+ # "second OAP joins the cluster" order.
+ oap-a:
+ condition: service_healthy
+
+ otlp-emitter:
+ build:
+ context: ../../runtime-rule/mal-storage/otlp-emitter
+ networks:
+ - e2e
+ environment:
+ OTLP_ENDPOINT: http://oap-a:11800
+ EMITTER_SERVICE: inspect-e2e-svc
+ EMITTER_INSTANCE: inspect-e2e-i1
+ depends_on:
+ oap-a:
+ condition: service_healthy
+
+networks:
+ e2e:
diff --git a/test/e2e-v2/cases/inspect/postgresql/e2e.yaml
b/test/e2e-v2/cases/inspect/postgresql/e2e.yaml
new file mode 100644
index 0000000000..f11ae2b7f0
--- /dev/null
+++ b/test/e2e-v2/cases/inspect/postgresql/e2e.yaml
@@ -0,0 +1,59 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Inspect API e2e (existing aware path + new foreign-metric path) — PostgreSQL.
+# Two OAPs share one PostgreSQL; oap-a defines meter_inspect_e2e_pool, oap-b
does not.
+# The flow script asserts the aware path on oap-a (17128) and the foreign path
on
+# oap-b (17129, via --value-column / --value-type).
+
+setup:
+ env: compose
+ file: docker-compose.yml
+ timeout: 25m
+ init-system-environment: ../../../script/env
+ steps:
+ - name: set PATH
+ command: export PATH=/tmp/skywalking-infra-e2e/bin:$PATH
+ - name: install yq
+ command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh yq
+ - name: install swctl
+ command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh swctl
+ - name: install jq
+ command: |
+ if ! command -v jq >/dev/null 2>&1; then
+ curl -fsSL -o /tmp/skywalking-infra-e2e/bin/jq \
+
https://github.com/jqlang/jq/releases/download/jq-1.7.1/jq-linux-amd64
+ chmod +x /tmp/skywalking-infra-e2e/bin/jq
+ fi
+ - name: drive inspect aware + foreign flow
+ command: |
+ set -euo pipefail
+ export PATH=/tmp/skywalking-infra-e2e/bin:$PATH
+ export A_REST=http://127.0.0.1:17128
+ export B_REST=http://127.0.0.1:17129
+ bash test/e2e-v2/cases/inspect/inspect-foreign-flow.sh
+
+verify:
+ retry:
+ count: 1
+ interval: 1s
+ cases:
+ # The flow script drives every assertion inline; this trailing smoke check
+ # confirms the inspect port is still serving after the foreign-read phase.
+ - query: swctl --display json --admin-url=http://127.0.0.1:17128 admin
inspect metrics --regex meter_inspect_e2e_pool >/dev/null && echo ok
+ expected: ../expected/ok.txt
+
+cleanup:
+ on: always
diff --git
a/test/e2e-v2/cases/storage/expected/inspect-entities-unknown-metric.yml
b/test/e2e-v2/cases/storage/expected/inspect-entities-unknown-metric.yml
index 80d0b6f0e3..f327020b8d 100644
--- a/test/e2e-v2/cases/storage/expected/inspect-entities-unknown-metric.yml
+++ b/test/e2e-v2/cases/storage/expected/inspect-entities-unknown-metric.yml
@@ -12,6 +12,7 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-# /inspect/entities for a metric that is not in ValueColumnMetadata — 400
-# error JSON shape (curl -s strips status but the body is the ErrorResponse).
-error: 'unknown metric: nonexistent_metric_xyz'
+# /inspect/entities for a metric that is not in ValueColumnMetadata and
carries no
+# valueColumn/valueType — 400. The handler returns "metric unknown locally:
<name> —
+# provide valueColumn and valueType …"; the query reconstructs the {error: …}
prefix.
+error: 'metric unknown locally: nonexistent_metric_xyz'
diff --git a/test/e2e-v2/cases/storage/storage-cases.yaml
b/test/e2e-v2/cases/storage/storage-cases.yaml
index 799125791a..155889aa10 100644
--- a/test/e2e-v2/cases/storage/storage-cases.yaml
+++ b/test/e2e-v2/cases/storage/storage-cases.yaml
@@ -216,12 +216,13 @@ cases:
DAY=$(date -u +"%Y-%m-%d")
swctl --display json --admin-url=http://${oap_host}:${oap_17128} admin
inspect entities --metric service_cpm --start "${DAY} 00" --end "${DAY} 23"
--step HOUR | yq -P
expected: expected/inspect-entities-service-cpm-hour.yml
- # Negative — unknown metric: swctl exits non-zero and renders the typed error
- # envelope ("... HTTP 400: unknown metric: <name>"). Reconstruct the original
- # {error: ...} shape from the message so the expected file is unchanged.
+ # Negative — metric absent from this OAP: swctl exits non-zero and renders
the
+ # typed error envelope ("... HTTP 400: metric unknown locally: <name> —
provide
+ # valueColumn and valueType …"). The handler steers the caller toward the
+ # foreign-metric path; reconstruct the {error: ...} shape from the message
prefix.
- query: |
DAY=$(date -u +"%Y-%m-%d")
out=$(swctl --display yaml --admin-url=http://${oap_host}:${oap_17128}
admin inspect entities --metric nonexistent_metric_xyz --start "${DAY}" --end
"${DAY}" --step DAY 2>&1 || true)
- msg=$(echo "$out" | grep -o 'unknown metric: [a-z_0-9]*' | head -1)
+ msg=$(echo "$out" | grep -o 'metric unknown locally: [a-z_0-9]*' | head
-1)
yq -n ".error = \"${msg}\""
expected: expected/inspect-entities-unknown-metric.yml
diff --git a/test/e2e-v2/script/env b/test/e2e-v2/script/env
index ced8ba857f..373df11394 100644
--- a/test/e2e-v2/script/env
+++ b/test/e2e-v2/script/env
@@ -27,7 +27,7 @@ SW_BANYANDB_COMMIT=c2d925e4eae4d77edda94e1fd438243483960150
SW_AGENT_PHP_COMMIT=d1114e7be5d89881eec76e5b56e69ff844691e35
SW_PREDICTOR_COMMIT=54a0197654a3781a6f73ce35146c712af297c994
-SW_CTL_COMMIT=b447211a9319eeb29a445335e9c2536f8c1aa23d
+SW_CTL_COMMIT=90365c4bc59de3704ff81b4cefe55d09f706d00d
# Third-party image versions used by e2e infrastructure (not skywalking
# components). Pinned here so the matrix is reproducible.