songzhendong opened a new pull request, #141: URL: https://github.com/apache/skywalking-nodejs/pull/141
### Summary This PR extends the Node.js agent remote layer to align with the **Java `agent.core` gRPC architecture**, and expands runtime metrics from 6 to **12** `instance_nodejs_*` meters. 1. **gRPC v2** — `GRPCChannelManager` multi-backend failover, TLS/mTLS, per-hostname SNI, DNS expand (A/AAAA), optional periodic DNS re-resolve, IPv6 `formatHostPort`, `CommandService` ingestion. 2. **Runtime metrics** — six additional meters (`array_buffers`, `uptime`, `peak_malloced_memory`, `detached_contexts`, `old_space_used`, `new_space_used`); canonical env `SW_AGENT_RUNTIME_METRICS_*` with legacy aliases retained. **Not in this PR:** OAP MAL rules / dashboards (separate PR required), release version bump, Profile/Log/Event clients, full Command executor registration. ### Motivation - **Cross-language parity:** Java agent already uses `GRPCChannelManager`, TLS, DNS expansion, and periodic re-resolve; Node previously lacked equivalent failover and TLS/SNI behavior (notably Kubernetes Headless Service → Pod IP with cert hostname mismatch). - **Maintainability:** v1 refactor introduced `agent.core` layout; v2 completes remote connectivity features on that foundation. ### OAP companion (required for 12-meter dashboards) Meter **reporting** is in this PR. **OAP consumption** (MAL analyzer + dashboard docs + e2e-v2 TLS/DNS/auth cases) is a **separate PR** to `apache/skywalking`: | Item | Location | |------|----------| | 12 MAL rules | `meter-analyzer-config/nodejs-runtime.yaml` | | Dashboard doc | `docs/en/setup/backend/dashboards-nodejs-runtime.md` | | Node.js e2e (ssl / mtls / ssl-dns / auth) | `test/e2e-v2/cases/nodejs/*` | | Agent pin | `test/e2e-v2/script/env` → `SW_AGENT_NODEJS_COMMIT=7c7eba5` | Companion OAP PR: [songzhendong/skywalking #6](https://github.com/songzhendong/skywalking/pull/6). ### Intentional differences from Java (please review) | Area | Java | This PR | Tag | |------|------|---------|-----| | Auth failure (`UNAUTHENTICATED` / `PERMISSION_DENIED`) | Treated as network error → reconnect | Log warn, **no reconnect** | Stricter ops behavior | | mTLS paths set but PEM missing | Warn, continue one-way TLS | **throw** at channel build | Stricter | | Meter burst on reconnect | No per-tick snapshot cap | `SW_AGENT_RUNTIME_METRICS_MAX_SNAPSHOTS_PER_REPORT` default **1** | Tunable guard | | Exported `config` to apps | `Config.Agent.AUTHENTICATION` readable | `publicAgentConfig` omits `authorization` | Node-only hardening | Operational constraints (DNS lookup blocking, runtime sampling on main thread, etc.): [`docs/nodejs-grpc-constraints.md`](docs/nodejs-grpc-constraints.md). ### Key env vars (new / renamed) | Variable | Purpose | |----------|---------| | `SW_AGENT_IS_RESOLVE_DNS_PERIODICALLY` | Periodic DNS re-resolve (Java `collector.is_resolve_dns_periodically`) | | `SW_AGENT_GRPC_CHANNEL_CHECK_INTERVAL` | Channel health check interval (seconds) | | `SW_AGENT_FORCE_RECONNECTION_PERIOD` | Force reconnect period multiplier | | `SW_AGENT_FORCE_TLS` / `SW_AGENT_SSL_*` | TLS/mTLS material paths | | `SW_AGENT_COLLECTOR_GRPC_UPSTREAM_TIMEOUT` | gRPC call deadline (seconds) | | `SW_AGENT_RUNTIME_METRICS_*` | Runtime meter pipeline (replaces `SW_AGENT_NODEJS_RUNTIME_METRICS_*` as canonical) | Legacy `SW_AGENT_NODEJS_RUNTIME_METRICS_*`, `SW_AGENT_NVM_*` aliases still accepted with deprecation warnings. ### Test plan **Agent CI** ([Actions run @ `7c7eba5`](https://github.com/songzhendong/skywalking-nodejs/actions/runs/28783670639)): - [x] Build (Node 20 / 22 / 24) - [x] License - [x] Lint + TestLib - [x] **TestUnitRemote** (config / core / remote / runtime) × 20/22/24 - [x] **TestRemoteE2E** (`static-failover`, `dns-re-resolve`) × 20/22/24 - [x] **TestPlugins** matrix (express meterSize 12, axios, http, ioredis, mongodb, mysql, …) **OAP e2e-v2** ([songzhendong/skywalking #6](https://github.com/songzhendong/skywalking/pull/6), pin `7c7eba5`): - [x] Agent NodeJS Backend / Frontend / Frontend ES / Frontend ES Sharding - [x] Agent NodeJS SSL / SSL DNS / mTLS / Auth ### Files of note | Path | Role | |------|------| | `src/agent/core/remote/*` | Channel manager, TLS, DNS resolver, clients | | `src/agent/core/commands/*` | OAP command queue + dispatch framework | | `src/agent/core/meter/*` | 12 runtime meters | | `tests/remote-e2e/*` | Docker-compose failover E2E (CI) | | `tests/remote/*` | Unit tests for remote layer | | `docs/nodejs-grpc-constraints.md` | Node/grpc-js operational notes | --- ## Comparison tag legend | Tag | Meaning | |-----|---------| | [NEW] | Added in Node v2; no Java equivalent or not in Node v1 | | [PARITY] | Behavior/structure aligned with Java agent.core | | [DIVERGE] | Intentionally different from Java (documented design) | | [IMPROVE] | Stricter or more complete than Java | | [UNCHANGED] | Same limitation/behavior as Java; kept for parity | | [PARTIAL] | Infrastructure present; business executors / full capability still missing | | [MISSING] | Present in Java, still absent in Node (not a v2 regression) | --- ## 1. Remote connectivity / gRPC architecture 1.1 [PARITY] agent.core remote layer refactor - Remove legacy GrpcProtocol / SegmentObjectAdapter / HeartbeatClient / TraceReportClient - Add BootService + ServiceManager orchestration (Java ServiceManager equivalent) - Layered GRPCChannelManager / GRPCChannel / ChannelBuilder / ChannelDecorator - Files: src/agent/core/boot/, src/agent/core/remote/ 1.2 [PARITY] Multi-backend addresses and failover - Comma-separated SW_AGENT_COLLECTOR_BACKEND_SERVICES - Random index channel rotation (Java GRPCChannelManager.run model) - SW_AGENT_FORCE_RECONNECTION_PERIOD / SW_AGENT_GRPC_CHANNEL_CHECK_INTERVAL - grpc-js READY probe + watchConnectivityState 1.3 [NEW] Correct IPv6 gRPC target formatting - GRPCChannel / resolveAddress use formatHostPort - Java: NettyChannelBuilder.forAddress(host, port) supports this natively - Static [::1]:11800 and DNS AAAA records both work 1.4 [UNCHANGED] No custom HTTP/2 keepalive - StandardChannelBuilder sets max message size only - Java StandardChannelBuilder also omits keepalive - See docs/nodejs-grpc-constraints.md §7 1.5 [IMPROVE] No reconnect on auth failure - isGrpcAuthError → log warn; skip reportError/reconnect - Java: PERMISSION_DENIED/UNAUTHENTICATED still treated as network errors and retried 1.6 [PARITY] gRPC upstream timeout - SW_AGENT_COLLECTOR_GRPC_UPSTREAM_TIMEOUT (default 30s) - Java collector.grpc_upstream_timeout equivalent - GrpcUpstreamOptions.grpcUpstreamDeadlineMs() 1.7 [UNCHANGED] No per-stream hard cap on trace client stream - Full buffer splice send (~1000 segments) - Java DataCarrier batch consume — same design --- ## 2. TLS / mTLS 2.1 [PARITY] TLS channel construction - TLSChannelBuilder + StandardChannelBuilder - ca.crt present → TLS; cert+key+ca → mTLS - AgentPackagePath resolves relative paths (Java AgentPackagePath equivalent) - TlsMaterialCache async PEM load; PrivateKeyUtil decrypts key 2.2 [PARITY] TLS SNI - deriveTlsServerNameForConnectHost when DNS resolves to IP - grpc.ssl_target_name_override avoids using IP as SNI - Java: equivalent SSL context logic 2.3 [PARITY] FORCE_TLS / SECURE without CA file - Node: grpc.credentials.createSsl(), system trust store - Java: FORCE_TLS can use system trust store too - When ca.crt exists, PEM under AgentPackagePath still preferred 2.4 [IMPROVE] mTLS paths configured but material missing - throw Error('mTLS material unavailable') - Java TLSChannelBuilder: warn then continue one-way TLS (Node is stricter) 2.5 [IMPROVE] TLS material read hardening - TlsMaterialCache: reject symlinks, 256 KiB cap, realpath, key permission check - Zero private key buffers on shutdown 2.6 [NEW] E2E coverage (OAP companion PR) - ssl / mtls / ssl-dns / auth compose + e2e.yaml --- ## 3. DNS resolution and periodic re-resolve 3.1 [NEW] BackendAddressResolver - parseStaticBackendAddresses / splitHostPort / formatHostPort - expandBackendAddresses (Java InetAddress.getAllByName equivalent) - IPv6 bracket literals, AAAA record expansion 3.2 [NEW] Periodic DNS re-resolution - SW_AGENT_IS_RESOLVE_DNS_PERIODICALLY=true - runCheck reconnect branch refreshes grpcServers - Same index IP change does not rebuild channel (Java parity tests) 3.3 [NEW] DNS lookup guards (Node-specific code constants, not env) - DNS_LOOKUP_TIMEOUT_MS=5000, MAX_DNS_LOOKUP_RECORDS=64 (BackendAddressResolver.ts) - Java blocks on background threads; no such caps needed 3.4 [NEW] E2E - tests/remote-e2e/dns-re-resolve - tests/remote-e2e/static-failover - OAP: ssl-dns e2e + repoint-dns.sh --- ## 4. Runtime metrics (Meter API) 4.1 [NEW] Node.js runtime metrics pipeline (v1 baseline) - RuntimeSampler → RuntimeMetricsCollector → MeterSender - gRPC MeterReportService.collect (client streaming) - Java equivalent: JVMService + JVMMetricsSender (JVMMetric proto, not Meter) - v1 six instance_nodejs_* meters: process_cpu (normalized by logical CPU count) heap_used / heap_total / heap_limit rss / external_memory 4.2 [NEW] v2 adds six instance_nodejs_* meters - instance_nodejs_array_buffers - instance_nodejs_uptime / peak_malloced_memory - instance_nodejs_detached_contexts - instance_nodejs_old_space_used / new_space_used - 12 total (v1 six in 4.1) 4.3 [PARITY] Environment variables (AgentConfig.ts / README.md) - SW_AGENT_RUNTIME_METRICS_* (REPORTER_ACTIVE / COLLECT_PERIOD / REPORT_PERIOD / BUFFER_SIZE) - SW_AGENT_RUNTIME_METRICS_MAX_SNAPSHOTS_PER_REPORT (default 1) - SW_AGENT_RUNTIME_METRICS_HEAP_SPACE_DETAIL (default true) - Default collect/report 1000ms / buffer 600; legacy SW_AGENT_NVM_* / SW_AGENT_NODEJS_* still supported 4.4 [IMPROVE] Per-snapshot collectedAt timestamp - Java: JVMService.setTime() per record 4.5 [DIVERGE] Configurable snapshots per stream tick - SW_AGENT_RUNTIME_METRICS_MAX_SNAPSHOTS_PER_REPORT, default 1 - Java MeterSender/JVMMetricsSender has no such cap 4.6 [PARITY] Graceful shutdown on UNIMPLEMENTED - OAP without Meter API → warn + MeterSender.shutdown() - Java MeterSender.java same pattern 4.7 [NEW] SW_AGENT_RUNTIME_METRICS_HEAP_SPACE_DETAIL - false skips v8.getHeapSpaceStatistics(), lower main-thread cost - old_space_used / new_space_used report 0 4.8 [NEW] OAP companion (MAL/UI for v2 six meters) - `oap-server/server-starter/src/main/resources/meter-analyzer-config/nodejs-runtime.yaml` (12 rules) - `docs/en/setup/backend/dashboards-nodejs-runtime.md` (OAP setup doc; Horizon UI consumes bundled templates) - MAL unit test `oap-server/analyzer/meter-analyzer-scripts-test/src/test/resources/scripts/mal/test-meter-analyzer-config/nodejs-runtime.data.yaml` (12 items); e2e covers trace/TLS/DNS/auth --- ## 5. Trace reporting 5.1 [PARITY] TraceSegmentServiceClient rewrite - BootService pattern same tier as MeterSender - buffer + 1s report timer; channelOverride + stub - Segment.transform() emits proto SegmentObject directly 5.2 [PARITY] Restart report loop after flush - 481b290/916f457 fix: manual flush no longer stops reporting 5.3 [IMPROVE] report timer uses unref() - Avoid blocking Node exit when collector unreachable - Java has no event-loop semantics 5.4 [PARITY] Dispatch OAP Commands from collect callback - TraceSegmentServiceClient → CommandService.receiveCommand(commands) - Java TraceSegmentServiceClient.receiveCommand equivalent 5.5 [IMPROVE] Requeue on pre-write failure - requeueSegments(): failed before write → segments back to buffer - MeterSender.requeueSnapshots() same pattern --- ## 6. Service management / heartbeat 6.1 [PARITY] ServiceManagementClient - reportInstanceProperties + keepAlive alternation - PROPERTIES_REPORT_PERIOD_FACTOR - AuthenticationDecorator metadata 6.2 [PARITY] AgentIDDecorator - gRPC interceptor injects agent id 6.3 [PARITY] Heartbeat response Command dispatch - ServiceManagementClient → CommandService.receiveCommand(commands) - Java ServiceManagementClient.receiveCommand equivalent --- ## 7. Boot lifecycle 7.1 [PARITY] Startup order (sync bootstrap) - agent.start(): PluginInstaller.install() → ServiceManager.boot() - Java: installClassTransformer → ServiceManager.boot() - installAsync() deprecated wrapper; production path uses sync install 7.2 [UNCHANGED] booted flag set even when boot partially fails - ServiceManager.boot(): void; per-service try/catch; always this.booted = true - index.ts: started = true unconditionally after install+boot - Java: same — instrumentation before boot; failures logged only 7.3 [PARITY] flush / destroy - SpanContext.flush → ServiceManager.flush (trace + meter) 7.4 [NEW] whenReady() readiness hook - export whenReady(): Promise<void>; resolves after bootstrap - Does not guarantee OAP connectivity; for tests / optional await - Java premain is sync; no equivalent API --- ## 8. Configuration / environment variables 8.1 [NEW] v2 remote/TLS/DNS vars (see Agent README) - SW_AGENT_IS_RESOLVE_DNS_PERIODICALLY - SW_AGENT_FORCE_RECONNECTION_PERIOD / SW_AGENT_GRPC_CHANNEL_CHECK_INTERVAL - SW_AGENT_COLLECTOR_HEARTBEAT_PERIOD / SW_AGENT_COLLECTOR_GRPC_UPSTREAM_TIMEOUT - SW_AGENT_SSL_TRUSTED_CA_PATH / SSL_CERT_CHAIN_PATH / SSL_KEY_PATH - SW_AGENT_FORCE_TLS / SW_AGENT_SECURE / SW_AGENT_AUTHENTICATION - DNS lookup timeout/record caps: code constants, not env (see 3.3) 8.2 [NEW] Runtime metrics vars (see 4.3, 4.5, 4.7) 8.3 [PARITY] Deprecated alias normalization - normalizeDeprecatedRuntimeMetricOptions() - NVM/JVM legacy env names + startup warn 8.4 [PARITY] FORCE_TLS without CA uses system trust store (see 2.3) 8.5 [IMPROVE] publicAgentConfig omits authorization from app export - index.ts: export { publicAgentConfig as config } - Java: Config.Agent.AUTHENTICATION is public static — Node hides token from apps only --- ## 9. Plugins / instrumentation 9.1 [IMPROVE] installNormal sorted by filename - Java: skywalking-plugin.def ordered load - Bundled path still hard-coded order [DIVERGE / platform limit] 9.2 [UNCHANGED] Instrument before boot (see 7.1) --- ## 10. Java has, Node still lacks / partial (out of v2 scope) 10.1 [PARTIAL] CommandService / CommandExecutorService - Implemented: receiveCommand queue + serialNumber dedup + dispatch - Trace / Heartbeat / Meter callbacks all call receiveCommand - Missing: registerExecutor() never called — ProfileTask/CDS executors not registered - Unknown commands: warn and ignore 10.2 [MISSING] ProfileTask* (3 services) 10.3 [MISSING] AsyncProfilerTask* (3 services) 10.4 [MISSING] LogReportServiceClient 10.5 [MISSING] EventReportServiceClient 10.6 [MISSING] ConfigurationDiscoveryService 10.7 [MISSING] SamplingService (trace sampling) 10.8 [MISSING] StatusCheckService / ServiceInstanceGenerator 10.9 [MISSING] JVMService + JVMMetricsSender (Node uses Meter instead) 10.10 [MISSING] Public MeterService API (custom business meters) --- ## 11. Intentionally unchanged for Java parity Intentionally unchanged for Java parity — v2 left as-is: 11.1 [UNCHANGED] Partial boot failure still leaves instrumentation active (install before boot, see 7.2) 11.2 [UNCHANGED] No hard cap on trace single-stream burst 11.3 [UNCHANGED] CA availability probe on event loop (async `preloadTlsMaterials`; no Java-style background CA watcher) Note: former “silent mTLS downgrade” now throws (see 2.4); auth failure no-reconnect is [IMPROVE] (see 1.5), not unchanged. --- ## 12. Tests and CI 12.1 [NEW] Remote unit test suite - GRPCChannelManager / BackendAddressResolver / TLSChannelBuilder - MeterSender / TraceSegmentServiceClient / ServiceManagementClient - GRPCChannel / AuthenticationDecorator / ServiceManager, etc. 12.2 [NEW] Runtime unit tests - RuntimeSampler / RuntimeMetricsCollector 12.3 [NEW] Node remote-e2e (docker-compose, CI) - dns-re-resolve / static-failover / support/helpers.ts / common/server.ts 12.4 [NEW] OAP e2e-v2 cases (companion PR) - [songzhendong/skywalking PR #6](https://github.com/songzhendong/skywalking/pull/6) · pin `7c7eba5` - Cases: Backend, Frontend, Frontend ES, Frontend ES Sharding, ssl, mtls, ssl-dns, auth 12.5 [NEW] whenReady unit tests - tests/core/whenReady.test.ts 12.6 [NEW] Config unit test suite - AgentConfig.test.ts / tls / dns / heartbeat --- ## 13. Quick comparison matrix | Area | Items | NEW | PARITY | DIVERGE | IMPROVE | UNCHANGED | PARTIAL | MISSING | |------|-------|-----|--------|---------|---------|-----------|---------|---------| | Remote/gRPC | 7 | 1 | 3 | 0 | 1 | 2 | 0 | 0 | | TLS/mTLS | 6 | 1 | 3 | 0 | 2 | 0 | 0 | 0 | | DNS | 4 | 4 | 0 | 0 | 0 | 0 | 0 | 0 | | Runtime Metrics | 8 | 4 | 2 | 1 | 1 | 0 | 0 | 0 | | Trace | 5 | 0 | 3 | 0 | 2 | 0 | 0 | 0 | | Heartbeat / mgmt | 3 | 0 | 3 | 0 | 0 | 0 | 0 | 0 | | Boot | 4 | 1 | 2 | 0 | 0 | 1 | 0 | 0 | | Config | 5 | 2 | 2 | 0 | 1 | 0 | 0 | 0 | | Plugins | 2 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | | Java has, Node lacks | 10 | 0 | 0 | 0 | 0 | 0 | 1 | 9 | | Java parity unchanged | 3 | 0 | 0 | 0 | 0 | 3 | 0 | 0 | | Tests/CI | 6 | 6 | 0 | 0 | 0 | 0 | 0 | 0 | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
