david-streamlio commented on issue #49: URL: https://github.com/apache/pulsar-connectors/issues/49#issuecomment-4938261032
Investigated the Kusto emulator approach end-to-end (ran `mcr.microsoft.com/azuredataexplorer/kustainer-linux` locally, wrote the test, ran the real sink against it). **The emulator is not viable for this connector** — documenting so no one repeats the effort. **What works against kustainer:** plain HTTP unauthenticated (auth is a non-issue — `/v1/rest/auth/metadata` returns `AzureAD: null`, so the SDK skips AAD), control commands (`.create table`, `.ingest inline`), and queries (`Table | count`). **What doesn't — and it's exactly what the sink uses.** `ADXSink` only ever calls `ingestClient.ingestFromStream(...)` (`ADXSink.java:148`). Verified. That routes through one of two SDK clients, and the emulator services neither: - **Queued ingest** (default): the SDK's `ResourceManager` runs `.get ingestion resources`, which the emulator rejects — `AdminCommandWrongEndpointException: Cannot get ingestion resources from this service endpoint`. The emulator is a single engine node with no Data-Management endpoint, blob storage, or queues, which queued ingestion requires. - **Managed streaming ingest**: returns `BadRequest_StreamingIngestionPolicyNotEnabled`; after enabling the `streamingingestion` policy at both DB and table level (confirmed `IsEnabled: true`), it then returns `BadRequest_EntityNotFound`. The emulator advertises the policy but doesn't implement the streaming data node. End-to-end result: sink wrote 10 records, `Table | count` stayed at 0, background flush threw the SDK errors above. The only way data enters kustainer is via `.ingest inline` control commands, which `IngestClient` never issues — so a passing test would have to insert data itself and assert on it, exercising none of the sink's real code. That is vacuous, so nothing was shipped. **Recommendation.** The emulator route (option in the issue title) is a dead end — a known Microsoft limitation. Realistic alternatives: 1. Keep `ADXSinkE2ETest` as the real coverage but **gate it on CI repository secrets** (run when creds are present, skip-with-notice otherwise) instead of the current silent `SkipException`, so the coverage is at least exercised on scheduled/secret-backed runs. 2. Add a **mock-based unit test** (mock `IngestClient`) asserting `ADXSink` batches, acks, and retries correctly — no emulator, and it covers the connector's own logic. 3. (Larger) refactor `ADXSink` to optionally ingest via `.ingest inline` control commands, which *would* be emulator-testable but changes production behavior — needs maintainer buy-in. Suggest retitling this issue away from 'add a runnable emulator test' toward option 1+2. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
