mmodzelewski opened a new issue, #2388:
URL: https://github.com/apache/iggy/issues/2388
## Overview
Implement an HTTP-based configuration provider that allows connectors to
fetch sink and source configurations from remote HTTP APIs. The provider must
adapt to different API conventions (RESTful, query-based, wrapped responses)
rather than forcing a single standard.
## Problem
Organizations need to integrate Iggy connectors with existing configuration
management systems, but these systems use different API conventions:
- **URL patterns vary:** Some APIs use RESTful paths
(`/sinks/{key}/configs`), others use query parameters
(`/api?type=sink&key={key}`)
- **Response formats differ:** Many APIs wrap responses in metadata
structures (`{"status": "ok", "data": {...}}`)
- **Custom requirements:** Multi-tenant systems, API versioning, and legacy
integrations require flexible URL structures
A rigid HTTP provider would force organizations to build adapters or
proxies. We need the provider to adapt to existing APIs instead.
## Proposed Solution
A flexible HTTP provider with two core capabilities:
### 1. URL Templates
Customize URL patterns using template variables:
```toml
[connectors.url_templates]
create_sink = "/v2/sinks/{key}" # Path-based
get_sink_config = "/api?action=get&key={key}&v={version}" # Query-based
```
**Variables:** `{key}`, `{version}`, `{type}`
### 2. Response Transformation
Extract data from nested/wrapped responses using dot-notation:
```toml
[connectors.response]
data_path = "data.config" # Extract from {"data": {"config": {...}}}
error_path = "error.message" # Extract error messages
```
## Configuration
### Basic Configuration
```toml
[connectors]
config_type = "http"
base_url = "http://config-server:8080";
timeout = "30 s"
[connectors.custom_headers]
Authorization = "Bearer <token>"
X-Custom = "value"
```
### Configuration Options
| Parameter | Type | Required | Default | Description
|
|-----------------------|--------|----------|------------------|---------------------------------------------|
| `config_type` | string | Yes | - | Set to
`"http"` |
| `base_url` | string | Yes | - | Base URL of
the configuration API |
| `timeout` | string | No | `"30 s"` | Request
timeout (e.g., `"30 s"`, `"5 min"`) |
| `custom_headers` | map | No | `{}` | HTTP
headers (auth, tenant ID, etc.) |
| `url_templates` | map | No | RESTful defaults | Custom URL
patterns |
| `response.data_path` | string | No | none | Path to
extract response data |
| `response.error_path` | string | No | none | Path to
extract error messages |
### Default Behavior (RESTful)
Without customization, the provider uses RESTful conventions:
```
POST /sinks/{key}/configs - Create sink
GET /sinks/{key}/configs - List versions
GET /sinks/{key}/configs/{version} - Get version
GET /sinks/{key}/configs/active - Get active
PUT /sinks/{key}/configs/active - Set active
DELETE /sinks/{key}/configs?version={v} - Delete
# Same pattern for /sources/{key}/configs
GET /configs/active - All active configs
GET /configs/active/versions - Version info
```
## How It Works
### Required Operations
Configuration servers must implement 14 operations (6 for sinks, 6 for
sources, 2 global). The exact URLs are customizable.
**Sink Operations:**
1. Create sink configuration
2. Get all versions for a sink
3. Get specific version
4. Get active configuration
5. Set active version
6. Delete configuration
**Source Operations:** (same as sinks, different endpoints)
**Global Operations:**
1. Get all active configurations
2. Get version metadata
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
