Rishavbhattarai opened a new issue, #12823: URL: https://github.com/apache/seatunnel/issues/12823
### Search before asking - [x] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement. ### Description ## Overview Design proposal for a new **Intercom Source** connector (Connector V2), addressing the Intercom row in #10753. It would be a `connector-http-intercom` module built on `connector-http-base`, the same foundation as the Asana, Linear and Splunk connectors, with a small v1 scope. I checked current `dev`: there is no Intercom module, plugin mapping, docs or open PR. I have also opened a design issue for a Typeform Source (#12822) and will keep the two scopes separate. ## 1. v1 scope: endpoint and record type - **Record type:** contacts only. Conversations, tickets, companies and the search endpoints are out of scope for v1 and can follow in separate PRs. - **Endpoint:** `GET https://api.intercom.io/contacts`. - **Region:** an optional `base_url`, default `https://api.intercom.io`. Intercom documents `https://api.eu.intercom.io` and `https://api.au.intercom.io` for workspaces hosted in those regions. - **Version header:** Intercom's API is versioned with an `Intercom-Version` header. The connector will send a pinned version, set through an optional `api_version` option, so the response shape does not change when the workspace's default version changes. The reference page lists `2.17` as its default, which is what I plan to pin; I will confirm the field names against that version before implementation. - **Fields:** a fixed schema of top-level scalar fields: `id`, `role`, `email`, `name`, `phone`, `external_id`, `created_at`, `updated_at`. The reference page says a contact has 47 properties but does not list them in the content I could read, so I will check this list against the reference before coding and update this issue if a field name differs. Nested objects such as custom attributes are out of scope for v1. ## 2. Authentication - **v1: access token only**, through a required `api_key` option. OAuth is deferred. - The reference page documents `Authorization: Bearer <token>`. The connector builds that header itself, so users supply the raw token. ## 3. Secret boundary - **Where it lives:** only in the `api_key` option, a plain string option like the other HTTP connectors. I found no sensitive-option marker in them and am not introducing one; I will follow a maintainer-preferred convention if there is one. - **On the wire:** copied into the `Authorization` header and nowhere else. Never put in `params` or `body`. - **Logs and errors:** the connector logs only status codes and attempt counts, never headers, the config, or the parameter object. Failures use the base reader's `REQUEST_FAILED` message (status code and response body, no request headers). - **Job state:** the connector persists nothing, so the token cannot reach checkpoint state. - **Not covered:** whether the engine masks `api_key` when it logs the job config is outside this connector, and I have not verified it. ## 4. Pagination, incremental sync, and delivery semantics - **Pagination:** uses the base reader's existing `CURSOR` pagination, with no change to the shared client. Intercom's pagination guide says the response carries a `pages.next` object holding a `starting_after` cursor, that the cursor is sent back as the `starting_after` query parameter, and that you keep going "until there is no longer a `next` object". I will set the cursor field to `starting_after` and the response path to `$.pages.next.starting_after`. The base reader parses with JSONPath errors suppressed, so a missing `next` object gives an empty cursor and ends the read. - **Page size:** `per_page` defaults to 150, which the guide gives as the maximum. - **Incremental contract (v1):** a bounded snapshot of all contacts. The list endpoint has no `updated since` filter that I could find, so v1 does not offer one. The cursor lives in memory for one run and is never persisted. An incremental mode could be built later on the search endpoint, which takes a POST body. - **Checkpoint and resume:** single split, nothing persisted. After a failure the read restarts from the first page. - **Duplicates:** possible. A restart can re-emit rows already sent, and contacts created or merged during the read can appear or disappear between pages. Delivery is at-least-once; every row carries the contact `id` so sinks can deduplicate or upsert. No exactly-once claim. - **Mode:** bounded only. Streaming mode is rejected. ## 5. Rate limits and retries - I could not confirm Intercom's rate limit from the pages I read, so I am not stating a number here. I will check the rate-limit page before implementation. - The base client retries only `IOException`s, so a `429` or `5xx` would currently fail the job. Like the Asana connector, the Intercom reader would override `executeRequest()` to also retry `429` and `5xx` with bounded exponential backoff, reusing the existing retry options. After the cap is reached the job fails with a clear error. - `Retry-After` is not supported in v1, because `HttpResponse` does not expose headers. That can be a separate small PR to the shared client if maintainers want it. ## 6. Memory behaviour Pages are fetched and emitted one at a time, each bounded by `per_page` (at most 150), so the full result set is never held in memory. ## 7. Dependencies, license, and shading - **No new third-party runtime dependency.** The connector reuses `connector-http-base` and what it already carries. - **Test scope only:** `okhttp3:mockwebserver`, the same test dependency the Asana and Splunk modules use, with its version managed by the parent pom. ## 8. Deterministic testing (no live credentials) - **Unit tests:** `MockWebServer` to check request construction, multi-page cursor paging (including the last page with no `next`), field mapping, the auth and version headers, and retry on `429` and `5xx`. - **E2E:** a SeaTunnel job against a mock HTTP server that serves canned Intercom-shaped JSON (several pages and an injected `429`), so it needs no token and no network access. ## 9. Deliverables New module and parent-pom registration; `plugin-mapping.properties` entry (`seatunnel.source.Intercom = connector-http-intercom`); `plugin_config` and `seatunnel-dist/pom.xml` updates; factory and plugin name consistency (`getPluginName()` and `factoryIdentifier()` both `"Intercom"`); English and Chinese docs; unit tests; E2E test. ## Out of scope for v1 OAuth; conversations, tickets, companies and other record types; the search endpoints and incremental sync; nested and custom attributes; streaming mode; `Retry-After` support. ### Usage Scenario Support and product teams that keep customer records in Intercom want to load their contacts into a warehouse or database (for example Snowflake, ClickHouse or MySQL) for analysis, instead of exporting by hand. ### Related issues Tracking issue #10753 (Intercom row). Same pattern as the Asana connector (#12712, #12719). Separate from the Typeform proposal (#12822). ### Are you willing to submit a PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
