Rishavbhattarai opened a new issue, #12822: URL: https://github.com/apache/seatunnel/issues/12822
### Search before asking - [x] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement. ### Description ## Overview Design proposal for a new **Typeform Source** connector (Connector V2), addressing the Typeform row in #10753. It would be a `connector-http-typeform` module built on `connector-http-base`, the same foundation as the Asana, Linear and Splunk connectors, with a small v1 scope. I checked current `dev`: there is no Typeform module, plugin mapping, docs or open PR. ## 1. v1 scope: endpoint and record type - **Record type:** form responses only. Forms, workspaces, webhooks and the Create API are out of scope. - **Endpoint:** `GET https://api.typeform.com/forms/{form_id}/responses`, scoped by a required `form_id`. - **Optional filters, passed straight to Typeform:** `since`, `until` (Unix seconds or ISO 8601 UTC), and `response_type` (Typeform's default is `completed`). - **Region:** an optional `base_url`, default `https://api.typeform.com`, so accounts on Typeform's EU data center can set `https://api.eu.typeform.com`. - **Fields:** `response_id`, `landing_id`, `token`, `landed_at`, `submitted_at`, plus `answers` and `hidden` as raw JSON strings. Answer shapes differ by question type, so v1 does not try to flatten them. I still need to confirm during implementation that the base reader can emit a nested object as a JSON string; if it cannot, I will say so here before changing the field list. ## 2. Authentication - **v1: personal access token only**, through a required `api_key` option. OAuth 2.0 is deferred. - The token needs the `responses:read` scope (listed on Typeform's scopes page). It goes in the `Authorization` header, which the connector builds itself, so users supply the raw token. Typeform's token guide says to pass the token in the `Authorization` header but does not show the prefix. I plan to send `Bearer <token>` and will confirm it against a free Typeform account before opening the PR. ## 3. Secret boundary - **Where it lives:** only in the `api_key` option, a plain string option like the other HTTP connectors. I found no sensitive-option marker in them and am not introducing one; I will follow a maintainer-preferred convention if there is one. - **On the wire:** copied into the `Authorization` header and nowhere else. Never put in `params` or `body`. - **Logs and errors:** the connector logs only status codes and attempt counts, never headers, the config, or the parameter object. Failures use the base reader's `REQUEST_FAILED` message (status code and response body, no request headers). - **Job state:** the connector persists nothing, so the token cannot reach checkpoint state. - **Not covered:** whether the engine masks `api_key` when it logs the job config is outside this connector, and I have not verified it. ## 4. Pagination, incremental sync, and delivery semantics - **Pagination:** uses the base reader's existing `CURSOR` pagination, with no change to the shared client. Typeform pages with `before` (the reference page describes `before` and `after` as exclusive cursors). Default sort is newest first, so `before=<token of the last item on the page>` returns the next older page. The base reads the next cursor from the response with a JSONPath and uses the first value returned, so I will set the cursor field to `before` and the response path to `$.items[-1:].token`. I checked this path offline: it returns the last item's token, and an empty list on an empty page, which ends the read. - **Not confirmed from Typeform's docs:** which item field is the cursor value (I plan `token`, which each response carries), and the maximum `page_size`. The reference page names neither. I will verify both against a real account and state the result in the PR. `page_size` will default to 100. - **Incremental contract (v1):** a bounded snapshot of one form's responses, optionally narrowed by `since` and `until`. No incremental state is stored, so nothing carries over between runs. The cursor lives in memory for one run. - **Checkpoint and resume:** single split, nothing persisted. After a failure the read restarts from the first page. - **Duplicates:** possible only on restart, where rows already sent can be re-emitted. Because paging runs newest to oldest with a `before` cursor, responses submitted during the read arrive at the head and do not shift the pages being read. Every row carries `response_id`, so sinks can deduplicate or upsert. No exactly-once claim. - **Freshness:** Typeform warns that responses from roughly the last 30 minutes may not be returned. I will document this in the connector docs. - **Mode:** bounded only. Streaming mode is rejected. ## 5. Rate limits and retries - Typeform allows 2 requests per second per account for the Responses API. At 100 rows per page, a form with 10,000 responses takes about 100 requests, so roughly 50 seconds at the limit. - The base client retries only `IOException`s, so a `429` or `5xx` would currently fail the job. Like the Asana connector, the Typeform reader would override `executeRequest()` to also retry `429` and `5xx` with bounded exponential backoff, reusing the existing retry options. After the cap is reached the job fails with a clear error. - `Retry-After` is not supported in v1, because `HttpResponse` does not expose headers. That can be a separate small PR to the shared client if maintainers want it. ## 6. Memory behaviour Pages are fetched and emitted one at a time, each bounded by `page_size`, so the full result set is never held in memory. ## 7. Dependencies, license, and shading - **No new third-party runtime dependency.** The connector reuses `connector-http-base` and what it already carries. - **Test scope only:** `okhttp3:mockwebserver`, the same test dependency the Asana and Splunk modules use, with its version managed by the parent pom. ## 8. Deterministic testing (no live credentials) - **Unit tests:** `MockWebServer` to check request construction, multi-page cursor paging (including the empty last page), field mapping, `since`/`until` pass-through, the auth header, and retry on `429` and `5xx`. - **E2E:** a SeaTunnel job against a mock HTTP server that serves canned Typeform-shaped JSON (several pages and an injected `429`), so it needs no token and no network access. ## 9. Deliverables New module and parent-pom registration; `plugin-mapping.properties` entry (`seatunnel.source.Typeform = connector-http-typeform`); `plugin_config` and `seatunnel-dist/pom.xml` updates; factory and plugin name consistency (`getPluginName()` and `factoryIdentifier()` both `"Typeform"`); English and Chinese docs; unit tests; E2E test. ## Out of scope for v1 OAuth 2.0; webhooks; the forms, workspaces and Create API endpoints; flattening `answers`; persisted incremental state; streaming mode; `Retry-After` support. ### Usage Scenario Teams that collect survey, lead or feedback data in Typeform want to load the responses into a warehouse or database (for example Snowflake, ClickHouse or MySQL) for analysis, instead of exporting CSV files by hand. ### Related issues Tracking issue #10753 (Typeform row). Same pattern as the Asana connector (#12712, #12719). ### Are you willing to submit a PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
