Rishavbhattarai commented on issue #12822: URL: https://github.com/apache/seatunnel/issues/12822#issuecomment-6224875159
Thanks @DanielLeens. I checked the open points against Typeform's docs and against the current HTTP base reader. **API facts** - **Authorization:** `Authorization: Bearer <token>`. Typeform's [hands-on guide](https://www.typeform.com/developers/get-started/hands-on/) and [Responses walkthrough](https://www.typeform.com/developers/responses/walkthroughs/) both show it in their curl examples, and their official JS client sends `bearer ${token}` (`src/create-client.ts`). - **Cursor:** `before`, set to the `token` of the last response on the page. The walkthrough says "The value for the `before` parameter should be the `token` value for the 25th response." The [reference](https://www.typeform.com/developers/responses/reference/retrieve-responses/) describes `before` as exclusive. Typeform's JS client pages with `response_id` instead, so the mock tests will give `token` and `response_id` different values to show which one the connector sends. - **Max page size:** 1000, default 25 (walkthrough: "Max. value is 1000"). The connector will default to 100 and reject values outside 1..1000. - Also confirmed: 2 requests per second per account, and the EU base URLs `https://api.eu.typeform.com` and `https://api.typeform.eu`. One change to the design: v1 will not expose `sort`. The `before` cursor depends on the default `submitted_at,desc` order, and a different sort would break paging. **`answers` and `hidden` through the base reader** I ran a throwaway test with the current `HttpSourceReader` and `JsonDeserializationSchema` against a mock server. It served three Typeform-shaped pages, with cursor `before` and response path `$.items[-1:].token`. All columns were STRING. - Paging needed no shared-client change. The requests went `?page_size=2`, then `?before=r2`, then `?before=r5`, and the empty third page ended the read. - `answers` with 12 answer types (text, boolean, email, number, date, choices, choice with `other`, file_url, payment, phone_number, a decimal and an integer above 2^53) came out byte-identical to the source JSON. Key order, escaped quotes and newlines, and unicode all survived. `hidden` kept its key order too, and two runs gave the same output. - An unknown answer type with nested arrays and nulls passed through unchanged. - A hidden value that is a JSON-looking string stays an escaped string. Edge cases: | Input | Column value | |---|---| | `"answers": []` | NULL | | `"hidden": {}` | `"{}"` | | `null` or key missing | NULL | The empty-array case comes from `JsonPathProcessorImpl.extractValue`, which returns null for an empty list. Since the connector PR should not touch shared code, I'll document that `answers` is NULL when a response has no answers and pin it in a unit test. Typeform's own reference example includes a response with `"answers": []`, so real data will hit this. I'll put these sources in the PR description as well. If the design looks settled, could you assign this issue to me? I'll start the PR after that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
