NixonWahome opened a new pull request, #10938:
URL: https://github.com/apache/seatunnel/pull/10938
Closes #10356
### Purpose of this pull request
Adds a new `Salesforce` source connector for SeaTunnel V2, scoped to the V1
shape suggested in the issue:
- **Batch source** (`Boundedness.BOUNDED`, single-split) — no streaming /
PushTopic / CDC in this PR.
- **Bulk API 2.0** for reads — creates a query job at
`/services/data/<v>/jobs/query`, polls until `JobComplete`, streams CSV
results, and follows the `Sforce-Locator` header for pagination. No
REST/SOAP fallback path.
- **Multi-table aligned with the existing `tables_configs` model** — uses
`ConnectorCommonOptions.TABLE_CONFIGS`; each entry takes a
`table_path: database.ObjectName` and an optional per-table `filter` /
`soql_query`. Single-object mode via `object_name` is also supported and
mutually exclusive with `tables_configs`.
Auth is OAuth2 Username-Password (`client_id`, `client_secret`, `username`,
`password`, optional `security_token`). Schemas are discovered via
`/describe` and mapped to SeaTunnel `SeaTunnelDataType`s (id/string → STRING,
boolean → BOOLEAN, double/currency/percent → DOUBLE, date → DATE,
datetime → TIMESTAMP, time → TIME, base64 → BYTES). `SupportColumnProjection`
is declared.
**Explicitly out of scope** (deferred to follow-ups so this first review
stays
focused):
- No SOQL pushdown beyond `soql_query` / `filter` options
- No streaming / PushTopic / Salesforce CDC
- No incremental checkpointing — single split, full batch read
- No sink
### Does this PR introduce _any_ user-facing change?
Yes — new connector. Documentation:
`docs/en/connector-v2/source/Salesforce.md`.
### How was this patch tested?
Two test classes in the connector module:
**`SalesforceSourceFactoryTest`** — factory + option-rule coverage:
- Required auth options enforced (`client_id`, `client_secret`, `username`,
`instance_url`)
- Single-object mode valid with and without optional params
- Single-object mode with `soql_query` override
- Multi-object mode via `tables_configs`
- `OptionValidationException` thrown when both `object_name` and
`tables_configs` are set, or when neither is set
**`SalesforceClientTest`** — client logic against an embedded
`com.sun.net.httpserver` mocking the Salesforce REST + Bulk endpoints:
- Auth success populates `access_token` + `instance_url`
- Auth failure (401) raises `SalesforceConnectorException`
- `/describe` JSON maps Salesforce types to the expected SeaTunnel `SqlType`s
- Bulk query happy path parses CSV results into `Object[]` rows
- Bulk query follows `Sforce-Locator` across multiple result pages
- Bulk query throws when the job ends in `Failed` state
No Docker / mockserver container — the embedded HTTP server runs in-process
and the suite completes in a few seconds.
**E2E module not included in this PR.** Salesforce E2E needs live org
credentials (Bulk API 2.0 has no public sandbox we can wire into the CI
matrix). Happy to follow up with a mockserver-based E2E module in a separate
PR if reviewers prefer that pattern over the in-process tests above.
### Check list
* [x] No new Jar binary packages introduced (reuses existing
`commons-csv`, `httpclient`, shaded Jackson).
* [x] Documentation added: `docs/en/connector-v2/source/Salesforce.md`.
* [x] No incompatible changes.
* [x] Connector registration:
1. [x] `plugin-mapping.properties` —
`seatunnel.source.Salesforce = connector-salesforce`
2. [x] `seatunnel-dist/pom.xml` — `<scope>provided</scope>` dependency
3. [x] `.github/workflows/labeler/label-scope-conf.yml` — salesforce
label block added
4. [ ] E2E module — see "How was this patch tested?" above
5. [x] `config/plugin_config` — `connector-salesforce` listed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
