countradooku opened a new issue, #3469: URL: https://github.com/apache/iggy/issues/3469
### Description Add a Databricks connector for Apache Iggy. Recommendation: scope the first connector as a **sink**, not a combined sink + source. Databricks is primarily a lakehouse/warehouse destination for streaming and event pipelines, and Iggy's connector model maps that naturally to `Iggy stream -> sink -> Databricks table`. A Databricks source can be useful later via Delta Change Data Feed (`table_changes`) or SQL polling, but it has a different stateful contract: table-version cursors, chunk/page offsets, CDF retention windows, and schema-change behavior. Keeping the source separate should make implementation, review, and testing more tractable. Related context: - #2753 lists Databricks as a proposed data warehouse **sink** target. - #1852 tracks Delta Lake connectors, and `core/connectors/sinks/delta_sink` already covers direct Delta table/object-store writes. A Databricks connector should cover Databricks workspace, SQL warehouse, Unity Catalog, and managed-table ingestion flows that are not addressed by a raw Delta table URI alone. References: - Databricks SQL Statement Execution API: https://docs.databricks.com/aws/en/dev-tools/sql-execution-tutorial - Databricks `COPY INTO`: https://docs.databricks.com/aws/en/sql/language-manual/delta-copy-into - Databricks Change Data Feed for a possible future source connector: https://docs.databricks.com/aws/en/tables/features/change-data-feed ### Affected area / component Connectors ### Proposed solution Add `core/connectors/sinks/databricks_sink/` implementing `iggy_connector_sdk::Sink`. Suggested scope: - Config fields for workspace host, authentication token/OAuth/service-principal credentials, SQL warehouse ID, catalog/schema/table, ingestion mode, batch size, timeout/retry options, and optional staging location or Unity Catalog volume. - Store all credentials as `SecretString` and ensure serialized/debug output redacts secrets. - MVP ingestion path: batch Iggy messages and load them into a Databricks table through the Databricks SQL Statement Execution API against a SQL warehouse. - Preferred high-throughput path: stage JSON or Parquet files to cloud storage or a Unity Catalog volume, then run `COPY INTO` or `MERGE` through the Statement Execution API. This should fit retry/replay behavior better than per-row inserts because `COPY INTO` can skip already-loaded files. - Follow existing connector patterns from `delta_sink`, `iceberg_sink`, and `doris_sink` for batching, retryable vs permanent errors, logging, metrics, secret handling, docs, and example config. - Add focused tests for config validation, SQL/request generation, retry/error mapping, and docs/examples. Integration testing may need to be gated behind Databricks credentials or covered with a local HTTP mock first. Potential future source connector, tracked separately if maintainers agree: - Read Delta Change Data Feed via `table_changes()` or explicit SQL polling. - Persist Databricks table version and chunk/page offset in source state. - Document CDF retention and non-additive schema-change limitations. ### Alternatives considered - Implement both sink and source in one issue: rejected because source semantics are substantially different and should not block the higher-value ingestion sink. - Treat the existing Delta Lake sink as sufficient: rejected because Databricks-specific deployments often need workspace auth, SQL warehouses, Unity Catalog/Volumes, and managed table loading paths. - Use the generic HTTP sink: possible for raw SQL calls, but too low-level for table-aware batching, idempotency, schema handling, and connector-specific error mapping. ### Contribution - [ ] I'm willing to submit a pull request to implement this feature ### Good first issue - [ ] I think this could be a good first issue for a new contributor -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
