merlau commented on issue #12723: URL: https://github.com/apache/seatunnel/issues/12723#issuecomment-6416943101
Thanks for working on this @chovy-3012! Having a native BigQuery Source connector in SeaTunnel is a great addition. I wanted to share two design thoughts from a GCP data architecture and cost perspective: #### 1. Scope: Explicitly Keep This Strictly Bounded-Batch (No CDC) Keeping the scope strictly to bounded-batch is 100% the right decision. * BigQuery does not expose a transaction log / WAL / binlog socket for streaming CDC egress. * Attempting CDC natively inside the connector would require polling table functions (`CHANGES()`) into intermediate tables, creating high slot contention, API quota exhaustion, and state-reconciliation complexity. * For users needing streaming CDC out of BigQuery, Google's recommended architecture is **BigQuery Continuous Queries $\rightarrow$ Cloud Pub/Sub $\rightarrow$ SeaTunnel's existing `GooglePubSub` Source**. Clarifying in the documentation that streaming CDC is handled via Pub/Sub will prevent scope creep and future feature requests attempting to shoehorn streaming CDC into this source. #### 2. Partitioning & Cost Guardrails in `dataset_id + table_id` Mode BigQuery on-demand scanning costs ($6.25/TB) can escalate quickly if users read tables without partition filters. * It is great that `maximum_bytes_billed` and `dry_run` are already included in the proposal—kudos for having cost safety in mind! * For the `dataset_id + table_id` mode, how will partition pruning be supported? If a user selects a table without writing custom SQL, they will inadvertently perform a full table scan. * **Suggestion:** Consider adding an optional `partition_filter` (or `row_restriction`) parameter (e.g., `_PARTITIONDATE >= '2026-01-01'`) for direct table reads. Additionally, if the table has `requirePartitionFilter = true` enabled in BigQuery metadata, the factory should validate that a partition filter is provided before submitting the job. #### 3. Note on Splits / Parallelism (re: @SEZ9's point #2) Since this implementation uses the GCP SDK / REST query pagination (`max_results`), the job will inherently be a single-split reader. * For a V1 MVP, a single split is clean, straightforward, and avoids concurrency issues. * For future high-throughput / multi-split requirements, SeaTunnel could eventually introduce a split enumerator backed by the BigQuery Storage Read API (`ReadSession`), but for standard query and table exports, this REST-based batch approach is a solid baseline. Looking forward to the PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
