SEZ9 commented on issue #12723: URL: https://github.com/apache/seatunnel/issues/12723#issuecomment-6432176134
Thanks @merlau — all three points are helpful, and they map cleanly onto what was already agreed in this thread. **1. Bounded-batch only, no CDC.** Agreed and already the plan. The source is strictly bounded-batch with at-least-once delivery, and that will be stated explicitly in the PR description and in the EN/ZH docs. I will add a short note in the docs pointing users who need streaming change capture out of BigQuery to the Continuous Queries → Pub/Sub → existing `GooglePubSub` Source path, so the scope boundary is clear from day one. **2. Partition pruning in `dataset_id` + `table_id` mode.** Fair point — without a filter, direct table reads would scan the whole table even with `maximum_bytes_billed` / `dry_run` in place. This is not implemented yet. What I intend to do: - Add an optional `partition_filter` option that is only valid in `dataset_id` + `table_id` mode (rejected with a clear message if combined with `query`, consistent with the exactly-one-read-mode rule already agreed). - Check table metadata for `requirePartitionFilter` and fail fast at validation time, before any job is submitted, if the flag is set and no `partition_filter` is supplied. - Document the exact validation error messages alongside the existing read-mode validation message in the PR description. **3. Single split.** Yes — the first PR will be a single-split reader built on REST query pagination (`max_results`). A Storage Read API–based split enumerator is out of scope for this PR; it would also pull in an additional dependency, and we have committed to no new third-party dependency here. I'll note it in the docs as a possible future enhancement rather than hint at parallelism the connector doesn't provide. Remaining items on my side: implement `partition_filter` + `requirePartitionFilter` validation with tests, add the CDC-scope and single-split notes to the EN/ZH docs, and link the one focused PR here once it is ready. If you have a preferred name between `partition_filter` and `row_restriction`, let me know before I lock the option name. <!-- streview-comment:2062 --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
