chovy-3012 opened a new pull request, #13088: URL: https://github.com/apache/seatunnel/pull/13088
Yes, I am contributing to SeaTunnel. ## Purpose Resolves #12723 The existing `connector-bigquery` module only supports writing **to** BigQuery (sink). This PR adds a BigQuery **Source** connector so users can read data out of BigQuery into downstream systems (Iceberg, ClickHouse, MySQL, etc.) without custom scripts or CSV exports. ## Changes **Source implementation (6 new classes):** - `BigQuerySourceFactory` — config validation, query resolution (custom SQL or table-read mode), schema inference - `BigQuerySourceReader` — query execution, type conversion (BigQuery → SeaTunnel), fetch_size batching, dry_run support - `BigQuerySourceSplitEnumerator` — single-split strategy, checkpoint/restore - `BigQuerySource`, `BigQuerySourceSplit`, `BigQuerySourceState` — source entry point and state types **Option refactoring:** - Extracted 7 shared options (project_id, dataset_id, table_id, credentials, emulator, universe_domain) into `BigQueryBaseOptions extends ConnectorCommonOptions`, following the AmazonSQS/Cassandra pattern. Source and Sink option classes now extend the base. Zero changes in consumer code via Java static field inheritance. **Type converter:** - Added reverse mapping (BigQuery → SeaTunnel) in `BigQueryTypeConverter`: BOOL, INT64, FLOAT64, NUMERIC/BIGNUMERIC, STRING, BYTES, DATE, TIME, TIMESTAMP, DATETIME, STRUCT (nested rows), REPEATED (arrays) **15 source options:** project_id (required), query, dataset_id, table_id, max_results, fetch_size, location, use_query_cache, dry_run, maximum_bytes_billed, partition_filter, emulator_host, service_account_key_path, service_account_key_json, universe_domain **Validation rules:** - `query` and `dataset_id`+`table_id` are mutually exclusive - `partition_filter` only valid with table-read mode; rejected with `query` - `requirePartitionFilter` tables fail fast if no `partition_filter` provided **Tests:** 221 unit tests (54 new source/converter tests). Extended existing `BigqueryIT` E2E suite. **Docs:** EN/ZH documentation with option table, authentication modes, query modes, lifecycle/delivery semantics, and 4 config examples. ## Backward Compatibility No option keys, types, defaults, or descriptions changed. No public APIs changed. No new dependencies introduced — uses existing GCP SDK and shading. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
