jk-shah opened a new issue, #41226:
URL: https://github.com/apache/beam/issues/41226
### Description
In continuous unbounded streaming pipelines (e.g. Kafka or Pub/Sub to Apache
Iceberg), `beam.managed.Write(beam.managed.ICEBERG)` needs predictable
micro-batch commit intervals to commit table snapshots and advance commit
watermarks against REST catalogs (Google Cloud BigLake REST Catalog, Apache
Polaris, AWS Glue).
### Proposed Architecture
Expose declarative streaming commit configuration options on
`beam.managed.Write(beam.managed.ICEBERG)`:
```python
beam.managed.Write(
beam.managed.ICEBERG,
config={
"table": "my_catalog.db.table",
"warehouse": "gs://my-bucket/wh",
"trigger_interval": "60s", # snapshot commit cadence
"checkpoint_interval": "30s", # runner checkpoint alignment
"mode": "append" # or merge-on-read
}
)
```
### Key Capabilities
1. **Coordinated Commits**: Aligns micro-batch file writes with periodic
atomic snapshot commits.
2. **REST Catalog Integration**: Automatic OAuth credential vending and
token refresh for BigLake and Polaris REST catalogs.
3. **Dataflow Streaming Engine Alignment**: Seamless execution with Dataflow
Windmill streaming checkpoints.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
