laskoviymishka opened a new issue, #2062: URL: https://github.com/apache/iceberg-go/issues/2062
### Background We are heading toward a v1.0.0 release of iceberg-go. Under Go semver, once we tag 1.0.0 any breaking change to the public API means a new `/v2` module path, so the pre-1.0 window is the one cheap chance to fix API and layering warts. This is an umbrella issue to collect those breaking cleanups in one place so we can agree on scope and sequence them, rather than discovering them right after the freeze. Two efforts already in flight roll up under this: - #2044 covers decoupling the metadata and REST client from the Arrow execution stack, plus an in-process REST testkit. - #696 covers shrinking the dependency tree for consumers that do not need every backend (the IO / cloud-SDK side, via the driver pattern in #980 and the per-cloud split in #1874 / #1961). The intent is to keep things additive or behind facades where we can, and be explicit about the few genuinely breaking changes that have to land before the tag. Measurements and the detailed Arrow analysis live in #2044. ### 1. Dependency footprint and layering Goal: a consumer that only reads/writes metadata or talks to a REST catalog should not link the Arrow execution stack or a cloud SDK it never uses. - [ ] Isolate the Arrow / Substrait / compute code in `table` behind engine-facing sub-packages, keeping the public `Transaction` / `Scan` types in place as facades (#2044). - [ ] Drop the `pterm` terminal-UI dependency from `table` so `table` and `catalog/rest` build for `GOOS=js GOARCH=wasm` (#2044). - [ ] Make AWS SigV4 signing an optional backend so `catalog/rest` does not link the AWS SDK unless signing is used. This removes `rest.WithAwsConfig(aws.Config)` from the public option set (breaking). - [ ] Decide whether the root Literal API stays Arrow-backed. `VariantLiteral` and `DecimalLiteral` currently expose `parquet/variant` and `arrow/decimal128` types, so a fully Arrow-free metadata core requires a breaking Literal-API change, which is pre-1.0 or `/v2` only. Details in #2044. - [ ] Continue the backend split from #696: separate the heavy catalog backends (`glue`, `sql`, `hive`) and the io backends into their own modules so the minimal consumer's module graph shrinks. `glue.WithAwsConfig(aws.Config)` has the same AWS-type leak as the REST option above. ### 2. Public API consistency (breaking, so pre-1.0 or `/v2` only) - [ ] `io.IO`: `Open` and `Remove` take no `context.Context`, while `BulkRemovableIO.DeleteFiles` does. Thread ctx through the base interface for a consistent, cancellable IO surface. - [ ] Catalog capabilities: view operations and `RegisterTable` are not part of the core `Catalog` interface and are supported unevenly across the REST / SQL / Hive / Glue / Hadoop implementations. Decide whether these become optional capability interfaces (e.g. a `ViewableCatalog`) so support is discoverable and uniform. - [ ] table write API: the string-path and DataFile method pairs overlap (`AddFiles` / `AddDataFiles`, `ReplaceDataFiles` / `ReplaceDataFilesWithDataFiles`, `ReplaceFiles` / `ReplaceFilesWithDeleteFiles`). Settle on canonical names before we freeze all of them. - [ ] `PartitionField.SourceID()` reads ambiguously next to the `SourceIDs` field now that a field can have multiple source ids. Rename or clarify. - [ ] `Schema.FindFieldByIDRef` (and `FieldsRef`) take and expose `internal.SchemaRef`, leaking an internal type through the public API. Hide it. ### 3. Deprecations to remove at the freeze The freeze is the natural point to actually delete the accumulated `Deprecated:` symbols rather than carry them into a stable API: - [ ] `Type.ToHumanStr` (superseded by `ToHumanStrType`). - [ ] `DataFileBuilder.DistinctValueCounts` (distinct_counts / field 111 is deprecated in the spec). - [ ] `io/gocloud` `ParseAWSConfig` / `ParseGCSConfig` redirect shims (superseded by the per-cloud `s3.ParseAWSConfig` / `gcs.ParseGCSConfig`). - [ ] the remaining redirect helpers (the `WithMaxConcurrency` and `RewriteFiles.ApplyResult` shims). ### Explicitly out of scope for v1 To keep the freeze achievable, I would not block v1 on these: - The full `table` package decomposition in #1149. It is maintainer ergonomics rather than API semantics, and moving exported symbols across packages is breaking, so it fits a later `/v2` better than the 1.0 window. - The Arrow dependency on the scan path (`Scan.ToArrowRecords` / `ReadTasks` return `arrow.RecordBatch`). That coupling is real and useful, so we document the pin rather than remove it. ### Ask Mainly want to align on scope and ordering. My suggested sequence: the additive testkit from #2044 first, then the non-breaking isolation edges (pterm, engine-facing sub-packages), then the breaking option / interface changes grouped so we tag once. I already have working prototypes for the pterm and SigV4 items and can open them as draft PRs against this. Happy to split any box out into its own issue. cc @zeroshade @nssalian @slachiewicz -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
