slachiewicz opened a new issue, #2044:
URL: https://github.com/apache/iceberg-go/issues/2044
### Summary
Currently, `github.com/apache/iceberg-go` is organized as a single
monolithic Go module. Importing any component—such as `catalog/rest` or core
table metadata structures—drags in the entire dependency tree:
- Physical execution / columnar engine: `apache/arrow-go/v18`,
`geoarrow-go`, `substrait-go`
- Storage SDKs: `aws-sdk-go-v2`, `azure-sdk-for-go`, `gocloud.dev`,
`google.golang.org/api`
- SQL / ORM drivers: `uptrace/bun` (Postgres, MySQL, Oracle, MSSQL, SQLite)
- Integration tooling: `docker/docker`, `testcontainers-go`
### Motivation & Pain Points
1. **Lightweight Utilities & Services**:
Projects that only interact with Iceberg at the metadata layer (e.g.
catalog proxies, REST server gateways, metadata translation bridges like Apache
XTable/polytable, or serverless functions) cannot consume `iceberg-go` without
pulling in tens of megabytes of binary overhead, CGO/SIMD routines, and
transitive dependency version locks.
2. **WASM / Edge Runtimes**:
Building for `GOOS=js GOARCH=wasm` or constrained edge runtimes fails due
to OS-specific syscalls, Docker, and database dependencies in the root module
graph.
3. **Integration Testing Overhead**:
The current REST catalog test suite relies on `dev/docker-compose.yml`
spinning up `tabulario/iceberg-rest` (JVM Java container) and MinIO. There is
currently no in-process, pure-Go test server/mock, making local test iteration
and container-less CI runs challenging.
### Precedent: Apache Iceberg (Java)
In the Java repository, this separation is already well-established:
- `iceberg-api`: Schemas, types, expressions, partition specs. Zero
execution/cloud dependencies.
- `iceberg-core`: Manifest Avro codecs, table metadata JSON, commit
operations.
- `iceberg-arrow` / `iceberg-spark`: Vectorized readers and query engine
adapters.
### Proposal
1. **Modularize the Go Module Structure**:
- Separate core specification models (Schema, Types, PartitionSpec,
TableMetadata, and Avro manifest codecs) and `catalog/rest` into lightweight
packages/submodules with minimal dependencies (standard library + pure-Go
JSON/Avro).
- Keep physical columnar execution (`arrow-go`, `substrait-go`) in
dedicated engine packages.
2. **In-Process REST Mock / Testkit**:
- Provide an in-memory/in-process Go REST catalog server for tests,
enabling sub-millisecond, container-less unit testing of the REST catalog
client.
### Next Steps & Contribution
We would be happy to propose and contribute incremental refactoring PRs for
this:
1. First, an in-process REST catalog testkit / mock to simplify
`catalog/rest` testing without Docker.
2. An incremental separation of metadata/REST models to reduce dependency
coupling.
Would the community and maintainers be open to discussing this direction?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
