smaheshwar-pltr opened a new issue, #1091: URL: https://github.com/apache/iceberg-cpp/issues/1091
Currently, `RestCatalog` doesn't document whether one instance can be shared between threads. Query engines usually share one catalog across concurrent queries (for example [Spark](https://github.com/apache/iceberg/blob/f0e7de3c2f207a49ac3a241ac7346726c08c8f1b/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java#L728-L777) within a session, [Trino](https://github.com/trinodb/trino/blob/83129f579dd499561c4cf04e1ba9f1f3e6f74687/plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/rest/TrinoIcebergRestCatalogFactory.java#L110-L151), and [DuckDB](https://github.com/duckdb/duckdb-iceberg/blob/8b0909cc64a1616da32f9146cc820478ff1df895/src/iceberg_attach.cpp#L371-L381)), and iceberg-rust requires catalogs to be `Send + Sync` ([code](https://github.com/apache/iceberg-rust/blob/e03179f2f5a18c04e7f4ab0c39e3a73c7cf290bf/crates/iceberg/src/catalog/mod.rs#L55-L58)). Without that guarantee, an embedder has to serialize every catalog call behind its own lock. That costs throughput, and one hung request then blocks every other caller. In https://github.com/smaheshwar-pltr/duckdb-proto-iceberg, we observed that making a catalog-heavy test about 4x slower. `RestCatalog` is close to thread-safe already. After `Make()`, the only state it changes is the default catalog that `AsCatalog()` caches in a `weak_ptr`. It reads and writes that cache without a lock, so concurrent `AsCatalog()` calls race ([rest_catalog.cc#L565-L573](https://github.com/apache/iceberg-cpp/blob/72fbb5aef6cc5950107af385fca76b5bef9978aa/src/iceberg/catalog/rest/rest_catalog.cc#L565-L573)). **Note:** sharing a catalog also depends on https://github.com/apache/iceberg-cpp/issues/1089, because every request goes through one `HttpClient`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
