smaheshwar-pltr opened a new issue, #1089: URL: https://github.com/apache/iceberg-cpp/issues/1089
`HttpClient` sends every request through one `cpr::ConnectionPool` ([http_client.cc#L201](https://github.com/apache/iceberg-cpp/blob/9c478f7571e89ef6388c4fa776d7a4e5f448cbac/src/iceberg/catalog/rest/http_client.cc#L201)). That pool wraps a libcurl share handle that shares the connection cache, and [libcurl doesn't support](https://curl.se/libcurl/c/CURLSHOPT_SHARE.html) using a shared connection cache from several threads at once. When two requests overlap, the process can crash. A `RestCatalog` sends all its requests through one `HttpClient`, so requests can overlap even when the application only calls the catalog from one thread: - Metrics reports are posted from the executor passed to `RestCatalog::Make` ([rest_metrics_reporter.cc#L73-L74](https://github.com/apache/iceberg-cpp/blob/9c478f7571e89ef6388c4fa776d7a4e5f448cbac/src/iceberg/catalog/rest/rest_metrics_reporter.cc#L73-L74)). - OAuth2 tokens are refreshed on the `auth_executor` passed to `RestCatalog::Make` ([auth_session_internal.h#L284](https://github.com/apache/iceberg-cpp/blob/9c478f7571e89ef6388c4fa776d7a4e5f448cbac/src/iceberg/catalog/rest/auth/auth_session_internal.h#L284)). - Vended S3 credentials are refreshed through LoadCredentials when the AWS SDK signs a request. The refresh runs on whichever thread is doing the S3 I/O ([arrow_s3_file_io.cc#L701-L708](https://github.com/apache/iceberg-cpp/blob/9c478f7571e89ef6388c4fa776d7a4e5f448cbac/src/iceberg/arrow/s3/arrow_s3_file_io.cc#L701-L708), [rest_catalog.cc#L91](https://github.com/apache/iceberg-cpp/blob/9c478f7571e89ef6388c4fa776d7a4e5f448cbac/src/iceberg/catalog/rest/rest_catalog.cc#L91)). Applications that share one catalog between threads run into it directly. The shared pool was introduced in apache/iceberg-cpp#530, which replaced a mutex-guarded `cpr::Session`. ### Reproduction Run this against the REST fixture that `rest_catalog_integration_test` uses: ```cpp HttpClient client; auto session = auth::AuthSession::MakeDefault({}); std::vector<std::thread> threads; for (int i = 0; i < 32; ++i) { threads.emplace_back([&] { for (int j = 0; j < 200; ++j) { (void)client.Get("http://localhost:8181/v1/config";, {}, {}, *DefaultErrorHandler::Instance(), *session); } }); } for (auto& thread : threads) thread.join(); ``` With libcurl 8.22.0, this aborted in 6 of 6 runs with heap corruption (`free(): corrupted unsorted chunks` or `double free or corruption`). Sharing one `RestCatalog` between 32 threads that each call `LoadTable` also crashes, with both 8.22.0 and 8.5.0. Neither reproduces with 8.18.0, the version CI uses, so CI doesn't catch it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
