laxmanmamididd commented on PR #7363: URL: https://github.com/apache/datafusion-comet/pull/7363#issuecomment-6957991556
Thanks, @andygrove. Pushed 3978fa5a8, on top of #7464's latest commit (9f36bb3f5) and merged with `main` again (d1b61e258). **Expiry in the table config.** Polaris sends `s3.session-token-expires-at-ms` with `client.refresh-credentials-endpoint` ([docs](https://polaris.apache.org/releases/1.7.0/vended-credentials/)). The Unity Catalog deployment I checked also includes the expiry in `loadTable`'s config, but it sends no refresh endpoint, so this provider isn't selected for it. A catalog could still rotate the `s3.*` keys without an expiry, so 3978fa5a8 adds a note to the provider section of `s3-credential-providers.md`: each instance holds an HTTP client, and without a later expiry each reload keeps another instance until the 64-entry bound evicts it. **MinIO run.** I ran it once on Spark 4.0 and JDK 17 (harness below, not committed): - MinIO STS minted the credentials with its shortest lifetime, 15 minutes. A REST catalog vended them along with `s3.session-token-expires-at-ms`, `client.refresh-credentials-endpoint` and `uri`. A local `/credentials` endpoint minted new STS credentials on each call and counted the calls. - Two catalogs pointed at that REST catalog: `vended`, which picked up the provider automatically, and `control`, set to `s3.comet.credential.provider.class=""`. The test wrote 400,000 rows in 8 files, planned a native scan through each catalog, then waited until a minute after the first credentials expired. - `vended` read all of it: `count = 400000`, `sum(id) = 79999800000`. The refresh endpoint got no calls during planning and one during the read. - `control` failed with `PermissionDenied`, from S3 `InvalidAccessKeyId` on the data files. MinIO deletes expired STS keys, so it reports this instead of `ExpiredToken`. My first assertion only accepted "expired", so the run reported FAILED on that check after both scans had behaved as above. The harness below accepts both errors. Also run after the merge: the provider and dispatcher JUnit tests (34 passed), `CometIcebergNativeScanSuite`, `CometIcebergNativeSuite`, `CometS3CredentialReleaseSuite` and `CometS3CredentialBridgeSuite` (187 passed, 7 canceled by version gates), and the `iceberg_common` Rust tests (31 passed). <details> <summary>VendedRefreshManualSuite.scala</summary> ```scala package org.apache.comet.cloud.s3 import java.net.{InetSocketAddress, URI} import java.nio.charset.StandardCharsets import java.util.concurrent.atomic.AtomicInteger import org.apache.spark.sql.comet.CometIcebergNativeScanExec import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper import com.sun.net.httpserver.HttpServer import org.apache.comet.{CometConf, CometS3TestBase} import org.apache.comet.iceberg.RESTCatalogHelper import software.amazon.awssdk.auth.credentials.{AwsBasicCredentials, StaticCredentialsProvider} import software.amazon.awssdk.regions.Region import software.amazon.awssdk.services.sts.StsClient import software.amazon.awssdk.services.sts.model.{AssumeRoleRequest, Credentials} /** * One-off manual check, not for CI: a native Iceberg scan through a REST catalog that vends * short-lived MinIO STS credentials with a refresh endpoint keeps reading after the first * credentials expire. A control catalog that opts out of the provider must fail instead. Takes * about 16 minutes, since MinIO's shortest STS lifetime is 15. */ class VendedRefreshManualSuite extends CometS3TestBase with AdaptiveSparkPlanHelper with RESTCatalogHelper { override protected val testBucketName = "vended-refresh-bucket" private val durationSeconds = 900 private def mint(): Credentials = { val sts = StsClient .builder() .endpointOverride(URI.create(minioContainer.getS3URL)) .region(Region.US_EAST_1) .credentialsProvider( StaticCredentialsProvider.create(AwsBasicCredentials.create(userName, password))) .build() try { sts .assumeRole( AssumeRoleRequest .builder() .roleArn("arn:aws:iam::000000000000:role/comet-vended-refresh") .roleSessionName("comet-vended-refresh") .durationSeconds(durationSeconds) .build()) .credentials() } finally { sts.close() } } private def s3Config(c: Credentials): Map[String, String] = Map( "s3.access-key-id" -> c.accessKeyId(), "s3.secret-access-key" -> c.secretAccessKey(), "s3.session-token" -> c.sessionToken(), "s3.session-token-expires-at-ms" -> c.expiration().toEpochMilli.toString) private def json(m: Map[String, String]): String = m.map { case (k, v) => s""""$k":"${v.replace("\"", "\\\"")}"""" }.mkString("{", ",", "}") test("a native scan keeps reading after the first vended credentials expire") { assume(icebergAvailable, "Iceberg not available in classpath") // The refresh endpoint: each call mints new STS credentials. val refreshCalls = new AtomicInteger() val server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0) server.createContext( "/credentials", exchange => { refreshCalls.incrementAndGet() val body = s"""{"storage-credentials":[{"prefix":"s3","config":${json(s3Config(mint()))}}]}""" .getBytes(StandardCharsets.UTF_8) exchange.getResponseHeaders.add("Content-Type", "application/json") exchange.sendResponseHeaders(200, body.length) exchange.getResponseBody.write(body) exchange.close() }) server.start() val refreshEndpoint = s"http://127.0.0.1:${server.getAddress.getPort}/credentials" val first = mint() val firstExpiry = first.expiration().toEpochMilli val vended = s3Config(first) ++ Map( "s3.endpoint" -> minioContainer.getS3URL, "s3.path-style-access" -> "true", "client.region" -> "us-east-1", "client.refresh-credentials-endpoint" -> refreshEndpoint) val warehouse = s"s3a://$testBucketName/warehouse-vended-refresh" try { withRESTCatalog(vendedCredentials = vended, warehouseLocation = Some(warehouse)) { (restUri, _, _) => def catalog(name: String): Seq[(String, String)] = Seq( s"spark.sql.catalog.$name" -> "org.apache.iceberg.spark.SparkCatalog", s"spark.sql.catalog.$name.catalog-impl" -> "org.apache.iceberg.rest.RESTCatalog", s"spark.sql.catalog.$name.uri" -> restUri, s"spark.sql.catalog.$name.warehouse" -> warehouse) withSQLConf( (catalog("vended") ++ catalog("control") ++ Seq( // The control opts out, so its native scan signs with the first credentials. "spark.sql.catalog.control.s3.comet.credential.provider.class" -> "", CometConf.COMET_ICEBERG_NATIVE_ENABLED.key -> "true", "spark.sql.adaptive.enabled" -> "false")): _*) { spark.sql("CREATE NAMESPACE vended.db") spark.sql("CREATE TABLE vended.db.t (id BIGINT, s STRING) USING iceberg") // Several files, so the scan makes many requests. (0 until 8).foreach { i => spark.sql( s"INSERT INTO vended.db.t SELECT id, repeat('x', 100) " + s"FROM range(${i * 50000}, ${(i + 1) * 50000})") } // Plan both scans now, so their FileIO properties carry the first credentials. val vendedDf = spark.sql("SELECT count(*), sum(id) FROM vended.db.t") val controlDf = spark.sql("SELECT count(*), sum(id) FROM control.db.t") Seq(vendedDf, controlDf).foreach { df => assert( collect(df.queryExecution.executedPlan) { case s: CometIcebergNativeScanExec => s }.nonEmpty, s"expected a native Iceberg scan:\n${df.queryExecution.executedPlan}") } val callsAtPlan = refreshCalls.get() val waitMs = firstExpiry - System.currentTimeMillis() + 60000L logInfo(s"Waiting ${waitMs / 1000}s for the first credentials to expire") Thread.sleep(math.max(waitMs, 0L)) assert(System.currentTimeMillis() > firstExpiry) val rows = vendedDf.collect() val n = 400000L assert(rows.head.getLong(0) == n) assert(rows.head.getLong(1) == n * (n - 1) / 2) val callsAfterRead = refreshCalls.get() assert(callsAfterRead > callsAtPlan, "the provider never called the refresh endpoint") val err = intercept[Exception](controlDf.collect()) val msg = Iterator .iterate[Throwable](err)(_.getCause) .takeWhile(_ != null) .map(_.getMessage) .mkString(" | ") logInfo(s"Control failed as expected: $msg") println( s"RESULT vended rows=${rows.head} refreshCalls plan=$callsAtPlan " + s"afterRead=$callsAfterRead; control error: ${msg.take(600)}") // MinIO drops expired STS keys, so it answers InvalidAccessKeyId, not ExpiredToken. assert( Seq("ExpiredToken", "InvalidAccessKeyId", "PermissionDenied").exists(msg.contains), s"control failed for another reason: $msg") } } } finally { server.stop(0) } } } ``` </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
