laxmanmamididd commented on PR #7363:
URL: 
https://github.com/apache/datafusion-comet/pull/7363#issuecomment-6957991556

   Thanks, @andygrove. Pushed 3978fa5a8, on top of #7464's latest commit 
(9f36bb3f5) and merged with `main` again (d1b61e258).
   
   **Expiry in the table config.** Polaris sends 
`s3.session-token-expires-at-ms` with `client.refresh-credentials-endpoint` 
([docs](https://polaris.apache.org/releases/1.7.0/vended-credentials/)). The 
Unity Catalog deployment I checked also includes the expiry in `loadTable`'s 
config, but it sends no refresh endpoint, so this provider isn't selected for 
it. A catalog could still rotate the `s3.*` keys without an expiry, so 
3978fa5a8 adds a note to the provider section of `s3-credential-providers.md`: 
each instance holds an HTTP client, and without a later expiry each reload 
keeps another instance until the 64-entry bound evicts it.
   
   **MinIO run.** I ran it once on Spark 4.0 and JDK 17 (harness below, not 
committed):
   
   - MinIO STS minted the credentials with its shortest lifetime, 15 minutes. A 
REST catalog vended them along with `s3.session-token-expires-at-ms`, 
`client.refresh-credentials-endpoint` and `uri`. A local `/credentials` 
endpoint minted new STS credentials on each call and counted the calls.
   - Two catalogs pointed at that REST catalog: `vended`, which picked up the 
provider automatically, and `control`, set to 
`s3.comet.credential.provider.class=""`. The test wrote 400,000 rows in 8 
files, planned a native scan through each catalog, then waited until a minute 
after the first credentials expired.
   - `vended` read all of it: `count = 400000`, `sum(id) = 79999800000`. The 
refresh endpoint got no calls during planning and one during the read.
   - `control` failed with `PermissionDenied`, from S3 `InvalidAccessKeyId` on 
the data files. MinIO deletes expired STS keys, so it reports this instead of 
`ExpiredToken`. My first assertion only accepted "expired", so the run reported 
FAILED on that check after both scans had behaved as above. The harness below 
accepts both errors.
   
   Also run after the merge: the provider and dispatcher JUnit tests (34 
passed), `CometIcebergNativeScanSuite`, `CometIcebergNativeSuite`, 
`CometS3CredentialReleaseSuite` and `CometS3CredentialBridgeSuite` (187 passed, 
7 canceled by version gates), and the `iceberg_common` Rust tests (31 passed).
   
   <details>
   <summary>VendedRefreshManualSuite.scala</summary>
   
   ```scala
   package org.apache.comet.cloud.s3
   
   import java.net.{InetSocketAddress, URI}
   import java.nio.charset.StandardCharsets
   import java.util.concurrent.atomic.AtomicInteger
   
   import org.apache.spark.sql.comet.CometIcebergNativeScanExec
   import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
   
   import com.sun.net.httpserver.HttpServer
   
   import org.apache.comet.{CometConf, CometS3TestBase}
   import org.apache.comet.iceberg.RESTCatalogHelper
   
   import software.amazon.awssdk.auth.credentials.{AwsBasicCredentials, 
StaticCredentialsProvider}
   import software.amazon.awssdk.regions.Region
   import software.amazon.awssdk.services.sts.StsClient
   import software.amazon.awssdk.services.sts.model.{AssumeRoleRequest, 
Credentials}
   
   /**
    * One-off manual check, not for CI: a native Iceberg scan through a REST 
catalog that vends
    * short-lived MinIO STS credentials with a refresh endpoint keeps reading 
after the first
    * credentials expire. A control catalog that opts out of the provider must 
fail instead. Takes
    * about 16 minutes, since MinIO's shortest STS lifetime is 15.
    */
   class VendedRefreshManualSuite
       extends CometS3TestBase
       with AdaptiveSparkPlanHelper
       with RESTCatalogHelper {
   
     override protected val testBucketName = "vended-refresh-bucket"
   
     private val durationSeconds = 900
   
     private def mint(): Credentials = {
       val sts = StsClient
         .builder()
         .endpointOverride(URI.create(minioContainer.getS3URL))
         .region(Region.US_EAST_1)
         .credentialsProvider(
           
StaticCredentialsProvider.create(AwsBasicCredentials.create(userName, 
password)))
         .build()
       try {
         sts
           .assumeRole(
             AssumeRoleRequest
               .builder()
               .roleArn("arn:aws:iam::000000000000:role/comet-vended-refresh")
               .roleSessionName("comet-vended-refresh")
               .durationSeconds(durationSeconds)
               .build())
           .credentials()
       } finally {
         sts.close()
       }
     }
   
     private def s3Config(c: Credentials): Map[String, String] = Map(
       "s3.access-key-id" -> c.accessKeyId(),
       "s3.secret-access-key" -> c.secretAccessKey(),
       "s3.session-token" -> c.sessionToken(),
       "s3.session-token-expires-at-ms" -> c.expiration().toEpochMilli.toString)
   
     private def json(m: Map[String, String]): String =
       m.map { case (k, v) => s""""$k":"${v.replace("\"", "\\\"")}"""" 
}.mkString("{", ",", "}")
   
     test("a native scan keeps reading after the first vended credentials 
expire") {
       assume(icebergAvailable, "Iceberg not available in classpath")
   
       // The refresh endpoint: each call mints new STS credentials.
       val refreshCalls = new AtomicInteger()
       val server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0)
       server.createContext(
         "/credentials",
         exchange => {
           refreshCalls.incrementAndGet()
           val body =
             
s"""{"storage-credentials":[{"prefix":"s3","config":${json(s3Config(mint()))}}]}"""
               .getBytes(StandardCharsets.UTF_8)
           exchange.getResponseHeaders.add("Content-Type", "application/json")
           exchange.sendResponseHeaders(200, body.length)
           exchange.getResponseBody.write(body)
           exchange.close()
         })
       server.start()
       val refreshEndpoint = 
s"http://127.0.0.1:${server.getAddress.getPort}/credentials";
   
       val first = mint()
       val firstExpiry = first.expiration().toEpochMilli
       val vended = s3Config(first) ++ Map(
         "s3.endpoint" -> minioContainer.getS3URL,
         "s3.path-style-access" -> "true",
         "client.region" -> "us-east-1",
         "client.refresh-credentials-endpoint" -> refreshEndpoint)
       val warehouse = s"s3a://$testBucketName/warehouse-vended-refresh"
   
       try {
         withRESTCatalog(vendedCredentials = vended, warehouseLocation = 
Some(warehouse)) {
           (restUri, _, _) =>
             def catalog(name: String): Seq[(String, String)] = Seq(
               s"spark.sql.catalog.$name" -> 
"org.apache.iceberg.spark.SparkCatalog",
               s"spark.sql.catalog.$name.catalog-impl" -> 
"org.apache.iceberg.rest.RESTCatalog",
               s"spark.sql.catalog.$name.uri" -> restUri,
               s"spark.sql.catalog.$name.warehouse" -> warehouse)
             withSQLConf(
               (catalog("vended") ++ catalog("control") ++ Seq(
                 // The control opts out, so its native scan signs with the 
first credentials.
                 "spark.sql.catalog.control.s3.comet.credential.provider.class" 
-> "",
                 CometConf.COMET_ICEBERG_NATIVE_ENABLED.key -> "true",
                 "spark.sql.adaptive.enabled" -> "false")): _*) {
               spark.sql("CREATE NAMESPACE vended.db")
               spark.sql("CREATE TABLE vended.db.t (id BIGINT, s STRING) USING 
iceberg")
               // Several files, so the scan makes many requests.
               (0 until 8).foreach { i =>
                 spark.sql(
                   s"INSERT INTO vended.db.t SELECT id, repeat('x', 100) " +
                     s"FROM range(${i * 50000}, ${(i + 1) * 50000})")
               }
   
               // Plan both scans now, so their FileIO properties carry the 
first credentials.
               val vendedDf = spark.sql("SELECT count(*), sum(id) FROM 
vended.db.t")
               val controlDf = spark.sql("SELECT count(*), sum(id) FROM 
control.db.t")
               Seq(vendedDf, controlDf).foreach { df =>
                 assert(
                   collect(df.queryExecution.executedPlan) { case s: 
CometIcebergNativeScanExec =>
                     s
                   }.nonEmpty,
                   s"expected a native Iceberg 
scan:\n${df.queryExecution.executedPlan}")
               }
               val callsAtPlan = refreshCalls.get()
   
               val waitMs = firstExpiry - System.currentTimeMillis() + 60000L
               logInfo(s"Waiting ${waitMs / 1000}s for the first credentials to 
expire")
               Thread.sleep(math.max(waitMs, 0L))
               assert(System.currentTimeMillis() > firstExpiry)
   
               val rows = vendedDf.collect()
               val n = 400000L
               assert(rows.head.getLong(0) == n)
               assert(rows.head.getLong(1) == n * (n - 1) / 2)
               val callsAfterRead = refreshCalls.get()
               assert(callsAfterRead > callsAtPlan, "the provider never called 
the refresh endpoint")
   
               val err = intercept[Exception](controlDf.collect())
               val msg = Iterator
                 .iterate[Throwable](err)(_.getCause)
                 .takeWhile(_ != null)
                 .map(_.getMessage)
                 .mkString(" | ")
               logInfo(s"Control failed as expected: $msg")
               println(
                 s"RESULT vended rows=${rows.head} refreshCalls 
plan=$callsAtPlan " +
                   s"afterRead=$callsAfterRead; control error: 
${msg.take(600)}")
               // MinIO drops expired STS keys, so it answers 
InvalidAccessKeyId, not ExpiredToken.
               assert(
                 Seq("ExpiredToken", "InvalidAccessKeyId", 
"PermissionDenied").exists(msg.contains),
                 s"control failed for another reason: $msg")
             }
         }
       } finally {
         server.stop(0)
       }
     }
   }
   ```
   
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to