laxmanmamididd commented on PR #7464:
URL: 
https://github.com/apache/datafusion-comet/pull/7464#issuecomment-6935019884

   Thanks for the careful read, @andygrove.
   
   > Could the rule restrict eviction to the case where the older entry's 
expiry is known to have passed or the new one is known to be later, and leave 
the no-expiry case to the LRU bound?
   
   Yes, done in 6a3279e1a. `FileIoCacheKey::supersedes` now evicts only when 
both entries carry `s3.session-token-expires-at-ms` and the new one is later. 
Any other difference keeps both entries and leaves them to the 64-entry LRU 
bound, as the release does. That covers properties with no expiry, such as an 
OAuth token or a tenant id, and an expiry on one side only. Two sessions or 
tenants that share a catalog name no longer evict each other on every task. The 
cost is that a provider re-vended without an expiry is closed only when the LRU 
bound retires its entry. REST catalogs that vend S3 credentials send the 
expiry, so they still get prompt release.
   
   Changes:
   - `supersedes` and its doc comment, and the eviction paragraph in 
`s3-credential-provider-design.md`, now say the above.
   - `cache_keeps_entries_without_a_later_expiry` replaces 
`cache_evicts_when_only_one_entry_has_an_expiry`. It covers two tenants with 
different OAuth properties, a one-sided expiry, and the LRU bound still 
retiring the least recently used entry.
   - The eviction tests, `CometS3CredentialReleaseSuite` and the 
superseded-FileIO test in `CometS3CredentialBridgeSuite` now give re-vended 
properties a later expiry.
   
   Tested locally on Spark 4.0 and JDK 17: the `iceberg_common` Rust tests (27 
passed), plus `CometS3CredentialReleaseSuite` and 
`CometS3CredentialBridgeSuite` against Minio (11 passed). #7363 is rebased onto 
this commit and merged with `main` to clear its conflict.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to