Hi Prashant, these are great questions. > Now talking about FILE_TYE datatype I am unsure how we are mixing FILE level access with the FILE_TYPE support which is still in talks, sure we can repurpose this machinery but they are not the same ?
My goal in starting this discussion isn't to repurpose the thought process that went into /presign endpoint support for the planning use case, but to start the discussion on the use case for FILE type, so we can actively consider those requirements for existing spec and proposed spec discussions. I've put together two documents with the goal of unpacking this use case in depth: - File Type on IRC governed tables: use case analysis (https://s.apache.org/irc-file-type-use-case) [1] - Batch Signing in Iceberg REST Catalog (https://s.apache.org/irc-batch-signing) [2] I'd appreciate a review on these documents for anyone who's interested in starting to dive into the FILE use case discussion. I'll add this to the agenda for the FILE sync discussions so we can get to these at some point. Here are my current positions to your questions, Prashant, on FILE type: > 1. Do we always want to give FILE_TYPE pointing uri access only via pre-sign ? I think other modes of access work as well, but the requirement differs depending on whether the delegated access is used by the Iceberg client or handed to an external service. Within the Iceberg client, FILE access is well covered by the existing modes: the client already holds whatever the catalog granted for the table and can renew it. For an external service, I think pre-signed URLs are the mode that fits easily fits a hand-off, given that its expiration tends to be longer than the other access delegation modes, and delegated access is also self-contained in the url. > 2. If the file.uri doesn't belong to the table prefix ... how would the > server know this file.uri belongs to the table (as this is embedded deep > within a row of table) to authorize the access ? does having this requirement > Is forcing the server to keep a very expensive trie index in its working > state ? or are we intentionally trying to be less restrictive from spec pov I think this is a policy authoring question, and we have many precedents in Iceberg (e.g. ReadRestrictions, existing StorageCredential, /sign policies) where we've deferred policy authoring outside from the scope of the IRC spec. An Iceberg REST Catalog Server will rely on its own policies to determine if access to a file.uri should be allowed or not, and I think we'd benefit from standardizing how we'd enable access to it, assuming authorization strategy exists in the catalog. > 3. How will the server orchestrate the way of reading data files via vended > creds and FILE_TYPE via pre-signed urls if we go with #1, some sign config or > table level metadata ? presently it's either do vended creds only or remote > signing only. I think this is a client-side concern, not a server orchestration one. The FileIO the catalog issues for a table at load time carries whatever access it was granted for the table's files and, with the batch endpoint, the ability to sign locations on request. Reading data files and requesting signatures for FILE locations are then two calls on the same FileIO, and nothing in the spec needs to switch modes at the table level. I put together a POC (https://github.com/apache/iceberg/pull/18199) [3] that shows the second half: a Spark table-valued function takes the FileIO from the table it was invoked on and signs FILE locations in batches through it. > 3. Can FILE_TYPE access also use vended creds if yes how does the server > orchestrate that ? I've thought about this, and I think it would be helpful to spell out the requirements of FILE_TYPE access first: - do we want to isolate the access delegation decision between table's metadata and data file location prefix from that of its FILE locations? - do we want to accommodate for varying validity period of the token (short for within client, longer if intended for sharing) If we are only thinking of a simpler use case where we assume shared access delegation decision, and same validity period for both questions, then the existing spec and FileIO should work. If we are looking for other requirements, I think we can think about expanding the spec's support for additional arguments. Sung [1] https://s.apache.org/irc-file-type-use-case [2] https://s.apache.org/irc-batch-signing [3] https://github.com/apache/iceberg/pull/18199 On 2026/09/22 17:37:30 Prashant Singh wrote: > Hey Dan, > > Thanks for taking a look at this! > > Regarding the server returning signed urls being slightly different from > supporting pre-signed being slightly different problems, I have a slightly > different POV here (I know we discussed a bit offline as well just wanted > to share our thoughts here as well, as how we have approached this problem > so far) . > Because how else are we planning to refresh these vended urls if that > expires before consumption ? > Scan planning and execution i.e reading these files happen in two different > phases of these engines. Typically a spark driver would plan it and > executor would consume it, it may happen that the url expires before it is > consumed by the executor. > In this case we need to have a refresh mechanism like we do for the vended > creds like /credentials endpoint [1], there is no requirement to call it > only when your creds expire to refresh it in the protocol, a client can > directly call it if it wants creds. Hence the design had both aspects to > make it a complete solution [2] otherwise this is not usable for > practically long running queries please check this out check practical > example here [3]. And we need to keep that simple otherwise we would need > very heavy machinery to keep them in sync and ask the server to re-plan. > Orthogonally I would additionally, also recommend passing the planId to the > pre-sign call for auditability POV like i mentioned in the spec PR [4] and > also advocate for the need for batch sign at-least which could be leveraged > at an executor level need not be at a task level [5], like we do we do for > vended creds. > > Regarding repurposing the sign vs creating the new endpoint I think it's > fine if we want to re-purpose the sign endpoint, the proposal had both [6] > and we initially had approach 1 as the presign dedicate endpoint because > existing request / response model is already called remote signing which is > already well defined term in industry, also it helps in discoverability pov > as from /config we can return this feature is supported or not, i > understand we want to say remote sign in the context the server is doing is > doing signing part (which can be further remote-sign / pre-sign)... I was > trying to narrow down such requirements before making the changes. > > I agree we should align on the implementation here and avoid any duplicate > effort especially when there is active work / discussion going on, Let's > work together to get this in, I will check the PRs to find a path forward ! > > Now talking about FILE_TYE datatype I am unsure how we are mixing FILE > level access with the FILE_TYPE support which is still in talks, sure we > can repurpose this machinery but they are not the same ? I have tons of > questions on this front specially with FILE_TYPE uri ownership from table > pov, I would suggest concluding on them before making concrete assumptions, > I agree we should leverage this thread / community sync to discuss this > more. > > Thanks Sung for starting this, a couple of questions I would like to > contribute to this thread that comes on top of my mind, happy to discuss > this in sync as well: > 1. Do we always want to give FILE_TYPE pointing uri access only via > pre-sign ? > 2. If the file.uri doesn't belong to the table prefix ... how would the > server know this file.uri belongs to the table (as this is embedded deep > within a row of table) to authorize the access ? does having this > requirement > Is forcing the server to keep a very expensive trie index in its working > state ? or are we intentionally trying to be less restrictive from spec pov > 3. How will the server orchestrate the way of reading data files via vended > creds and FILE_TYPE via pre-signed urls if we go with #1, some sign config > or table level metadata ? presently it's either do vended creds only or > remote signing only. > 3. Can FILE_TYPE access also use vended creds if yes how does the server > orchestrate that ? > > Looking forward to the discussion ahead / FILE_TYPE syncs. > > Best, > Prashant Singh > > [1] > https://github.com/apache/iceberg/commit/35a02d035e40344523fdb7a0933e07a8433ea763 > [2] > https://docs.google.com/document/d/1WyCnD63t4AIOzCgP3dWug17kZ6NnVxfOMwOEKqZx6ss/edit?disco=AAACCbiQWNk > [3] https://github.com/delta-io/delta-sharing/issues/941 > [4] https://github.com/apache/iceberg/pull/17332#discussion_r3992567118 > [5] > https://github.com/williamhyun/iceberg/pull/206/changes#diff-7347bb69aaf4647d472cb8d0f3aedfcdff3158a0a9d23aa0e050c5fc180e304fR31 > [6] > > https://docs.google.com/document/d/1WyCnD63t4AIOzCgP3dWug17kZ6NnVxfOMwOEKqZx6ss/edit?tab=t.ez1lkov8k91#heading=h.vjmmhe69p5j3 > <https://docs.google.com/document/d/1WyCnD63t4AIOzCgP3dWug17kZ6NnVxfOMwOEKqZx6ss/edit?tab=t.ez1lkov8k91#heading=h.vjmmhe69p5j3> > > > > On Wed, Sep 16, 2026 at 12:11 AM Daniel Weeks <[email protected]> wrote: > > > I left comments on both sets of PRs and agree we need to align the > > implementations. > > > > The work for presigned urls via plan/tasks is slightly different from > > remote pre-signing. Plan/tasks urls are assumed to already be presigned > > when handed back, so the IO implementations only need to detect and execute > > the request. Remote pre-signing is a step prior to this where a storage > > native path (e.g. s3://...) is routed to the catalog for pre-signing prior > > to opening a stream (via the same path as scan/plan). > > > > I think we should get remote-presigning into a state where it's a viable > > alternative to the current remote signing (largely to address Azure's > > limitations on remote signing) and this should leverage the same > > infrastructure for plan/tasks. (Also, this does not require introducing a > > /presign endpoint). > > > > Once pre-signing is in place, we can leverage it as a foundation for some > > of the file based access delegation. I know there is some PoC work in that > > area (bulk signing), but I don't agree with the direction that's going and > > we need to align on how to introduce bulk operations in a way that works > > for File delegation. > > > > We'll discuss more at the File sync, but much of this is common across all > > of these use cases and we need to introduce the implemention in a way that > > aligns with all of the various use cases. > > > > -Dan > > > > On Wed, Sep 16, 2026 at 3:12 AM Sung Yun <[email protected]> wrote: > > > >> Thanks Prashant, that makes sense. I’m glad to hear you are carrying > >> William’s work forward. > >> > >> If the plan is still to introduce /presign for the plan API response, go > >> ahead; I will subscribe to the PR discussion and fold in what comes out of > >> this thread as the use case discussion evolves. I agree that we can pool > >> our efforts here. > >> > >> As noted in the FILE proposal feedback, and in the review comments on > >> William's PR where we began discussing FILE governance, I think it will > >> help to keep this thread on that use case. What file column usage patterns > >> need, and what SDK or spec changes follow from it. > >> > >> Sung > >> > >> On 2026/09/15 21:33:52 Prashant Singh wrote: > >> > Thanks for starting this thread, Sung! > >> > > >> > I wanted to share some additional background on the work already > >> underway > >> > in this area. William and I opened a spec proposal about three months > >> ago > >> > focused on supporting file-level access through pre-signed URLs [1]. The > >> > initial goal was to enable partition-level sharing (with one file > >> belonging > >> > to a single partition). > >> > > >> > We went through several POCs [2] and a number of community discussions > >> to > >> > evaluate whether the existing remote-signing and batch remote-signing > >> > approaches would be sufficient. In particular, the Azure POC [3] helped > >> > highlight some of the limitations of remote signing across cloud > >> providers > >> > and provided useful evidence for considering pre-signed URLs as an > >> > additional access-delegation mechanism. > >> > > >> > The work has two main pieces: SDK support for the /plan endpoint to > >> return > >> > pre-signed URLs [4], and support for obtaining/refreshing pre-signed > >> URLs > >> > through the signing path [5][6], along with the associated POC [7]. > >> > > >> > I noticed that you have opened PRs #18080 and #18110 for FILE-type > >> access. > >> > The foundational work from the existing PRs, including returning > >> pre-signed > >> > URLs from the plan API and adding client-side support, should be > >> directly > >> > reusable for the FILE work. Given the amount of discussion and > >> > implementation already underway, I'd love to collaborate on building on > >> > these existing tracks rather than creating parallel implementations. > >> > > >> > Happy to work through this together and make sure the existing work can > >> > support the FILE use cases you outlined as well. > >> > > >> > Best, > >> > Prashant > >> > > >> > [1] https://lists.apache.org/thread/odof6m2npvktwd51cz8qnrxjv95ws4wm > >> > [2] > >> > > >> https://github.com/williamhyun/iceberg/pulls?q=is%3Apr+state%3Aopen+author%3Awilliamhyun+sign > >> > [3] https://github.com/williamhyun/Azure-SAS-Header-POC > >> > [4] https://github.com/apache/iceberg/pull/17457 > >> > [5] https://github.com/apache/iceberg/pull/17332 > >> > [6] > >> > > >> https://docs.google.com/document/d/1WyCnD63t4AIOzCgP3dWug17kZ6NnVxfOMwOEKqZx6ss/edit?tab=t.ez1lkov8k91#heading=h.vjmmhe69p5j3 > >> > [7] https://github.com/williamhyun/iceberg/pull/206 > >> > > >> > On Mon, Sep 14, 2026 at 3:53 PM Sung Yun <[email protected]> wrote: > >> > > >> > > Hi folks, > >> > > > >> > > > >> > > There is already work underway to return pre-signed URLs from a plan > >> API > >> > > response, which I believe William and Prashant have scoped to support > >> that > >> > > user journey [1]. As I was reviewing that work, I began thinking ahead > >> > > about delegated access patterns for the ongoing FILE type [2] work, > >> and as > >> > > Dan mentioned in this comment [3], we have yet to have a dedicated > >> > > discussion on access delegation for FILE. So I wanted to start this > >> thread > >> > > to begin that discussion. > >> > > > >> > > > >> > > Today access is delegated through remote signing and vended > >> credentials, > >> > > and I believe both can support FILE type access delegation use cases > >> as > >> > > long as the object is read by a storage client. With multimodal > >> access, > >> > > the interesting work often happens in other services, for example an > >> > > inference service, that need to retrieve the objects to run the > >> inference > >> > > itself, the result of which is then returned to the calling process > >> for > >> > > further analytics. While we could require that these functions run in > >> the > >> > > calling process, or that those services act as storage clients > >> themselves, > >> > > I think enabling FILE reference URLs to be signed so that they are > >> > > optimized for sharing with other services will help Iceberg's FILE > >> > > integration plug readily into the existing multimodal ecosystem. > >> > > > >> > > > >> > > While work is still underway on the FILE type discussions to finalize > >> the > >> > > shape of the Iceberg type, I'd like to start the discussion on what > >> access > >> > > enhancements would need to be added to the Iceberg REST Catalog spec > >> and > >> > > its supporting SDKs. I've put together a spec PR based on recent > >> catalog > >> > > community discussion [4] on pre-signed urls, along with a draft PR > >> that > >> > > POCs the integration on the client side [5]. > >> > > > >> > > > >> > > I'd love to get the community's thoughts on whether client initiated > >> > > pre-signing + sharing URLs is an additional mode we want to introduce > >> to > >> > > enable FILE access sharing as we look to integrate the Iceberg client > >> into > >> > > the multimodal ecosystem. > >> > > > >> > > > >> > > I'd also be interested to hear what other consumer patterns people are > >> > > running into beyond integrating with an inference service, which is > >> the > >> > > case I've been designing against. > >> > > > >> > > > >> > > Sung > >> > > > >> > > > >> > > [1] https://github.com/apache/iceberg/pull/17457 > >> > > > >> > > [2] https://s.apache.org/iceberg-fileref > >> > > > >> > > [3] > >> https://github.com/apache/iceberg/pull/17332#discussion_r3992139348 > >> > > > >> > > [4] https://github.com/apache/iceberg/pull/18080 > >> > > > >> > > [5] https://github.com/apache/iceberg/pull/18110 > >> > > > >> > > >> > > >
