Sorry, one more thing about the methods:
Table reloadTable(Table); // or,
Table reloadTable(TableIdentifier, Table) // where Table could be NULL
I want to highlight that it is super easy to provide a default
implementation which just loads the table. Then later, catalog
implementations can just add their clever tricks to make it more
efficient.
Cheers,
Zoltan
On Thu, Nov 21, 2024 at 9:53 AM Zoltán Borók-Nagy <borokna...@apache.org> wrote:
>
> Hi,
>
> I agree with Gabor that the support of efficiently reloading Iceberg
> tables is a generic problem that applies to all catalog
> implementations.
> I also think that the programming API, especially the Iceberg Java
> library is very important, as almost all Iceberg clients use this
> library to interact with Iceberg tables, no matter which catalog they
> reside in.
> Even the engines that are mostly written in C++ (Impala, Starrocks,
> Doris) interact with Iceberg tables from their Java frontends using
> the Iceberg Java library.
>
> "Since libraries are open-source, I can modify them as needed for my
> use case" - if you want to maintain a private fork, then sure,
> otherwise you really want to avoid introducing breaking changes. Also,
> you want to introduce new features in a way that is acceptable for the
> community. In that sense, modifying a library's interface is not much
> easier than modifying a server's interface. Of course, clients of a
> library have control over when to upgrade, a privilege you don't
> always have for server APIs, but this is why API versioning was
> invented, anyway, we are diverging from the main topic here.
>
> Since this reloadTable() method could be useful for other Catalog
> implementations as well, I think we would like to add a new method to
> org.apache.iceberg.catalog.Catalog that doesn't take any
> implementation-specific detail about the underlying catalog. To
> overcome this, catalogs could embed catalog-specific information into
> the Table object when they initially load the table, e.g. "String
> catalogVersion". In the case of the REST Catalog the catalogVersion
> string would be the ETag. Other catalogs might not even need to add
> anything, as the metadata_location of the Table object is sufficient.
>
> This way the API would be simple and generic:
>
> Table reloadTable(Table); // or,
> Table reloadTable(TableIdentifier, Table) // where Table could be NULL
>
> Cheers,
> Zoltan
>
> On Thu, Nov 21, 2024 at 1:51 AM Taeyun Kim <taeyun....@innowireless.com>
> wrote:
> >
> > Hi Gabor,
> >
> > On HTTP Caching:
> >
> > If an HTTP client library performs caching by default and doesn’t allow
> > disabling it, I believe that library shouldn’t be used - at least in the
> > context of this discussion.
> > The kind of HTTP client library I have in mind is one that handles encoding
> > and decoding of HTTP headers and body, as well as connection pooling. The
> > responsibility for interpreting headers, status, and body content should
> > remain with the application. While caching support can be provided, it
> > should be optional.
> > When using a library that behaves as I described, the issues you mentioned
> > in points 4) and 5) shouldn’t arise, as the library wouldn’t interfere with
> > caching.
> > For reference, the Rust reqwest crate (which Iceberg-Rust appears to use)
> > seems to operate as expected in this regard.
> >
> > On Programming Languages and APIs:
> >
> > One of my points was that there doesn’t seem to be a reason to center the
> > discussion around Java (and its libraries).
> > BTW, I don’t think it’s necessary for the functions in the iceberg-rust
> > library to be identical to those in the Java library. Optimal solutions may
> > vary by language, and library developers may have different design goals.
> > Personally, my primary focus is on the REST catalog API specification,
> > rather than language-specific library APIs. (To avoid confusion, I’ll refer
> > to the REST catalog API as the "specification" from here on.)
> > Library APIs are (merely) implementations designed to make the
> > specification easier to use. Since libraries are open-source, I can modify
> > them as needed for my use case (and, in fact, I’ve made modifications to
> > iceberg-rust for my purposes). However, the specification defines the
> > interface between different applications or servers, making it immutable
> > for practical purposes.
> >
> > On ETags:
> >
> > The decision to use ETags is not just an implementation detail - it is part
> > of the specification itself. In my view, it is far more significant than
> > the signature of a library API function. I’ve outlined the reasons for this
> > above.
> >
> > On the Proposal:
> >
> > I agree that the current function (loadTable(TableIdentifier)) cannot be
> > freshness-aware. This is expected, as the caller doesn’t provide the
> > version it holds, leaving the callee with no basis for comparison.
> > On the other hand, the proposed new function signature doesn’t seem to
> > provide a way for the caller to supply ETags (or equivalent identifiers
> > representing specific table versions for other catalog types). Is such
> > information intended to be embedded within the Table structure?
> > To me, it seems clearer to explicitly provide such information (like ETags)
> > rather than embedding it in the Table structure. That said, I might be
> > misunderstanding the intention here.
> >
> > Thank you.
> >
> >
> > -----Original Message-----
> > From: "Gabor Kaszab" <gaborkas...@apache.org>
> > To: <dev@iceberg.apache.org>;
> > Cc:
> > Sent: 2024-11-19 (화) 21:26:01 (UTC+09:00)
> > Subject: Re: [DISCUSS] REST: Way to query if metadata pointer is the latest
> >
> >
> > Hi,
> >
> >
> > Thanks for sharing your view, Taeyun! I think there are many levels of
> > representation here and we might not mean the same with our points. I think
> > in general an interaction between a query engine and an Iceberg REST
> > catalog has these different layers:
> > 1) The engine (Impala, Spark, Trino, etc.).
> > 2) Catalog API of the Iceberg lib offers loadTable(TableIdentifier) that
> > returns a Table object. Different language implementations seem to have the
> > same API (Java, Rust, etc.).
> > 3) The particular implementation of a catalog that implements the above
> > loadTable(TableIdentifier) function. In this example the RESTCatalog /
> > RESTSessionCatalog.
> > 4) RESTClient implemented by HTTPClient (used by the REST catalog) to
> > communicate with the REST server (still implemented within Iceberg)
> > 5) The external HTTPClient
> > (org.apache.hc.client5.http.impl.classic.CloseableHttpClient) that
> > orchestrates the HTTP traffic between the client and the server
> >
> >
> > Let me reflect on your comments:
> > - HTTP caching
> > With the above layers in mind if I'm not mistaken HTTP Caching would be
> > configured in 4) and the actual caching of HTTP responses would be in 5).
> > This is what I meant by HTTP layer. With HTTP Caching the control of how
> > long a cached TableMetadata is stored will no longer be in 1) and would be
> > in 4) - 5). I don't think that any of the engines that cache table metadata
> > would want to have this loss of control.
> >
> >
> > - Programming language
> > I'm not sure I get your point with this. The Catalog API seems the same
> > regardless of programming language (See the links for 2) ).
> >
> >
> > - ETags
> > An ETag is an implementation detail that is relevant for HTTP
> > communication. We can't extend the Catalog API in 2) nor in 3) with
> > functions that are aware of ETags (e.g. return ETags or accept ETags as
> > param). Those APIs are common for all the Catalog implementations including
> > ones that don't leverage ETags for HTTP traffic.
> >
> >
> > Proposal:
> > - Catalog API
> > I don't think that the current Catalog.loadTable(TableIdentifier) API in 2)
> > is suitable for a freshness-aware table loading use case. It wouldn't be
> > transparent to the clients if that actual catalog implementation avoids
> > reloading the table if it hasn't changed or if that catalog implementation
> > reloads the table unconditionally with this API call.
> > Also it doesn't seem straightforward what the API should return if the
> > table is considered fresh. This API returns a Table object and in case we
> > get a 304 without a body from the catalog server, we won't have a way to
> > construct a Table object as a return value for this API. I already shared
> > my concerns for caching the LoadTableResponses within 4) - 5)
> >
> >
> > So I think we need a new API on the Catalog for this purpose. Thanks Zoltan
> > for the naming suggestion, after I sent my mail yesterday I also thought
> > that I could've come up with a more intuitive name.
> > This can either be:
> > a) Table reloadTable(TableIdentifier, Table)
> > b) Table loadTableIfChanged(TableIdentifier, Table)
> >
> >
> > With this Catalog level API we could provide the current known state of
> > that particular table as a parameter, and if the client side of the catalog
> > implementation finds that the table hasn't changed it can return this Table
> > object for the current state. With this approach no caching would be needed
> > in 2) - 5). It's up to the catalog implementation how it finds out if the
> > table has been changed or not.
> >
> >
> > - REST API, REST spec
> > The REST API could use the ETag approach to check table freshness. As
> > described in previous mails this could reduce the number of round trips to
> > refresh a table to one without the need of separately checking the
> > freshness. We could use the same endpoint as we do for the current
> > loadTable(), with an additional mention of an optional ETag being used and
> > also including the 304 into the possible responses.
> >
> >
> > For this approach we have to cache the [TableIdentifier -> last ETag]
> > mapping somewhere. I think 4), the Iceberg specific HTTPClient could be
> > suitable for this purpose, however, this seems too low level for this
> > purpose. We can also consider RESTSessionCatalog to cache the ETags.
> >
> >
> > This is something to be considered, but for REST catalog implementations
> > that don't support the ETag based implementation, they would just simply
> > perform a regular loadTable operation, not bothering with sending 304
> > codes. We can also investigate if we should get an exception if that
> > particular REST implementation doesn't support the ETag approach, so that
> > clients could notice if there is no freshness-aware table loading under the
> > hood.
> >
> >
> > - Other catalog types
> > Currently we focus on the REST catalog implementation but the above Catalog
> > API proposal could work for other catalog types too. The internal
> > implementation could be different, though. Initially they could throw a
> > NotImplementedException.
> >
> >
> > I hope this makes sense and I haven't missed any details or previous
> > comments.
> >
> >
> > Regards,
> > Gabor
> >
> >
> >
> >
> > On Tue, Nov 19, 2024 at 5:17 AM Taeyun Kim <taeyun....@innowireless.com>
> > wrote:
> >
> > Hi,
> >
> > Here are my thoughts:
> >
> > - HTTP Layer: To my knowledge, there isn’t a separate "HTTP layer" in this
> > context, so concerns about control over caching shouldn’t be an issue. The
> > header approach I mentioned simply involves handling additional headers
> > when using HTTP client libraries to interact with the REST API.
> >
> > - Programming Language: For reference, I don’t use Java - I use Rust and
> > C++. Personally, I hope Iceberg’s specifications avoid including
> > Java-specific features and that the cross-language compatible REST catalog
> > becomes the primary catalog for Iceberg.
> >
> > - API Perspective: Given the above, I may not be in the best position to
> > comment on Java APIs. However, regarding Gabor’s proposed API (Table
> > loadTable(Table existingTable)), I believe it would be difficult for an
> > ETag-based REST catalog to support this API since it cannot provide an
> > ETag. Instead, I’d like to suggest an alternative API:
> > Option<Table, tag> loadTableIfNoneMatch(TableIdentifier, Option<tag>)
> > Initially, the client would provide None as the tag. If the tag is not None
> > and matches the latest table tag, the API returns None (= not updated). If
> > the tag is None or does not match the latest table tag, the API returns a
> > new (Table, tag) pair.
> >
> > Thank you.
> >
> >
> > -----Original Message-----
> > From: "Zoltán Borók-Nagy" <borokna...@cloudera.com.invalid>
> > To: <dev@iceberg.apache.org>;
> > Cc:
> > Sent: 2024-11-19 (화) 03:16:05 (UTC+09:00)
> > Subject: Re: [DISCUSS] REST: Way to query if metadata pointer is the latest
> >
> >
> > Hey Everyone,
> >
> >
> > Thanks Gábor, I think the proposed interface would be very useful to any
> > engine that employs caching, e.g. Impala.
> > And it is pretty neat that it is catalog-agnostic, i.e. we just give all
> > the information we have about the table and let the catalog implementation
> > efficiently reload it.
> >
> >
> > I might have a nitpick suggestion about the name to clearly express the
> > intent: loadTable -> reloadTable (or, refreshTable)
> >
> >
> > Cheers,
> > Zoltan
> >
> >
> >
> >
> > On Mon, Nov 18, 2024 at 5:17 PM Gabor Kaszab
> > <gaborkas...@cloudera.com.invalid> wrote:
> >
> > Hi Iceberg Community,
> >
> >
> > This is a great conversation so far, and thanks everyone for the valuable
> > inputs!
> > I'd like to articulate 2 things that we have to keep in mind with the
> > design:
> >
> >
> > 1: There are 2 interfaces here that we should consider:
> > What I mean by this is that so far we have been talking about the REST
> > spec, more narrowly the HTTP communication between Iceberg's REST client
> > and the REST server. I think the proposed solution with the ETag absolutely
> > makes sense within this context.
> > However, the usual way of a client interacting with an Iceberg catalog
> > (including REST) is the Catalog API in the library. This API offers a
> > loadTable(TableIdentifier) function that returns a Table object. With the
> > above HTTP-based solution in mind I don't think we could give any
> > meaningful results if the HTTP layer finds that the table hasn't changed. I
> > argued already against pushing the caching responsibilities from the
> > clients into the HTTP layer (mostly because of losing the control over the
> > cache, and also observability won't be straightforward) so let's assume for
> > now that we won't do caching in the HTTP layer, only execute the loadTable
> > calls to the REST catalog by setting the ETag. In case we get a 304 we
> > won't be able to construct a Table object to answer the
> > Catalog.loadTable(TableIdentifier) call. We could return null or throw an
> > exception but I don't find any of them appropriate.
> >
> >
> > 2: There are catalog types other than REST
> > I started this conversation focusing on the REST spec, but the more I think
> > of this the more I feel that the same functionality should be offered for
> > all the other catalog types too. Let's assume that we have an engine that
> > caches table metadata and initially uses REST catalog. For such an engine
> > the proposed solution would solve the problem of checking table freshnes
> > and also reloading the table metadata. A simple code for that could be
> > enough if we configured our HTTP client properly (just sketched a simple
> > example):
> >
> >
> > tableCache_.put(catalog_.loadTable(tableIdentifier));
> >
> >
> > Also let's assume we solve the issue in 1) and we can answer such a call
> > even if we get 304 from the server as the table is unchanged. So with this
> > solution with the REST catalog we can be sure that the table is only loaded
> > from the catalog if changed (or the age expired). But what if we configure
> > another catalog, let's say HiveCatalog. The very same code for that catalog
> > would trigger a table reload for every execution causing unexpected
> > performance issues.
> > I have to double check but I assume that this HTTP approach wouldn't be
> > feasible for other catalog types unfortunately.
> >
> >
> > I hope these arguments make sense :)
> >
> >
> >
> >
> > As a partial solution this is what I have in mind:
> > We can add another function into the catalog API for this purpose. Let's
> > say something like this:
> > Table loadTable(Table existingTable);
> >
> >
> > What advantages I see with this:
> > - This could solve issue 1) above. In case the table hasn't changed we can
> > simply return 'existingTable' without using HTTP Cache.
> > - The clients wouldn't need to explicitly call for isLatest() and such
> > functions to check for freshness, and they wouldn't need to trigger table
> > reloading for themselve. This API would be expected to cover this under the
> > hood.
> > - The current Catalog.loadTable(TableIdentifier) API wouldn't be enough for
> > all the catalog types on it's own, but with this one each catalog
> > implementations (e.g. HiveCatalog, REST catalog, etc.) then can implement
> > their own way of doing freshness checks and table reloads. For REST we
> > could follow the HTTP ETag approach, while for other catalogs we could
> > follow other approaches.
> >
> >
> > Regards,
> > Gabor
> >
> >
> > On Mon, Nov 18, 2024 at 8:48 AM Shani Elharrar <sh...@upsolver.com.invalid>
> > wrote:
> >
> > You're totally right. Perhaps using a "Content-Location" header might be a
> > better fit for that.
> >
> >
> > Shani.
> >
> >
> > On 18 Nov 2024, at 9:27, Taeyun Kim <taeyun....@innowireless.com> wrote:
> >
> >
> >
> > Hi,
> >
> >
> > Here are my thoughts:
> >
> >
> > - The value of ETag is (as far as I know) defined as an opaque string by
> > the specification, meaning the client shouldn’t interpret or assign any
> > significance to it, regardless of what the server specifies. It’s best to
> > avoid the client giving any particular meaning to the ETag value.
> > - One major advantage of the header approach compared to other methods is
> > that if an update has occurred, the updated content can be immediately
> > included in the response without requiring an additional request. This
> > saves one request-response round-trip (although It’s also possible to
> > define a separate endpoint with the same functionality).
> > - Since the Iceberg REST catalog server is effectively a type of HTTP
> > server, at least in theory, it may be expected to handle HTTP cache and
> > validation-related processes. The header approach can be seen as leveraging
> > this mechanism appropriately.
> > - The header approach doesn’t have to be limited to the
> > /v1/{prefix}/namespaces/{namespace}/tables/{table} endpoint. It could also
> > be applied to all GET-based endpoints, though this might broaden the scope
> > significantly.
> >
> >
> > Thank you.
> >
> >
> >
> >
> > -----Original Message-----
> > From: "Shani Elharrar" <sh...@upsolver.com.invalid>
> > To: <dev@iceberg.apache.org>;
> > Cc: <dev@iceberg.apache.org>;
> > Sent: 2024-11-18 (월) 16:21:16 (UTC+09:00)
> > Subject: Re: [DISCUSS] REST: Way to query if metadata pointer is the latest
> >
> >
> > Using the metadata file name as ETag is nice way to go. In that case,
> > adding HEAD method support to the loadTable endpoint will return the latest
> > metadata pointer, which can be used to support "isLatest" without returning
> > the body. It can be also leveraged in order to return the latest metadata
> > location of the table.
> >
> >
> > Shani.
> >
> >
> > On 18 Nov 2024, at 8:52, Yufei Gu <flyrain...@gmail.com> wrote:
> >
> >
> >
> >
> > Hi Taeyun,
> >
> > Thank you for the clear explanation.
> >
> > I agree that the ETag solution is more suitable. If we were going that way,
> > I'd propose a customized version number as an ETag—for instance, leveraging
> > the metadata.json file name as the identifier.
> >
> > To summarize, HTTP caching relies on headers (e.g., ETag or Last-Modified)
> > to validate whether a version is up-to-date, whereas the alternative
> > approach proposed above uses an additional parameter for verification. From
> > my perspective, there isn’t a fundamental difference between the two, so
> > I’m OK with either.
> >
> > A couple of points to note:
> >
> > Both approaches would require changes to the "loadTable" endpoint.
> > A minor advantage of HTTP caching is that it integrates seamlessly with
> > browsers, but since most clients of the Iceberg REST catalog aren’t
> > browsers, this may not be a significant factor.
> > I’d also recommend considering the requirement to retrieve multiple
> > tables(e.g., all tables under a namespace, or a list of table names) from
> > the catalog. This requires a new endpoint and may not work with HTTP
> > caching.
> >
> > Let me know your thoughts or if there’s anything else to consider.
> >
> > Yufei
> >
> >
> >
> >
> > On Sun, Nov 17, 2024 at 6:43 PM Taeyun Kim <taeyun....@innowireless.com>
> > wrote:
> > Hi,
> >
> > To Gabor:
> > It doesn’t seem necessary to interpret HTTP caching literally in this
> > context.
> > Simply using the HTTP headers defined by HTTP caching to check the
> > freshness of metadata should be sufficient.
> > There’s no requirement for the client to duplicate or store cached HTTP
> > responses.
> >
> > To Yufei:
> > As I understand it, the client doesn’t send its own timestamp but instead
> > uses the timestamp originally provided by the server in the Last-Modified
> > header.
> > Therefore, clock synchronization issues should not be a concern.
> >
> > Here’s the general flow of HTTP cache validation based on If-Modified-Since:
> >
> > - Client: initial request:
> >
> > GET (url) HTTP/1.1
> >
> > - Server response:
> >
> > HTTP/1.1 200 OK
> > Last-Modified: (date1)
> > Cache-Control: no-store, no-cache, max-age=0, must-revalidate,
> > proxy-revalidate
> > (with response body)
> >
> > - Client: validation request:
> >
> > GET (url) HTTP/1.1
> > If-Modified-Since: (date1)
> >
> > - Server response (if unchanged):
> >
> > HTTP/1.1 304 Not Modified
> > Last-Modified: (date1)
> > Cache-Control: no-store, no-cache, max-age=0, must-revalidate,
> > proxy-revalidate
> > (without response body)
> >
> > - Server response (if updated):
> >
> > HTTP/1.1 200 OK
> > Last-Modified: (date2)
> > Cache-Control: no-store, no-cache, max-age=0, must-revalidate,
> > proxy-revalidate
> > (with response body)
> >
> > However, using time-based freshness checks can present challenges, such as
> > parsing time formats or synchronizing file update times across servers.
> > To address these issues, HTTP cache validation based on ETag is also
> > defined in the specification.
> >
> > Here’s the flow for ETag-based validation:
> >
> > - Client: initial request:
> >
> > GET (url) HTTP/1.1
> >
> > - Server response:
> >
> > HTTP/1.1 200 OK
> > ETag: "(arbitrary string 1 generated by the server)"
> > Cache-Control: no-store, no-cache, max-age=0, must-revalidate,
> > proxy-revalidate
> > (with response body)
> >
> > - Client: validation request:
> >
> > GET (url) HTTP/1.1
> > If-None-Match: "(arbitrary string 1 generated by the server)"
> >
> > - Server response (if unchanged):
> >
> > HTTP/1.1 304 Not Modified
> > ETag: "(arbitrary string 1 generated by the server)"
> > Cache-Control: no-store, no-cache, max-age=0, must-revalidate,
> > proxy-revalidate
> > (without response body)
> >
> > - Server response (if updated):
> >
> > HTTP/1.1 200 OK
> > ETag: "(arbitrary string 2 generated by the server)"
> > Cache-Control: no-store, no-cache, max-age=0, must-revalidate,
> > proxy-revalidate
> > (with response body)
> >
> > The server can choose to use either If-Modified-Since or ETag for freshness
> > validation.
> > Alternatively, to simplify the implementation related to the Iceberg REST
> > catalog, it might make sense to define only the more accurate ETag-based
> > validation in the spec.
> > For reference, RFC 9110 recommends specifying both ETag and Last-Modified.
> > When both are provided, ETag takes precedence.
> >
> > Note on Cache-Control Headers:
> > The Cache-Control values in the examples above are intended to ensure that
> > the client validates freshness with the server on every request. Writing
> > the header in this extended format is primarily to accommodate outdated
> > HTTP/1.1 implementations. However, under the HTTP/1.1 specification, the
> > following is sufficient:
> >
> > Cache-Control: no-cache
> >
> > That’s all for now.
> > Thank you.
> >
> >
> > -----Original Message-----
> > From: "Yufei Gu" <flyrain...@gmail.com>
> > To: <dev@iceberg.apache.org>;
> > Cc:
> > Sent: 2024-11-16 (토) 02:51:05 (UTC+09:00)
> > Subject: Re: [DISCUSS] REST: Way to query if metadata pointer is the latest
> >
> >
> >
> > How does HTTP caching handle desynchronized clocks between clients and the
> > server?
> >
> > At t0, the client gets the latest table version.
> > At t1, the server makes a new commit.
> > At t2, the client sends a request with a timestamp t2, but due to
> > desynchronization, it refers to t0.
> >
> > The server may reply with 304 Not Modified, causing the client to think its
> > cache is up-to-date and miss the commit at t1.
> >
> >
> >
> > Yufei
> >
> >
> >
> >
> > On Fri, Nov 15, 2024 at 6:37 AM Gabor Kaszab <gaborkas...@apache.org> wrote:
> > Hi All,
> >
> >
> > First of all it's great to see that there are others who could benefit from
> > giving a solution to this problem. I appreciate all the comments and
> > feedback so far.
> > There were a number of different opinions, so let me start with summarizing
> > the different topics that came up:
> >
> >
> > New endpoint vs using an existing endpoint:
> > Based on the answers (Fokko, Yufei) I had the impression that we should be
> > careful when adding new REST endpoints, and we should examine the re-use of
> > existing endpoints first. Let's do that then, and in case we don't find it
> > feasible then we can still fall back to any of my initial proposals
> > (isLatest() or metadataLocation()).
> >
> >
> > Granularity of freshness checks:
> > It was brought up (Dmitri) that we might not want to do the metadata
> > freshness checks solely based on metadata location, but we should consider
> > doing more granular freshness checks. I personally don't see much benefit
> > of designing this solution like that, TBH, but seeing some use-cases could
> > help us understand the motivation here.
> > Let me share my opinion on some of the arguments:
> >
> >
> > "A change in metadata location does not necessarily mean a change in
> > metadata content"
> >
> >
> > AFAIK whenever Iceberg creates a new metadata file there is some change in
> > the metadata itself. There might not be a new snapshot, though in the cases
> > of e.g. a schema/partition evolution. But even in these cases triggering a
> > table reload could make sense to me (e.g. answering SHOW CREATE TABLE and
> > similar queries). Additionally, I'd assume the number of metadata location
> > changes that don't create a new snapshot is too negligible to optimize for.
> > Dmitri, let me know if I misunderstood something.
> >
> >
> > "it may still be beneficial to permit the client to ask for changes to
> > specific areas of metadata"
> >
> > This seems like a use-case that the partial metadata loading proposal could
> > solve. To identify the need to load a specific part of the metadata with
> > partial metadata loading seems an overkill to design with my proposal, if
> > this is what you have in mind. Also I found that the partial metadata
> > loading proposal faces serious headwinds, so I wouldn't rely on it at the
> > moment.
> >
> >
> > Re-using tableExists
> > I think there is a consensus here that tableExists returning a metadata
> > location could work but seems more like a workaround and could be
> > misleading for the users.
> >
> > Partial metadata loading could solve this:
> > (Yufei) I agree, it would be perfect for my use-case and I'm following the
> > discussion on the proposal. However, for me it seems, as I wrote above,
> > that the proposal faces serious headwinds now and I honestly wouldn't
> > expect a solution in the short term. But solving the freshness problems is
> > a more urgent thing to solve, not just for myself and Impala but apparently
> > to many other stakeholders in the community according to the interest on
> > this thread.
> > Hence, I propose to come up with a separate solution for freshness checks,
> > and we can still move to using partial metadata loading once that's out.
> >
> >
> > Use HTTPCache and If-Modified-Since with loadTable
> > This solution seems to do the trick for us. Let me do some research myself
> > to see if there are any difficulties implementing this. Currently, I have
> > more questions than answers wrt this approach :)
> > - The initial problem is to answer freshness questions for the cached
> > tables on the client side. If we introduce HttpCaching wouldn't we
> > introduce the same problem but on a different level of representation. We'd
> > then need to decide the freshness/staleness of the cached data in the HTTP
> > layer.
> > - If we cache the HTTP responses for a loadTable then we essentially cache
> > the content of the metadata.jsons including the snapshot and metadata log
> > and everything, plus the snapshot list (and I think the manifests for the
> > latest snapshot). I believe that the size of this can easily reach the low
> > megabytes range in memory, so in total keeping them in the HTTP Cache for
> > all the tables we have queried can easily mean that we keep a couple of GBs
> > in memory just for this purpose.
> > For engines that already cache table metadata wouldn't this mean that we
> > will cache some parts of the metadata redundantly?
> > - How would we decide what is the max-age of a cached table metadata in the
> > HTTP Cache? Would it be configurable so that each engine could use whatever
> > it prefers?
> >
> >
> > Sorry if any of the questions doesn't make sense, I just want to make sure
> > I understand all the aspects of this approach.
> >
> >
> > An additional topic I have in mind:
> > REST catalog vs other catalogs:
> > Now we are focusing our discussion on the REST spec, but I think it would
> > be beneficial to extend our focus and cover other catalog implementations
> > too. I don't think that this problem of data freshness is specific to REST
> > catalog, it could affect any table in any other catalog too.
> >
> >
> > I'll continue my investigation wrt the proposals, I just wanted to flush
> > out and sum up what we have now before the weekend.
> >
> >
> > Regards,
> > Gabor
> >
> >
> >
> >
> > On Fri, Nov 15, 2024 at 10:16 AM Jean-Baptiste Onofré <j...@nanthrax.net>
> > wrote:
> > Hi,
> >
> > I like the idea and it makes sense. As soon as it's clearly stated in
> > the spec (using If-Modified-Since header and 304 status code), it
> > looks good to me.
> >
> > Thanks !
> > Regards
> > JB
> >
> > On Fri, Nov 15, 2024 at 1:58 AM Taeyun Kim <taeyun....@innowireless.com>
> > wrote:
> > >
> > > Hi,
> > >
> > > (Apologies if this email is a duplicate. This is my third attempt.)
> > >
> > > I also need a way to ensure that my table data is up-to-date. For now,
> > > I’m handling this by setting an expiration period after which I fetch the
> > > data again, regardless of its freshness.
> > >
> > > Here are my thoughts on the current suggestions. Please correct me if
> > > I've misunderstood any of the points.
> > >
> > > - isLatest(): This function could be inefficient since it would require
> > > an additional round-trip to fetch the metadata if it’s not up-to-date.
> > > This would result in two round-trips overall, which seems suboptimal.
> > > - metadataLocation(): This has a similar issue as isLatest(). BTW,
> > > according to the REST catalog API documentation for LoadTableResult
> > > schema, it states, "Clients can check whether metadata has changed by
> > > comparing metadata locations after the table has been created."
> > > (https://github.com/apache/iceberg/blob/3659ded18d50206576985339bd55cd82f5e200cc/open-api/rest-catalog-open-api.yaml#L3175)
> > > This suggests that if the metadata location has changed, the metadata
> > > can be considered updated.
> > > - tableExists(): Based on the name, this function seems to serve a
> > > different purpose.
> > >
> > > Here is my suggestion:
> > >
> > > Since HTTP has built-in caching features
> > > (https://developer.mozilla.org/en-US/docs/Web/HTTP/Caching), and REST
> > > catalogs operate over HTTP, it seems natural to leverage HTTP caching
> > > mechanisms. For example, HTTP includes the If-Modified-Since header and
> > > the 304 Not Modified status code. Using this approach, we could achieve
> > > data freshness with a single round-trip, fetching updated data only if
> > > there are modifications.
> > >
> > > What do you think about defining the spec in this direction?
> > >
> > > Thank you.
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: "Yufei Gu" <flyrain...@gmail.com>
> > > To: <dev@iceberg.apache.org>;
> > > Cc:
> > > Sent: 2024-11-13 (수) 03:43:24 (UTC+09:00)
> > > Subject: Re: [DISCUSS] REST: Way to query if metadata pointer is the
> > > latest
> > >
> > >
> > >
> > > Hi Gamber,
> > >
> > > Thanks for the proposal! Impala isn’t unique in needing this—I've seen
> > > similar requirements from other engines.
> > >
> > > As others pointed out, using the “tableExists” endpoint seems like a
> > > workaround. I don't consider it a permanent way forward. We could address
> > > this by either modifying the current load table endpoint or introducing a
> > > new one, but ideally, we should avoid adding endpoints for every specific
> > > need. With that, partial metadata loading seems like a strong approach
> > > here, we will need certain agreement though. I'd suggest the community
> > > consider the use cases seriously. We need a way forward.
> > >
> > > I’m also not too concerned about using metadata file paths to verify the
> > > latest table version; clients can simply extract metadata filenames,
> > > which include the UUID.
> > >
> > > Yufei
> > >
> > >
> > >
> > >
> > > On Tue, Nov 12, 2024 at 7:46 AM Jean-Baptiste Onofré <j...@nanthrax.net>
> > > wrote:
> > >
> > > Hi Fokko
> > >
> > > I like the idea, but I think it's more a workaround and could be
> > > confusing for users :)
> > >
> > > Regards
> > > JB
> > >
> > > On Tue, Nov 12, 2024 at 2:53 PM Fokko Driesprong <fo...@apache.org> wrote:
> > > >
> > > > Hey Gabor,
> > > >
> > > > Thanks for raising this. While reading this, my first thought is to
> > > > leverage the `tableExists` operation:
> > > > https://github.com/apache/iceberg/blob/e3f39972863f891481ad9f5a559ffef093976bd7/open-api/rest-catalog-open-api.yaml#L1129-L1160
> > > >
> > > > This doesn't return anything today, but we could return a payload to
> > > > the latest metadata.json.
> > > >
> > > > Looking forward to what others think.
> > > >
> > > > Kind regards,
> > > > Fokko
> > > >
> > > >
> > > >
> > > >
> > > > Op di 12 nov 2024 om 14:33 schreef Shani Elharrar
> > > > <sh...@upsolver.com.invalid>:
> > > >>
> > > >> I recommend option (b), provided there is no partial metadata loading.
> > > >> We implemented option (b) internally to facilitate partial metadata
> > > >> loading, as we have tables with hundreds of thousands of snapshots.
> > > >> This results in metadata that occupies approximately 500 MB in memory
> > > >> (excluding the JsonNodes), which is a significant load for some of our
> > > >> services.
> > > >>
> > > >> Shani.
> > > >>
> > > >> On 12 Nov 2024, at 14:12, Gabor Kaszab <gaborkas...@apache.org> wrote:
> > > >>
> > > >> Hey Iceberg Community,
> > > >>
> > > >> Background:
> > > >> Impala is designed in a way to cache the Iceberg table metadata
> > > >> (BaseTable objects in practice) for faster access. Currently, Impala
> > > >> is tightly coupled with HMS and in turn with the HiveCatalog, and in
> > > >> order to keep the cached table objects up-to-date there is a
> > > >> notification mechanism driven by HMS to notify Impala about any
> > > >> changes in the table metadata.
> > > >> The Impala community is actively looking for ways to decouple HMS from
> > > >> Impala and provide a way to use Impala without the need for HMS, and
> > > >> get the Iceberg table metadata from other catalog Implementations
> > > >> mainly focusing now on REST catalogs.
> > > >>
> > > >> Problem to solve:
> > > >> We identified a particular missing functionality in the current REST
> > > >> spec: For engines that cache table metadata currently there is no way
> > > >> to check if that table metadata is up-to-date or not, and whether the
> > > >> engine should reload the metadata for that table or not without
> > > >> getting a whole table object from the catalog. For this I think the
> > > >> REST catalog (but in fact I think this could apply to any other
> > > >> catalogs) should be able to answer a question like:
> > > >> "Hi Catalog, I have this version of this table, is it up-to-date?"
> > > >>
> > > >> Proposal:
> > > >> I've been following the discussion about partial metadata loading that
> > > >> could be also used to answer the above question, but I have the
> > > >> impression now that the conversation stopped making any progress.
> > > >> So instead of waiting for partial metadata loading I propose to have
> > > >> an addition to the REST spec now to answer the question I raised above:
> > > >>
> > > >> a) boolean isLatest(TableIdentifier ident, String metadataLocation);
> > > >> b) String metadataLocation(TableIdentifier ident);
> > > >>
> > > >> Any of the above 2 approaches could help engines to decide if they
> > > >> have to invalidate/reload particular table metadata in the cache. I
> > > >> personally would go for option a) but would be open to hear other
> > > >> opinions.
> > > >>
> > > >> I'd like to know if the community could support me extending the REST
> > > >> spec with any of the 2 options.
> > > >>
> > > >> Regards,
> > > >> Gabor
> > > >>
> > > >>
> > 
