nevzheng commented on code in PR #11959:
URL: https://github.com/apache/gravitino/pull/11959#discussion_r3561912016
##########
catalogs/catalog-lakehouse-iceberg/src/main/java/org/apache/gravitino/catalog/lakehouse/iceberg/IcebergCatalog.java:
##########
@@ -68,6 +68,22 @@ public ViewCatalog asViewCatalog() {
return (ViewCatalog) ops();
}
+ @Override
+ public boolean shouldValidateConnectionForCreate() {
Review Comment:
Rory, can you elaborate on your concern?
Here's what I think you mean correct me if I'm off: Gravitino persists the
catalog into its own metastore on create, and today create never contacts the
remote server. Your concern is that by validating against the remote at create
time, we'd now fail the create if the remote REST server is down and
registering a federated catalog shouldn't require the remote to be reachable at
that exact moment.
My response: I'd like to propose a principle, Gravitino's metastore should
only store data known to be valid at create time.
[#11943](https://github.com/nevzheng/gravitino/issues/11943) shows the cost of
not having it: we accept a catalog that can never work, return 200, and it
fails later with a confusing NoSuchWarehouseException. That's a change from
today's behavior, and this PR implements it (I've updated it to make the intent
clearer). For now it's a focused change, Iceberg REST warehouse, but in
principle the same "validate before persist" idea extends to other backends and
other create-time checks.
How do you feel about that principle?
Conversely, if we decide the metastore may store unvalidated data
(federation pointers we haven't checked), that's a legitimate data-model choice
too, but we should document it explicitly. And under that model,
[#11943](https://github.com/nevzheng/gravitino/issues/11943) is actually WAI:
it's the expected consequence of storing without validating, not a bug.
Either way, I think the useful outcome is to pick the principle deliberately
and write it down, rather than leave it implicit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
