To help folks catch up on the latest discussions and interpretation of
the spec, I have summarized everything we discussed so far at the top
of the proposal document (here
<https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0>).
I have slightly updated the proposal to be in sync with the new
interpretation to avoid confusion. In summary:
* Remove default-catalog and default-namespace fields from the view
spec completely.
* Hence, we do not attempt to define separate view-level default
catalogs or namespaces.
Instead:
* If a table identifier inside a view lacks a catalog qualifier,
engines should resolve it using the current engine catalog at query time.
* Reference table identifiers in the metadata exactly as they appear
in the view SQL text.
* If an identifier lacks the catalog part at creation, it should still
lack a catalog in the stored metadata.
* Store UUIDs alongside table identifiers whenever possible.
Thanks,
Walaa.
On Fri, Apr 25, 2025 at 5:18 PM Walaa Eldin Moustafa
<wa.moust...@gmail.com> wrote:
Thanks for the contribution Benny! +1 to the confusion the
fallback creates. Also just to be clear, at this point and after
clarifying the current spec intentions, I am convinced that we
should remove the default catalog and default namespace fields
altogether.
Thanks,
Walaa.
On Fri, Apr 25, 2025 at 5:13 PM Benny Chow <btc...@gmail.com> wrote:
I'd like to contribute my opinions on this:
- I don't particularly like the current behavior of "default
to the view's catalog when default-catalog is not set".
Fundamentally, I believe the intent of default-catalog and
default-namespace is there to help users write more concise SQL.
- spark session catalog is engine specific and I don't think
we should design something that says first use this catalog,
then that catalog.. or that catalog. For example, resolving
identifiers using default-catalog -> view's catalog -> session
catalog is not good.
- We gotta support non-Iceberg tables otherwise I see no value
in putting views in the catalog to share with other engines
- Interoperability between different engine types is very hard
due to dialect issues... so I think we should focus on
supporting different clusters of the same engine type on a
shared catalog. For example, AI and BI clusters on Spark
sharing the same views in a REST catalog.
Coincidentally, I think the ultimate solution is along the
lines of something Russell proposed last year:
https://lists.apache.org/thread/hoskfx8y3kvrcww52l4w9dxghp3pnlm7
We've been looking at this interoperable identifier problem
through the lens of catalog resolution but maybe the right
approach is really about templating.
I would extend Russell's idea to allow identifiers in a view
to span catalogs to support non-Iceberg tables. Also, the
default-catalog property could be templated as well.
Thoughts?
Benny
On Fri, Apr 25, 2025 at 4:02 PM Walaa Eldin Moustafa
<wa.moust...@gmail.com> wrote:
Thanks Steven! How do you recommend making Spark
implementation conform to the spec? Do we need Spark SQL
extensions and/or Spark catalog APIs for that?
How do you recommend reconciling the inconsistencies I
shared regarding many resolution methods not consistently
being followed in different scenarios (view vs child table
resolution, query vs view resolution)? Note these occur
when the default catalog is set to a non-null value. If it
helps, I can share concrete examples.
Thanks,
Walaa.
On Fri, Apr 25, 2025 at 3:52 PM Steven Wu
<stevenz...@gmail.com> wrote:
The core issue is on the fall back behavior when
`default-catalog` is
not defined. Current view spec says the fallback
should be the catalog
where the view is defined. It doesn't really matter
what the catalog
is named (catalogX) by the read engine.
- If a view refers to the tables in the same catalog,
this is a
non-ambiguous and reasonable fallback behavior.
- If a view refers to tables from another catalog,
catalog names
should be included in the reference name already. So
no ambiguity
there either.
Potential inconsistent naming of catalog is a separate
problem, which
Iceberg view spec probably cannot solve. We can only
recommend that
catalog should be named consistently across usage for
better
interoperability on name references.
This proposal is to change the fallback behavior to
engine's session
default catalog. I am not sure it is better than the
current fallback
behavior.
> Today’s Spark behavior explicitly differs from this
idea. Spark resolves table identifiers during view
creation using the session’s default catalog, not a
supplied `default-catalog`.
I would argue that is a Spark implementation issue for
not conforming
to the spec.
On Fri, Apr 25, 2025 at 1:17 PM Walaa Eldin Moustafa
<wa.moust...@gmail.com> wrote:
>
> Hi Jan,
>
> Thanks again for continuing the discussion. I want
to highlight a few fundamental issues around the
interpretation of default-catalog:
>
> Here is the real catch:
>
> * default-catalog cannot logically be defined at
view creation time. It would be circular: the view
needs to exist before its metadata (and hence
default-catalog) can exist. This is visible in Spark’s
implementation, where `default-catalog` is not used.
>
> * Introducing a creation-time default-catalog
setting would require extending SQL syntax and engine
APIs to promote it to a first-class view concept. This
would be intrusive, non-intuitive, and realistically
very difficult to standardize across engines.
>
> * Today’s Spark behavior explicitly differs from
this idea. Spark resolves table identifiers during
view creation using the session’s default catalog, not
a supplied `default-catalog`.
>
> * Hypothetically even if we patched in a
creation-time default-catalog, it would create an
inconsistent binding model between tables vs views
(early vs late), and between tables in views and in
queries (again early vs late). For example, views and
tables in queries can withstand default catalog
renames, but tables cannot when they are used inside
views -- it even applies to views inside views, which
makes this very hard to reason about considering nesting.
>
> Thanks,
> Walaa
>
> On Fri, Apr 25, 2025 at 7:00 AM Jan Kaul
<jank...@mailbox.org.invalid> wrote:
>>
>> @Walaa:
>>
>> I would argue that when you run a CREATE VIEW
statement the query engine knowns which catalog the
view is being created in. So even though we typically
use late binding to resolve the view catalog at query
time, it can also be used at creation time.
>>
>> The query engine would need to keep track of the
"view catalog" where the view is going to be created
in. It can use that catalog to resolve partial table
identifiers if "default-catalog" is not set.
>>
>> It can lead to some unintuitive behavior, where
partial identifiers in the view query resolve to a
different catalog compared to using them outside of a
view.
>>
>> CREATE VIEW catalogA.sales.monthly_orders AS SELECT
* from sales.orders;
>>
>> If the session default catalog is not "catalogA",
the "sales.orders" in the view query would not be the
same as just referencing "sales.orders" in a normal
SQL statement. This is because without a
"default-catalog", the catalog name of "sales.orders"
would default to "catalogA", which is the view's catalog.
>>
>> Thanks,
>>
>> Jan
>>
>> On 4/25/25 04:05, Manu Zhang wrote:
>>>
>>> For example, if we want to validate that the
tables referenced in the view exist, how can we do
that when default-catalog isn't defined, since the
view hasn't been created or loaded yet?
>>
>> I don't think this is related to view spec. How do
we validate that a table exists without a default
catalog, or do we always use the current session catalog?
>>
>> Thanks,
>> Manu
>>
>> On Fri, Apr 25, 2025 at 5:59 AM Walaa Eldin
Moustafa <wa.moust...@gmail.com> wrote:
>>>
>>> Hi Jan,
>>>
>>> I think we still share the same understanding.
Just to clarify: when I referred to late binding as
“similar” to the proposal, I was acknowledging the
distinction between view-level and table-level
resolution. But as you noted, both follow a late
binding model.
>>>
>>> That said, this still raises an interesting
question and a potential gap: if default-catalog is
only defined at query time, how should resolution work
during view creation? For example, if we want to
validate that the tables referenced in the view exist,
how can we do that when default-catalog isn't defined,
since the view hasn't been created or loaded yet?
>>>
>>> Thanks,
>>> Walaa.
>>>
>>> On Thu, Apr 24, 2025 at 7:02 AM Jan Kaul
<jank...@mailbox.org.invalid> wrote:
>>>>
>>>> Yes, I have the same understanding. The view
catalog is resolved at query time.
>>>>
>>>> As you mentioned before, it's good to distinguish
between the physical catalog and it's reference used
in SQL statements. The important part is that the
physical catalog of the view and the tables referenced
in it's definition stay consistent. You could create a
view in a given physical catalog by referring to it as
"catalogA", as in your first point. If you then, given
a different setup, refer to the same physical catalog
as "catalogB" in another session/environment, the
behavior should still work.
>>>>
>>>> I would however rephrase your last point. Late
binding applies to the view catalog name and by
extension to all partial table references when no
"default-catalog" is present. Resolving the view
catalog name at query time is not opposed to storing
the view metadata in a catalog.
>>>>
>>>> Or maybe I don't entirely understand what you mean.
>>>>
>>>> Thanks
>>>>
>>>> Jan
>>>>
>>>> On 4/24/25 00:32, Walaa Eldin Moustafa wrote:
>>>>
>>>> Hi Jan,
>>>>
>>>> > The view is executed when it's being referenced
in a SQL statement. That statement contains the
information for the query engine to resolve the
catalog of the view.
>>>>
>>>> If I’m understanding correctly, that means:
>>>>
>>>> * If the view is queried as SELECT * FROM
catalogA.namespace.view, then catalogA is considered
the view’s catalog.
>>>>
>>>> * If the same view is later queried as SELECT *
FROM catalogB.namespace.view (after renaming catalogA
to catalogB, and keeping everything else the same),
then catalogB becomes the view’s catalog.
>>>>
>>>> Is that interpretation correct? If so, it sounds
to me like the catalog is resolved at query time,
based on how the view is referenced, not from any
stored metadata. That would imply some sort of a late
binding behavior (similar to the proposal), as opposed
to using some catalog that "stores" the view definition.
>>>>
>>>> Thanks,
>>>> Walaa
>>>>
>>>> On Tue, Apr 22, 2025 at 11:01 AM Jan Kaul
<jank...@mailbox.org.invalid> wrote:
>>>>>
>>>>> Hi Walaa,
>>>>>
>>>>> Thanks for clarifying the aspects of
non-determinism. Let me try to address your questions.
>>>>>
>>>>> 1. This is my interpretation of the current
spec: The view is executed when it's being referenced
in a SQL statement. That statement contains the
information for the query engine to resolve the
catalog of the view. The query engine then uses that
information to fetch the view metadata from the
catalog. It also needs to temporarily keep track of
which catalog it used to fetch the view metadata. It
can then use that information to resolve the table
references in the views SQL definition in case no
default catalog is specified.
>>>>>
>>>>> 2. The important part is that the catalog can be
referenced at execution time. As long as that's the
case I would assume the view can be created in any
catalog.
>>>>>
>>>>>
>>>>> I think your point is really valuable because
the current specification can lead to some unintuitive
behavior. For example for the following statement:
>>>>>
>>>>> CREATE VIEW catalogA.sales.monthly_orders AS
SELECT * from sales.orders;
>>>>>
>>>>> If the session default catalog is not
"catalogA", the "sales.orders" in the view query would
not be the same as just referencing "sales.orders" in
a normal SQL statement. This is because without a
"default-catalog", the catalog name of "sales.orders"
would default to "catalogA".
>>>>>
>>>>>
>>>>> However, I like the current design of the view
spec, because it has the "closure" property. Because
of the fact that the "view catalog" has to be known
when executing a view, all the information required to
resolve the table identifiers is contained in the view
metadata (and the "view catalog"). I think that if you
make the identifier resolution dependent on external
parameters, it hinders portability.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Jan
>>>>>
>>>>> On 4/22/25 18:36, Walaa Eldin Moustafa wrote:
>>>>>
>>>>> Hi Jan,
>>>>>
>>>>> Thanks for the thoughtful feedback.
>>>>>
>>>>> I think it’s important we clarify a key point
before going deeper:
>>>>>
>>>>> Non-determinism is not caused by session
fallback behavior—it’s a fundamental limitation of
using table identifiers alone, regardless of whether
we use the current rule, the proposed fallback to the
session’s default catalog, or even early vs. late binding.
>>>>>
>>>>> The same fully qualified identifier (e.g.,
catalogA.namespace.table) can resolve to different
objects depending solely on engine-specific routing
logic or catalog aliases. So determinism isn’t
guaranteed just because an identifier is "fully
qualified." The only reliable anchor for identity is
the UUID. That’s why the proposed use of UUIDs is not
just a hardening strategy. It’s the actual fix for
correctness.
>>>>>
>>>>> To move the conversation forward, could you help
clarify two things in the context of the current spec:
>>>>>
>>>>> * Where in the metadata is the “view catalog”
stored, so that an engine knows to fall back to it if
default-catalog is null?
>>>>>
>>>>> * Are we even allowed to create views in the
session's default catalog (i.e., without specifying a
catalog) in the current Iceberg spec?
>>>>>
>>>>> These questions are important because if we
can’t unambiguously recover the "view catalog" from
metadata, then defaulting to it is problematic. And if
views can't be created in the default catalog, then
the fallback rule doesn’t generalize.
>>>>>
>>>>> Thanks,
>>>>> Walaa.
>>>>>
>>>>>
>>>>> On Tue, Apr 22, 2025 at 3:14 AM Jan Kaul
<jank...@mailbox.org.invalid> wrote:
>>>>>>
>>>>>> Hi Walaa,
>>>>>>
>>>>>> thank you for your proposal. If I understood
correctly, you proposal is composed of three parts:
>>>>>>
>>>>>> - session default catalog as fallback for
"default-catalog"
>>>>>>
>>>>>> - session default namespace as fallback for
"default-namepace"
>>>>>>
>>>>>> - Late binding + UUID validation
>>>>>>
>>>>>> I have some comments regarding these points.
>>>>>>
>>>>>>
>>>>>> 1. Session default catalog as fallback for
"default-catalog"
>>>>>>
>>>>>> Introducing a behavior that depends on the
current session setup is in my opinion the definition
of "non-determinism". You could be running the same
query-engine and catalog-setup on different days, with
different default session catalogs (which is rather
common), and would be getting different results.
>>>>>>
>>>>>> Whereas with the current behavior, the view
always produces the same results. The current behavior
has some rough edges in very niche use cases but I
think is solid for most uses cases.
>>>>>>
>>>>>> 2. Session default namespace as fallback for
"default-namespace"
>>>>>>
>>>>>> Similar to the above.
>>>>>>
>>>>>> 3. Late binding + UUID validation
>>>>>>
>>>>>> If I understand it correctly, the current
implementation already uses late binding.
>>>>>>
>>>>>> Generally, having UUID validation makes the
setup more robust. Which is great. However, having
UUID validation still requires us to have a portable
table identifier specification. Even if we have the
UUIDs of the referenced tables from the view, there
simply isn't an interface that let's us use those
UUIDs. The catalog interface is defined in terms of
table identifiers.
>>>>>>
>>>>>> So we always require a working catalog setup
and suiting table identifiers to obtain the table
metadata. We can use the UUIDs to verify if we loaded
the correct table. But this can only be done after we
used some identifier. Which means there is no way of
using UUIDs without a functioning catalog/identifier
setup.
>>>>>>
>>>>>>
>>>>>> In conclusion, I prefer the current behavior
for "default-catalog" because it is more deterministic
in my opinion. And I think the current spec does a
good job for multi-engine table identifier resolution.
I see the UUID validation more of an additional
hardening strategy.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Jan
>>>>>>
>>>>>> On 4/21/25 17:38, Walaa Eldin Moustafa wrote:
>>>>>>
>>>>>> Thanks Renjie!
>>>>>>
>>>>>> The existing spec has some guidance on
resolving catalogs on the fly already (to address the
case of view text with table identifiers missing the
catalog part). The guidance is to use the catalog
where the view is stored. But I find this rule hard to
interpret or use. The catalog itself is a logical
construct—such as a federated catalog that delegates
to multiple physical backends (e.g., HMS and REST). In
such cases, the catalog (e.g., `my_catalog` in
`my_catalog.namespace1.table1`) doesn’t physically
store the tables; it only routes requests to
underlying stores. Therefore, defaulting identifier
resolution based on the catalog where the view is
"stored" doesn’t align with how catalogs actually
behave in practice.
>>>>>>
>>>>>> Thanks,
>>>>>> Walaa.
>>>>>>
>>>>>> On Sun, Apr 20, 2025 at 11:17 PM Renjie Liu
<liurenjie2...@gmail.com> wrote:
>>>>>>>
>>>>>>> Hi, Walaa:
>>>>>>>
>>>>>>> Thanks for the proposal.
>>>>>>>
>>>>>>> I've reviewed the doc, but in general I have
some concerns with resolving catalog names on the fly
with query engine defined catalog names. This
introduces some flexibility at first glance, but also
makes misconfiguration difficult to explain.
>>>>>>>
>>>>>>> But I agree with one part that we should store
resolved table uuid in view metadata, as table/view
renaming may introduce errors that's difficult to
understand for user.
>>>>>>>
>>>>>>> On Sat, Apr 19, 2025 at 3:02 AM Walaa Eldin
Moustafa <wa.moust...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Hi Everyone,
>>>>>>>>
>>>>>>>> Looking forward to keeping up the momentum
and closing out the MV spec as well. I’m hoping we can
proceed to a vote next week.
>>>>>>>>
>>>>>>>> Here is a summary in case that helps. The
proposal outlines a strategy for handling table
identifiers in Iceberg view metadata, with the goal of
ensuring correctness, portability, and engine
compatibility. It recommends resolving table
identifiers at read time (late binding) rather than
creation time, and introduces UUID-based validation to
maintain identity guarantees across engines, or
sessions. It also revises how default-catalog and
default-namespace are handled (defaulting both to the
session context if not explicitly set) to better align
with engine behavior and improve cross-engine
interoperability.
>>>>>>>>
>>>>>>>> Please let me know your thoughts.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Walaa.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Apr 16, 2025 at 2:03 PM Walaa Eldin
Moustafa <wa.moust...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> Thanks Eduard and Sung! I have addressed the
comments.
>>>>>>>>>
>>>>>>>>> One key point to keep in mind is that
catalog names in the spec refer to logical
catalogs—i.e., the first part of a three-part
identifier. These correspond to Spark's DataSourceV2
catalogs, Trino connectors, and similar constructs.
This is a level of abstraction above physical
catalogs, which are not referenced or used in the view
spec. The reason is that table identifiers in the view
definition/text itself refer to logical catalogs, not
physical ones (since they interface directly with the
engine and not a specific metastore).
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Walaa.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Apr 16, 2025 at 6:15 AM Sung Yun
<sungwy...@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> Thank you Walaa for the proposal. I think
view portability is a very important topic for us to
continue discussing as it relies on many assumptions
within the data ecosystem for it to function like
you've highlighted well in the document.
>>>>>>>>>>
>>>>>>>>>> I've added a few comments around how this
may impact the permission questions the engines will
be asking, and whether that is the desired behavior.
>>>>>>>>>>
>>>>>>>>>> Sung
>>>>>>>>>>
>>>>>>>>>> On Wed, Apr 16, 2025 at 7:32 AM Eduard
Tudenhöfner <etudenhoef...@apache.org> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Thanks Walaa for tackling this problem.
I've added a few comments to get a better
understanding of how this will look like in the actual
implementation.
>>>>>>>>>>>
>>>>>>>>>>> Eduard
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Apr 15, 2025 at 7:09 PM Walaa
Eldin Moustafa <wa.moust...@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Everyone,
>>>>>>>>>>>>
>>>>>>>>>>>> Starting this thread to resume our
discussion on how to reference table identifiers from
Iceberg metadata, a key aspect of the view
specification, particularly in relation to the MV
(materialized view) extensions.
>>>>>>>>>>>>
>>>>>>>>>>>> I had the chance to speak offline with a
few community members to better understand how the
current spec is being interpreted. Those conversations
served as inputs to a new proposal on how table
identifier references could be represented in metadata.
>>>>>>>>>>>>
>>>>>>>>>>>> You can find the proposal here [1]. I
look forward to your feedback and working together to
move this forward so we can finalize the MV spec as well.
>>>>>>>>>>>>
>>>>>>>>>>>> [1]
https://docs.google.com/document/d/1-I2v_OqBgJi_8HVaeH1u2jowghmXoB8XaJLzPBa_Hg8/edit?tab=t.0
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Walaa.
