Thank you Everyone for the inputs and thanks Kevin for describing the "we have a use case" part while I was offline! :)
Using `write.metadata.path` makes sense to me too. Yufei, since you had the preference for the other (location param based) approach, do you think you can live with keeping `write.metadata.path`? Best Regards, Gabor Ryan Blue <[email protected]> ezt írta (időpont: 2026. jan. 29., Cs, 1:49): > Thanks, Kevin. If you need to customize the metadata path within the view > location, then it makes more sense to me to use `write.metadata.path`. > > On Tue, Jan 27, 2026 at 4:39 PM Kevin Liu <[email protected]> wrote: > >> Thanks for starting this thread, and for all the great discussions. >> >> I can chime in on use cases from the Microsoft OneLake perspective. We >> have a requirement to customize the metadata path of the view metadata json >> file. We want the file path to have a specific naming convention that can >> be an arbitrary subdirectory name. >> >> Right now, using the `location` parameter, `metadata/` is added >> automatically as the subdirectory for the json file [1], ie >> {location}/metadata/view1.metadata.json >> We want to customize the name of the subdirectory with >> `write.metadata.path` [2], so the view will be stored in >> {location}/{write.metadata.path}/view1.metadata.json >> >> In summary, I prefer option 2 to give writers more control over the exact >> path of the metadata json. And this also follows the convention for >> creating an Iceberg table with the write.metadata.path property. >> >> Best, >> Kevin Liu >> >> >> [1] >> https://github.com/apache/iceberg/blob/1bf44d9fed8b99dd73115ec31659266248cbb363/core/src/main/java/org/apache/iceberg/view/BaseViewOperations.java#L173 >> [2] >> https://github.com/apache/iceberg/blob/1bf44d9fed8b99dd73115ec31659266248cbb363/core/src/main/java/org/apache/iceberg/view/BaseViewOperations.java#L166-L169 >> >> >> On Tue, Jan 27, 2026 at 2:43 PM Ryan Blue <[email protected]> wrote: >> >>> I think it is okay to go with option 1, but I don't see a justification >>> for why a change here is needed. Why break with existing convention when it >>> should not matter to users what location is used for metadata? >>> >>> I'd prefer option 2, to let people configure this if they choose. But I >>> think the best solution is to do nothing and encourage people not to care >>> about where metadata files end up. Looks like the justification is "we have >>> use cases". Could you share what those are? >>> >>> On Mon, Jan 26, 2026 at 1:27 PM Yufei Gu <[email protected]> wrote: >>> >>>> Given that the Iceberg community is moving toward a catalog based >>>> approach, I think option 1 should work reasonably well. From a reader >>>> perspective, there should be no behavior change, since the metadata path is >>>> always resolved from the catalog, regardless of whether the underlying path >>>> includes a metadata directory or not. For writers, as long as we converge >>>> on the convention of not implicitly adding a metadata subdirectory, this >>>> also seems acceptable. This is already how tables behave today. We have not >>>> enforced a strict rule for table metadata paths in the spec, and different >>>> impls have been able to interoperate without issues. >>>> >>>> Yufei >>>> >>>> >>>> On Fri, Jan 23, 2026 at 11:35 AM Gábor Kaszab <[email protected]> >>>> wrote: >>>> >>>>> Thank you All for your responses! >>>>> Let me summarize what we have so far to see what potential next steps >>>>> we can make. >>>>> >>>>> 1) Change 'location' property to not create a 'metadata/' subfolder >>>>> - Doesn't seem to be a great issue that already written tables >>>>> have 'metadata/' subfolder, while newer ones won't >>>>> - Easy to implement in Iceberg code >>>>> - Doing this will let us drop 'write.metadata.path' property >>>>> My concern: >>>>> - We only have control on Iceberg code, that means non-REST >>>>> catalogs, but different REST catalog implementations could still create >>>>> paths the old way. >>>>> - Is there a way we can enforce this? Adding to spec seems >>>>> overkill. >>>>> >>>>> 2) De-deprecate 'write.metadata.path' >>>>> - Using this will let us to avoid having 'metadata/' folder in >>>>> the path >>>>> - Doc >>>>> <https://iceberg.apache.org/docs/latest/view-configuration/> says >>>>> "Base location for metadata files". Seems ambiguous if this means that we >>>>> can have subfolders or not >>>>> - Seems common sense to not add 'metadata/' folder, but there >>>>> seems to be no strict guarantee that all REST catalog implementations >>>>> follow this >>>>> >>>>> WDYT about next steps? Maybe go with 2) and be more explicit in the >>>>> docs? >>>>> >>>>> Best Regards, >>>>> Gabor Kaszab >>>>> >>>>> Ryan Blue <[email protected]> ezt írta (időpont: 2026. jan. 16., P, >>>>> 1:03): >>>>> >>>>>> I think that the "non-catalog option" means that it works when you're >>>>>> pointed at a particular metadata.json file. Commits always have to go >>>>>> through some catalog-based mechanism (now that Hadoop/FS commits are >>>>>> deprecated in the spec). >>>>>> >>>>>> On Thu, Jan 15, 2026 at 1:23 AM Péter Váry < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi Yufei, >>>>>>> >>>>>>> I think keeping the non-catalog option available makes sense, and >>>>>>> adding an extra check for this non-standard approach doesn’t seem like a >>>>>>> big issue to me. That said, the community consensus appears to be >>>>>>> gradually >>>>>>> shifting away from the non-catalog approach. If you look at the REST >>>>>>> planning mode discussion [1], we’re even considering Iceberg tables >>>>>>> without >>>>>>> any metadata stored on the underlying storage. It might be worth >>>>>>> defining a >>>>>>> consistent, general approach for decisions like these. >>>>>>> >>>>>>> Thanks, >>>>>>> Peter >>>>>>> >>>>>>> [1] - [DISCUSS] REST: Scan Planning mode - >>>>>>> https://lists.apache.org/thread/z1g4y8b4ogdrn0jjtjlgg7yjgxdbzpvg >>>>>>> >>>>>>> Yufei Gu <[email protected]> ezt írta (időpont: 2026. jan. 15., >>>>>>> Cs, 0:17): >>>>>>> >>>>>>>> Peter, >>>>>>>> Based on the discussion(sorry, I don't have a link to share) so >>>>>>>> far, the community conclusion is that we should always keep the non >>>>>>>> catalog >>>>>>>> option available. In other words, even catalogs are the recommended >>>>>>>> way to >>>>>>>> locate view metadata, non catalog based access remains a supported >>>>>>>> option. >>>>>>>> This gives us flexibility to support different deployment models. >>>>>>>> >>>>>>>> Yufei >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Jan 14, 2026 at 2:20 AM Péter Váry < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> > For example, when locating the view metadata.json, clients would >>>>>>>>> need to check two possible locations, "/location" and >>>>>>>>> "/location/metadata". >>>>>>>>> >>>>>>>>> Do I understand correctly that users are expected to use the >>>>>>>>> Catalog to locate metadata.json? In my opinion they should not rely on >>>>>>>>> filenames to locate views.” >>>>>>>>> >>>>>>>>> Yufei Gu <[email protected]> ezt írta (időpont: 2026. jan. >>>>>>>>> 14., Sze, 3:49): >>>>>>>>> >>>>>>>>>> Thanks Gabor for raising this. The alternative approach is >>>>>>>>>> appealing, as it avoids supporting the same behavior through multiple >>>>>>>>>> properties. My main concern is potential impact on existing use >>>>>>>>>> cases. For >>>>>>>>>> example, when locating the view metadata.json, clients would need to >>>>>>>>>> check >>>>>>>>>> two possible locations, "/location" and "/location/metadata". This >>>>>>>>>> may not >>>>>>>>>> be a significant issue, since the spec does not mandate the metadata >>>>>>>>>> suffix, but it does introduce incompatibility with the existing >>>>>>>>>> convention. >>>>>>>>>> It would be helpful for others to chime in and share view use cases >>>>>>>>>> that >>>>>>>>>> rely on the "metadata" suffix today, and POV on whether it's OK to >>>>>>>>>> adapt >>>>>>>>>> the change. >>>>>>>>>> >>>>>>>>>> Yufei >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Jan 13, 2026 at 5:23 AM Gábor Kaszab < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hey Iceberg Community, >>>>>>>>>>> >>>>>>>>>>> *TLDR* >>>>>>>>>>> I'd like to de-deprecate the 'write.metadata.path' view property >>>>>>>>>>> <https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/view/ViewProperties.java> >>>>>>>>>>> and would like to see if there are any objections. >>>>>>>>>>> >>>>>>>>>>> *Details* >>>>>>>>>>> We have use-cases where it's necessary to configure tables and >>>>>>>>>>> views to have custom paths for metadata, in particular not to >>>>>>>>>>> enforce >>>>>>>>>>> adding the metadata/ folder under the hood. For both tables and >>>>>>>>>>> views there >>>>>>>>>>> is 'write.metadata.path' property that could achieve this. However, >>>>>>>>>>> for >>>>>>>>>>> views this is deprecated now. >>>>>>>>>>> >>>>>>>>>>> After talking to Yufei, the reason for this property being >>>>>>>>>>> deprecated is that for views this is somewhat duplicated >>>>>>>>>>> functionality with >>>>>>>>>>> the 'location' property seemingly serving the same purpose. There >>>>>>>>>>> are some >>>>>>>>>>> differences though: >>>>>>>>>>> - 'location': This gives the root folder of the view and >>>>>>>>>>> metadata would be written under 'location'/metadata folder. >>>>>>>>>>> - 'write.metadata.path': This has higher precedence over >>>>>>>>>>> 'location' and metadata would be written directly into >>>>>>>>>>> 'write.metadata.path' without the metadata/ folder. >>>>>>>>>>> >>>>>>>>>>> An *alternative approach* could be to drop >>>>>>>>>>> 'write.metadata.path' and change the behaviour of 'location' to not >>>>>>>>>>> create >>>>>>>>>>> a metadata/ folder under the hood. The view spec doesn't mention >>>>>>>>>>> explicitly >>>>>>>>>>> that metadata/ folder is a requirement, however, it gives an >>>>>>>>>>> example >>>>>>>>>>> <https://iceberg.apache.org/view-spec/#appendix-a-an-example> >>>>>>>>>>> where it is present and also mentions that it is there to follow the >>>>>>>>>>> structure of the tables. So changing this would be confusing at >>>>>>>>>>> least, but >>>>>>>>>>> could be considered a breaking change too. I wouldn't go this way. >>>>>>>>>>> >>>>>>>>>>> So long story short, I'd like to keep 'write.metadata.path' for >>>>>>>>>>> views despite the similar purpose as 'location', to avoid >>>>>>>>>>> generating the >>>>>>>>>>> metadata/ folder. >>>>>>>>>>> >>>>>>>>>>> Any feedback is appreciated! >>>>>>>>>>> Gabor >>>>>>>>>>> >>>>>>>>>>
