Hi Ryan, Apologies for the noise.
Jack and Walaa, let's move any conversations to the Discussions <https://github.com/onetable-io/onetable/discussions> board on github for the project. Also feel free to reach out to me directly if you prefer. I'm personally looking forward to learning more about the Rest Catalog and making sure we integrate well with it from OneTable. -Tim Brown On Wed, Dec 6, 2023 at 11:55 AM Ryan Blue <b...@tabular.io> wrote: > I'm not sure that this is the right place for a discussion about the > merits of their approach. > > This list is for Iceberg development. I encourage anyone interested to > follow up on the appropriate incubator list rather than here. > > I also think it's debatable whether advertising other projects is helpful > or wanted here, but I'd rather not add to the noise either way. > > Ryan > > On Tue, Dec 5, 2023 at 8:36 PM Jack Ye <yezhao...@gmail.com> wrote: > >> I recently did an analysis of the OneTable project, overall it made me a >> bit confused. >> >> From an end user's perspective, no one really wants to use all these 3 >> formats, and most companies do not have the engineering resources to >> maintain a stack of all these 3 formats. Eventually people pick one and >> just stick with it. >> >> If the goal is to provide a converter, then individual communities have >> developed different tools, such as Delta's Uniform, Iceberg's snapshot and >> migrate procedures, Hudi's bootstrap methods. The advantage of those tools >> is that the specific community knows the best way to convert a foreign data >> source to its native format, and can declare compatibility and fail >> whenever necessary. It is not bounded to the expressiveness of an internal >> data model like OneTable, OneField, OneSchema, etc. >> >> If the goal is format unification, at least for me being in the Iceberg >> community with a bit bias, a more straightforward way to achieve the goal >> is to extend the feature of "Iceberg external tables", where we can map >> Hive, Delta, Hudi and other table formats directly to Iceberg format behind >> a REST catalog, and make that readable. This is kind of related to a >> recent email thread I sent regarding the EXTERNAL/MANAGED syntax >> <https://lists.apache.org/thread/ohqfvhf4wofzkhrvff1lxl58blh432o6>. And >> linking back to this thread, that essentially makes Iceberg the unified >> format, and we are actually pretty close to achieving that. With this >> approach, you get not just conversion, you can (1) not do physical metadata >> conversion but directly convert table metadata at runtime to Iceberg data >> model, (2) query all the tables using a single unified Iceberg connector in >> all supported engines, and (3) it is a very standardized external table >> concept that all database system folks immediately understand. >> >> This makes me feel that we are trying to make OneTable a new table format >> without saying it is a new table format. Although the Apache Incubation >> proposal clearly says "OneTable is NOT a new table format", it is hard for >> me to envision a long-term roadmap that does not eventually make it a table >> format, with connectors and data maintenance features built directly >> against this internal model, which is kind of feels like what the >> commercial entity OneHouse is trying to do right now, but maybe I am wrong. >> >> What do you think? >> >> Best, >> Jack Ye >> >> On Tue, Dec 5, 2023 at 3:30 PM Jesús Camacho Rodríguez < >> jcama...@apache.org> wrote: >> >>> Currently, there is no established group discussions. The project was >>> recently open-sourced, and communication is currently done through GitHub. >>> (If the project is accepted into the ASF incubator, mailing lists will be >>> created). If you're interested in regular meetings, feel free to suggest it >>> to the community on GitHub. >>> >>> Thanks, >>> Jesús >>> >>> >>> On 2023/12/05 06:30:38 Gaurav Agarwal wrote: >>> > HI >>> > Thanks for this mail , I would like to know is there any group >>> discussion >>> > also happened or any call to discuss the issues. >>> > >>> > thanks >>> > >>> > >>> > On Tue, Dec 5, 2023 at 9:29 AM Walaa Eldin Moustafa < >>> wa.moust...@gmail.com> >>> > wrote: >>> > >>> > > Thanks Jesus for sharing OneTable. Looks like it touches upon some >>> of the >>> > > topics we discussed in the Rise of Table Formats panel at VLDB >>> > > <https://ceur-ws.org/Vol-3462/CDMS18.pdf> back in September. I was >>> > > browsing through the source code, and I ran into the OneField >>> > > < >>> https://github.com/onetable-io/onetable/blob/main/api/src/main/java/io/onetable/model/schema/OneField.java> >>> class >>> > > and noticed it has support for default values, which is good, but in >>> the >>> > > Iceberg spec, there are two default values >>> > > <https://iceberg.apache.org/spec/#default-values> (more details in >>> the >>> > > spec and respective PR). I was pointing this out as an example of >>> small >>> > > nuances that can differ from one format to another and was wondering >>> how >>> > > OneTable is planning to bridge them? >>> > > >>> > > Thanks, >>> > > Walaa. >>> > > >>> > > >>> > >>> >> > > -- > Ryan Blue > Tabular >
