I think we are going in the right direction. If we want to have a similar to Delta approach, we can do it easily.
The new Java implementation will provide an API to work with metadata for all the JVM connectors. Cpp library may provide the same but as a binary library. And applications (like spark, our planned CLI tool, top level cpp API, etc.) should be done using low-level "kernels". I think we should just discuss again how a kernel API should look like to provide a good balance between simplicity, codebase size, amount of dependencies and abilities to extend it. On Mon, 2024-08-12 at 08:53 +0000, Weibin Zeng wrote: > As I know (which may not correct) that the Delta kernel abstracts a > set of APIs for handling Parquet and JSON file formats, and then > implements a connector based on this foundation. > but I did not find how the kernel to process the metadata and > manifest data of table? > > And the Delta-kernel use java and rust as the basic implementation > look cool and make sense to me, and perhaps graphar could take > inspiration from this in the future. > > On 2024/08/02 10:17:26 Weibin Zeng wrote: > > Hi, Sem, I just read the blog about Delta Kernel and haven't read > > the code in detail yet > > But as you said, the concept of the `Delta kernel` is very close to > > our caseļ¼ since we have format protocol and want to provide SDKs > > for different ecosystem. > > > > thanks for bring the information again! > > > > Best > > Weibin Zeng > > > > > > > > > On 2024/07/28 15:12:41 Sem wrote: > > > Hello! > > > > > > I watched recently the talk about Delta format and found the > > > concept of > > > Delta Kernels very interesting. Instead of providing multiple > > > APIs, > > > they decided to provide so called kernels: extendable APIs with > > > default > > > implementations that can be used to create connectors. At the > > > moment > > > they are providing one kernel for JVM and one kernel for binary > > > connectors (in Rust). It is very close to our case. > > > > > > I think we can find a lot of useful tips by checking how it is > > > done in > > > Delta. > > > > > > The link to the talk: > > > https://www.databricks.com/dataaisummit/session/delta-kernel-anatomy-delta-lake-connector > > > Link to the code: > > > 1. JVM kernel: > > > https://github.com/delta-io/delta/tree/master/kernel > > > 2. Rust kernel: > > > https://github.com/delta-incubator/delta-kernel-rs > > > > > > A part of the documentation, related to kernels: > > > https://delta.io/blog/delta-kernel/ > > > > > > May we discuss it during the next community-sync? > > > > > > Best regards, > > > Sem > > > > > > > > > > ------------------------------------------------------------------- > > -- > > To unsubscribe, e-mail: dev-unsubscr...@graphar.apache.org > > For additional commands, e-mail: dev-h...@graphar.apache.org > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@graphar.apache.org > For additional commands, e-mail: dev-h...@graphar.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@graphar.apache.org For additional commands, e-mail: dev-h...@graphar.apache.org
