As I know (which may not correct) that the Delta kernel abstracts a set of APIs for handling Parquet and JSON file formats, and then implements a connector based on this foundation. but I did not find how the kernel to process the metadata and manifest data of table?
And the Delta-kernel use java and rust as the basic implementation look cool and make sense to me, and perhaps graphar could take inspiration from this in the future. On 2024/08/02 10:17:26 Weibin Zeng wrote: > Hi, Sem, I just read the blog about Delta Kernel and haven't read the code in > detail yet > But as you said, the concept of the `Delta kernel` is very close to our caseļ¼ > since we have format protocol and want to provide SDKs for different > ecosystem. > > thanks for bring the information again! > > Best > Weibin Zeng > > > > > On 2024/07/28 15:12:41 Sem wrote: > > Hello! > > > > I watched recently the talk about Delta format and found the concept of > > Delta Kernels very interesting. Instead of providing multiple APIs, > > they decided to provide so called kernels: extendable APIs with default > > implementations that can be used to create connectors. At the moment > > they are providing one kernel for JVM and one kernel for binary > > connectors (in Rust). It is very close to our case. > > > > I think we can find a lot of useful tips by checking how it is done in > > Delta. > > > > The link to the talk: > > https://www.databricks.com/dataaisummit/session/delta-kernel-anatomy-delta-lake-connector > > Link to the code: > > 1. JVM kernel: https://github.com/delta-io/delta/tree/master/kernel > > 2. Rust kernel: https://github.com/delta-incubator/delta-kernel-rs > > > > A part of the documentation, related to kernels: > > https://delta.io/blog/delta-kernel/ > > > > May we discuss it during the next community-sync? > > > > Best regards, > > Sem > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@graphar.apache.org > For additional commands, e-mail: dev-h...@graphar.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@graphar.apache.org For additional commands, e-mail: dev-h...@graphar.apache.org
