GitHub user jerryshao added a comment to the discussion: When using Lance as the lakehouse format, does Fileset become redundant for unstructured data management?
Hi @honam-data I think Lance and fileset are two separate things. 1. Lance is still a table; it has schema. The typical way of using Lance is still a structured way, like query and search. We put Lance table under the management of the relational catalog, so it still appears as a relational table. 2. The fileset is quite different. It is like a container managing the raw files/objects under this container. We virtualize this container and use a name identifier to represent this container, to separate the understanding of physical location as well as the virtual name. 3. Besides, Lance is used to embed the raw data into a vector and stored in the file; this vector can be used for semantic similarity search. But, a fileset is just a raw container; it doesn't process the files and understand the content of the file, it just unfies the way of accessing the files. So, they're actually two different concepts and things. They will not replace each other. It depends on how you use it. Typically, your raw data (text/video/audio file) can be managed by a fileset and accessed by GVFS. And the embedded content can be stored in Lance. They can all be managed by Gravitino to build your own AI lakehouse. GitHub link: https://github.com/apache/gravitino/discussions/11953#discussioncomment-17617292 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
