GitHub user honam-data added a comment to the discussion: When using Lance as the lakehouse format, does Fileset become redundant for unstructured data management?
Hi @jerryshao, thanks a lot for the detailed clarification — the distinction between Lance (structured table with schema) and Fileset (raw file container) makes sense. I have a follow-up question: you mentioned that Lance is used primarily for query and search, while Fileset doesn't process the files. But in a real AI pipeline, unstructured data carries a lot of metadata along with the raw content — things like file-level metadata (filename, size, format, upload time), business metadata (department, sensitivity level, compliance tags), and administrative metadata (owner, access history, lifecycle status). Would you recommend storing this metadata in the Lance table alongside the embeddings and content? Or should it stay in the Fileset layer? In other words, can/should a Lance table serve as the single source of truth for both the content AND the metadata of unstructured data, or is metadata management still best kept at the Fileset level? Curious how the community thinks about this metadata layer in the Lance + Fileset architecture. GitHub link: https://github.com/apache/gravitino/discussions/11953#discussioncomment-17693435 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
