GitHub user honam-data created a discussion: When using Lance as the lakehouse
format, does Fileset become redundant for unstructured data management?
Background
I'm exploring Gravitino as a unified metadata governance layer for an AI/ML
pipeline. My data architecture is:
Raw unstructured data (PDFs, images, audio)
→ stored in S3 (MinIO)
→ built into Lance tables (with embeddings, metadata, vector indexes)
→ governed by Gravitino
>From what I understand:
1. Fileset is Gravitino's abstraction for managing non-tabular, schema-free
data (training datasets, raw files). It manages file locations, credentials,
and lifecycle on HDFS/S3/Local.
2. Lance (via lakehouse-generic catalog, format=lance) is Gravitino's
integration for the Lance columnar format — managing table schemas, columns,
and vector indexes.
The Confusion
In practice, Lance itself is a multimodal lakehouse format that already handles
the full unstructured data lifecycle:
- Raw files are stored in S3 (e.g., s3://lance-bucket/)
- Lance tables are built on top, providing typed columns, metadata, vector
embeddings, and indexes — all self-contained in the same S3 location
- Through Gravitino's Lance integration, I get schema management, permission
control, and REST API access
This means Lance already covers the Fileset use case end-to-end. I can manage
raw unstructured data entirely through Lance tables under Gravitino, without
ever touching Fileset.
The Question
1. Does Lance make Fileset redundant? When you're already using Lance as the
lakehouse format, what value does Fileset add? Or are they meant for different
stages of the data pipeline?
2. What's the intended relationship? Are Fileset and Lance complementary legs
of the same journey (raw ingestion → structured table), or are they independent
alternatives with overlapping scopes?
3. What's the practical guidance? When should a user choose Fileset vs. Lance
Table for managing unstructured data in a real AI/ML pipeline?
Why This Matters
As Lance adoption grows (it's increasingly positioned as the standard format
for AI/multimodal lakehouses), this architectural ambiguity could confuse
newcomers. Clarifying the positioning would help users decide which abstraction
to use when.
I'd love to hear from maintainers and the community — is there a design
rationale behind keeping these two abstractions separate, or is this something
that may evolve as the Lance integration matures?
GitHub link: https://github.com/apache/gravitino/discussions/11953
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
