This is an automated email from the ASF dual-hosted git repository.
FANNG1 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new 421b075445 [#12382] feat(lance): Support AddColumn via Gravitino API
(#12383)
421b075445 is described below
commit 421b07544549fb08a945255ddeb759ef8fe50933
Author: godbiao <[email protected]>
AuthorDate: Mon Sep 21 12:16:10 2026 +0800
[#12382] feat(lance): Support AddColumn via Gravitino API (#12383)
### What changes were proposed in this pull request?
This PR adds first-phase `AddColumn` support for Lance tables through
the Gravitino table API.
The implementation:
- Supports nullable, top-level, append-only columns without default
values or auto-increment.
- Batches multiple `AddColumn` changes into one `Dataset.addColumns`
call.
- Uses Lance's native `NULL` backfill for existing rows.
- Hydrates declared or empty stored metadata before applying the
incremental column changes.
- Reuses `super.alterTable` to persist the requested columns,
`lance.version`, and remove `lance.declared`.
The implementation follows the existing Lance alter-table consistency
model. It does not introduce custom metadata CAS, physical rollback, or
strict schema reconstruction.
### Why are the changes needed?
Lance tables currently cannot add columns through the Gravitino table
API.
This change provides initial AddColumn support while keeping Lance as
the source of truth and keeping broader Lance-to-Gravitino schema
reconciliation outside the scope of this PR.
Fix: #12382
### Does this PR introduce _any_ user-facing change?
Yes.
Users can add nullable, top-level columns to Lance tables through the
Gravitino table API. Existing rows are backfilled with `NULL`.
The Lance REST `/add_columns` endpoint is not included in this change.
### How was this patch tested?
- Added unit tests for batched AddColumn, validation, declared/empty
metadata hydration, and zero-column declared tables.
- Added integration coverage verifying that historical rows are
backfilled with `NULL` and multiple columns produce one Lance version.
- Added integration coverage for direct AddColumn on a declared table
without a preceding `loadTable`.
Commands:
- `./gradlew spotlessApply`
- `./gradlew :catalogs:catalog-lakehouse-generic:test --tests
org.apache.gravitino.catalog.lakehouse.lance.TestLanceTableOperations
-PskipITs`
- `./gradlew :catalogs:catalog-lakehouse-generic:check -PskipITs`
---
.../lakehouse/lance/LanceTableOperations.java | 34 +++++++++-
.../lakehouse/lance/TestLanceTableOperations.java | 31 +++++++++
.../test/CatalogGenericCatalogLanceIT.java | 76 ++++++++++++++++++++++
docs/lakehouse-generic-lance-table.md | 37 ++++++++++-
4 files changed, 175 insertions(+), 3 deletions(-)
diff --git
a/catalogs/catalog-lakehouse-generic/src/main/java/org/apache/gravitino/catalog/lakehouse/lance/LanceTableOperations.java
b/catalogs/catalog-lakehouse-generic/src/main/java/org/apache/gravitino/catalog/lakehouse/lance/LanceTableOperations.java
index 5d9d65cd96..4d9c3dceda 100644
---
a/catalogs/catalog-lakehouse-generic/src/main/java/org/apache/gravitino/catalog/lakehouse/lance/LanceTableOperations.java
+++
b/catalogs/catalog-lakehouse-generic/src/main/java/org/apache/gravitino/catalog/lakehouse/lance/LanceTableOperations.java
@@ -825,7 +825,37 @@ public class LanceTableOperations extends
ManagedTableOperations {
LancePropertiesUtils.resolveLanceStorageOptions(catalogProperties,
table.properties());
try (Dataset dataset = openDataset(location, storageOptions)) {
for (TableChange change : changes) {
- if (change instanceof TableChange.DeleteColumn deleteColumn) {
+ if (change instanceof TableChange.AddColumn addColumn) {
+ String[] fieldName = addColumn.fieldName();
+ Preconditions.checkArgument(
+ fieldName.length == 1,
+ "Lance only supports adding top-level columns: %s",
+ String.join(".", fieldName));
+ String columnName = fieldName[0];
+ Preconditions.checkArgument(
+ addColumn.isNullable(),
+ "Lance only supports adding nullable columns because existing
rows are backfilled "
+ + "with null: %s",
+ columnName);
+ Preconditions.checkArgument(
+
TableChange.ColumnPosition.defaultPos().equals(addColumn.getPosition()),
+ "Lance only supports appending new columns: %s",
+ columnName);
+ Preconditions.checkArgument(
+ !addColumn.isAutoIncrement(),
+ "Lance does not support adding auto-increment columns: %s",
+ columnName);
+ Preconditions.checkArgument(
+ addColumn.getDefaultValue() == null
+ || addColumn.getDefaultValue().equals(DEFAULT_VALUE_NOT_SET),
+ "Lance does not support default values when adding columns: %s",
+ columnName);
+
+ Field field =
+ LanceDataTypeConverter.CONVERTER.toArrowField(
+ columnName, addColumn.getDataType(), true);
+ dataset.addColumns(List.of(field));
+ } else if (change instanceof TableChange.DeleteColumn deleteColumn) {
dataset.dropColumns(List.of(String.join(".",
deleteColumn.fieldName())));
} else if (change instanceof TableChange.AddIndex addIndex) {
IndexType indexType = IndexType.valueOf(addIndex.getType().name());
@@ -847,7 +877,7 @@ public class LanceTableOperations extends
ManagedTableOperations {
.build();
dataset.alterColumns(List.of(lanceColumnAlter));
} else {
- // Currently, only column drop/rename and index addition are
supported.
+ // Currently, only column add/drop/rename and index addition are
supported.
// TODO: Support change column type once we have a clear knowledge
about the means of
// castTo in Lance.
throw new UnsupportedOperationException(
diff --git
a/catalogs/catalog-lakehouse-generic/src/test/java/org/apache/gravitino/catalog/lakehouse/lance/TestLanceTableOperations.java
b/catalogs/catalog-lakehouse-generic/src/test/java/org/apache/gravitino/catalog/lakehouse/lance/TestLanceTableOperations.java
index 3556337d15..890c718a4d 100644
---
a/catalogs/catalog-lakehouse-generic/src/test/java/org/apache/gravitino/catalog/lakehouse/lance/TestLanceTableOperations.java
+++
b/catalogs/catalog-lakehouse-generic/src/test/java/org/apache/gravitino/catalog/lakehouse/lance/TestLanceTableOperations.java
@@ -56,6 +56,7 @@ import org.apache.gravitino.meta.TableEntity;
import org.apache.gravitino.rel.Column;
import org.apache.gravitino.rel.Table;
import org.apache.gravitino.rel.TableChange;
+import org.apache.gravitino.rel.expressions.literals.Literals;
import org.apache.gravitino.rel.expressions.sorts.SortOrder;
import org.apache.gravitino.rel.expressions.transforms.Transform;
import org.apache.gravitino.rel.indexes.Index;
@@ -798,6 +799,34 @@ public class TestLanceTableOperations {
.update(eq(ident), eq(TableEntity.class), eq(Entity.EntityType.TABLE),
any());
}
+ @Test
+ public void testAddColumnRejectsUnsupportedOptions() {
+ Table table = mock(Table.class);
+ when(table.properties()).thenReturn(Map.of(Table.PROPERTY_LOCATION,
"location"));
+ Dataset dataset = mock(Dataset.class);
+ Mockito.doReturn(dataset).when(lanceTableOps).openDataset("location",
Map.of());
+
+ List<TableChange> unsupportedChanges =
+ List.of(
+ TableChange.addColumn(new String[] {"parent", "nested"},
Types.StringType.get()),
+ TableChange.addColumn(new String[] {"required"},
Types.StringType.get(), false),
+ TableChange.addColumn(
+ new String[] {"first"}, Types.StringType.get(),
TableChange.ColumnPosition.first()),
+ TableChange.addColumn(
+ new String[] {"sequence"}, Types.LongType.get(), null, null,
true, true),
+ TableChange.addColumn(
+ new String[] {"with_default"},
+ Types.IntegerType.get(),
+ Literals.integerLiteral(1)));
+
+ for (TableChange change : unsupportedChanges) {
+ Assertions.assertThrows(
+ IllegalArgumentException.class,
+ () -> lanceTableOps.handleLanceTableChange(table, new TableChange[]
{change}));
+ }
+ verify(dataset, never()).addColumns(anyList());
+ }
+
@Test
public void testHandleLanceTableChangeRespectsOrder() {
Table table = mock(Table.class);
@@ -811,6 +840,7 @@ public class TestLanceTableOperations {
TableChange[] changes =
new TableChange[] {
+ TableChange.addColumn(new String[] {"added"},
Types.StringType.get()),
TableChange.renameColumn(new String[] {"old"}, "renamed"),
TableChange.addIndex(Index.IndexType.SCALAR, "idx_renamed", new
String[][] {{"renamed"}}),
TableChange.deleteColumn(new String[] {"renamed"}, false)
@@ -820,6 +850,7 @@ public class TestLanceTableOperations {
Assertions.assertEquals(7L, returnedVersion);
InOrder inOrder = Mockito.inOrder(dataset);
+ inOrder.verify(dataset).addColumns(List.of(Field.nullable("added", new
ArrowType.Utf8())));
inOrder.verify(dataset).alterColumns(anyList());
inOrder.verify(dataset).createIndex(any(IndexOptions.class));
inOrder.verify(dataset).dropColumns(anyList());
diff --git
a/catalogs/catalog-lakehouse-generic/src/test/java/org/apache/gravitino/catalog/lakehouse/lance/integration/test/CatalogGenericCatalogLanceIT.java
b/catalogs/catalog-lakehouse-generic/src/test/java/org/apache/gravitino/catalog/lakehouse/lance/integration/test/CatalogGenericCatalogLanceIT.java
index 277ed1ec6b..64ccdbd123 100644
---
a/catalogs/catalog-lakehouse-generic/src/test/java/org/apache/gravitino/catalog/lakehouse/lance/integration/test/CatalogGenericCatalogLanceIT.java
+++
b/catalogs/catalog-lakehouse-generic/src/test/java/org/apache/gravitino/catalog/lakehouse/lance/integration/test/CatalogGenericCatalogLanceIT.java
@@ -422,6 +422,82 @@ public class CatalogGenericCatalogLanceIT extends BaseIT {
RuntimeException.class, () ->
catalog.asTableCatalog().loadTable(newNameIdentifier));
}
+ @Test
+ public void testAddNullableColumnBackfillsNull() throws Exception {
+ String addColumnTableName = GravitinoITUtils.genRandomName(TABLE_PREFIX);
+ NameIdentifier tableIdentifier = NameIdentifier.of(schemaName,
addColumnTableName);
+ String tableLocation = String.format("%s/%s/%s", tempDirectory,
schemaName, addColumnTableName);
+ Map<String, String> properties = createProperties();
+ properties.put(Table.PROPERTY_TABLE_FORMAT, LANCE_TABLE_FORMAT);
+ properties.put(Table.PROPERTY_LOCATION, tableLocation);
+
+ catalog
+ .asTableCatalog()
+ .createTable(
+ tableIdentifier,
+ createColumns(),
+ TABLE_COMMENT,
+ properties,
+ Transforms.EMPTY_TRANSFORM,
+ Distributions.NONE,
+ new SortOrder[0]);
+
+ try (Dataset dataset = Dataset.open().uri(tableLocation).build()) {
+ SourcedTransaction transaction =
+ dataset
+ .newTransactionBuilder()
+ .operation(
+ Append.builder()
+ .fragments(
+ createFragmentMetadata(
+ tableLocation,
+ List.of(
+ new LanceDataValue(1, 100L, "first"),
+ new LanceDataValue(2, 200L, "second")),
+ dataset.getSchema()))
+ .build())
+ .transactionProperties(Map.of())
+ .build();
+ try (Dataset ignored = transaction.commit()) {
+ // The committed dataset is closed after the historical rows have been
written.
+ }
+ }
+
+ Table alteredTable =
+ catalog
+ .asTableCatalog()
+ .alterTable(
+ tableIdentifier,
+ TableChange.addColumn(
+ new String[] {"new_nullable_col"}, Types.StringType.get(),
"nullable column"));
+
+ Assertions.assertEquals(4, alteredTable.columns().length);
+ Assertions.assertEquals("new_nullable_col",
alteredTable.columns()[3].name());
+ Assertions.assertEquals("nullable column",
alteredTable.columns()[3].comment());
+
+ int rowCount = 0;
+ try (Dataset dataset = Dataset.open().uri(tableLocation).build();
+ LanceScanner scanner =
+ dataset.newScan(
+ new
ScanOptions.Builder().columns(List.of("new_nullable_col")).build());
+ ArrowReader reader = scanner.scanBatches()) {
+ Field addedStringField =
dataset.getSchema().findField("new_nullable_col");
+ Assertions.assertNotNull(addedStringField);
+ Assertions.assertTrue(addedStringField.isNullable());
+ Assertions.assertEquals(new ArrowType.Utf8(),
addedStringField.getType());
+
+ while (reader.loadNextBatch()) {
+ VectorSchemaRoot root = reader.getVectorSchemaRoot();
+ VarCharVector stringVector = (VarCharVector)
root.getVector("new_nullable_col");
+ for (int i = 0; i < root.getRowCount(); i++) {
+ Assertions.assertTrue(stringVector.isNull(i));
+ rowCount++;
+ }
+ }
+ }
+ Assertions.assertEquals(2, rowCount);
+ }
+
@Test
void testVersionCheckRefreshKeepsColumnTagsAndComments() {
String refreshCatalogName =
GravitinoITUtils.genRandomName("lance_version_check_catalog");
diff --git a/docs/lakehouse-generic-lance-table.md
b/docs/lakehouse-generic-lance-table.md
index 68d1a28dfc..bd5db84f5e 100644
--- a/docs/lakehouse-generic-lance-table.md
+++ b/docs/lakehouse-generic-lance-table.md
@@ -28,7 +28,7 @@ For Lance tables in a Generic Lakehouse Catalog, the
following table summarizes
|-----------|-----------------|
| List | ✅ Full |
| Load | ✅ Full |
-| Alter | Not support now |
+| Alter | ✅ Partial |
| Create | ✅ Full |
| Register | ✅ Full |
| Drop | ✅ Full |
@@ -162,6 +162,41 @@ Table operations follow standard relational catalog
patterns. See [Table Operati
The following sections provide examples and important details for working with
Lance tables.
+#### Add a Column
+
+Lance tables support adding nullable, top-level columns through the Gravitino
table API. New
+columns are appended to the schema, and Lance backfills existing rows with
`NULL`.
+
+```shell
+curl -X PUT -H "Accept: application/vnd.gravitino.v1+json" \
+ -H "Content-Type: application/json" -d '{
+ "updates": [
+ {
+ "@type": "addColumn",
+ "fieldName": ["new_column"],
+ "type": "string",
+ "comment": "New nullable column",
+ "position": "default",
+ "nullable": true,
+ "autoIncrement": false
+ }
+ ]
+}'
http://localhost:8090/api/metalakes/test/catalogs/generic_lakehouse_lance_catalog/schemas/schema/tables/lance_table
+```
+
+The following add-column options are not supported:
+
+- Nested columns
+- Non-nullable columns
+- `FIRST` or `AFTER` column positions
+- Default values
+- Auto-increment columns
+
+:::note
+This operation is available through the Gravitino table API and Java client.
The Lance REST
+`/add_columns` endpoint is not supported yet.
+:::
+
#### Create a Lance Table
<Tabs groupId='language' queryString>
