yuqi1129 commented on code in PR #13929:
URL: https://github.com/apache/gravitino/pull/13929#discussion_r4374637941
##########
lance/lance-rest-server/src/test/java/org/apache/gravitino/lance/integration/test/LanceSparkRESTServiceIT.java:
##########
@@ -287,6 +306,73 @@ public void testSelectFromEmptyTableViaSpark() {
Assertions.assertTrue(rows.isEmpty(), "Expected no rows for a newly
created empty table");
}
+ @Test
+ public void testSparkReadsTableCreatedThroughGravitino() {
+ // Creating through Gravitino's own API is the only path that reaches
+ // LanceTableOperations#createDataset, which leaves the Lance file format
version at
+ // lance-core's
+ // default whenever the table has no blob column - 2.2 as of lance-core
12.0.0, where earlier
+ // releases defaulted lower. Spark's CREATE TABLE goes through `declare`
instead, where
+ // Gravitino
+ // writes no dataset at all and the bundle writes it at its own version.
So this is the one
+ // combination that puts a Gravitino-written dataset in front of the
bundle's Lance, which in
+ // this matrix goes back to lance-core 2.0.0.
+ String schemaName = createSchema("spark_gravitino_created");
+ String tableName = newTableName("orders");
+
+ catalog
+ .asTableCatalog()
+ .createTable(
+ NameIdentifier.of(schemaName, tableName),
+ new Column[] {
+ Column.of("id", Types.IntegerType.get()), Column.of("score",
Types.FloatType.get())
+ },
+ "created through the Gravitino API",
+ // Deliberately no LANCE_TABLE_DECLARED: that is the `declare`
path, where Gravitino
+ // writes no dataset and the point of this test is lost.
+ ImmutableMap.of("format", "lance"),
+ Transforms.EMPTY_TRANSFORM,
+ null,
+ null);
+
+ // First the manifest on its own: a freshly created dataset holds no data
files, so this only
+ // proves the bundle can open a manifest written by lance-core 12.0.0.
+ Assertions.assertTrue(
+ sql("SELECT id, score FROM %s.%s", schemaName, tableName).isEmpty(),
+ "Expected no rows in a freshly created Lance dataset");
+
+ // Then the data files. Gravitino's API never writes rows, so seed them
here with the same
+ // lance-core the server runs - 12.0.0, at file format version 2.2 - and
read them back
+ // through the bundle. Without this the SELECT above would be the only
read in the test and
+ // the matrix would never decode a byte written by the new writer.
+ Table createdTable =
+ catalog.asTableCatalog().loadTable(NameIdentifier.of(schemaName,
tableName));
+
appendRowsWithServerLance(createdTable.properties().get(Table.PROPERTY_LOCATION));
+
+ List<Row> seededRows = sql("SELECT id, score FROM %s.%s ORDER BY id",
schemaName, tableName);
+ Assertions.assertEquals(2, seededRows.size());
+ assertRow(seededRows.get(0), 1, 1.5f);
+ assertRow(seededRows.get(1), 2, 2.5f);
+
+ // Writes are a different matter. A bundle whose Lance predates file
format version 2.2 has
+ // its commit rejected with
+ // "added files with version 2.0. However, the data storage version is
2.2."
+ // Only lance-spark 0.2.0 (lance-core 2.0.0) is affected; 0.4.0
(lance-core 4.0.0) and later
+ // write fine. This is a behaviour change from Gravitino's previous
lance-core 6.0.0, whose
+ // default was below 2.2, and it is tracked as a separate issue rather
than worked around here.
Review Comment:
The comment here and the PR description both say "tracked as a separate
issue", but I can't find one. Could you file it and link it here and in the PR
description?
Could you also say why we accept the regression instead of keeping the
previous behaviour? `LanceTableOperations#createDataset`
(catalog-lakehouse-generic) already calls `builder.dataStorageVersion(...)`
when there are blob columns. Pinning the old default when
`requiredFileFormatVersion` is empty, or exposing a table property, would be a
one-line change. It would keep lance-core 2.0.0 writers working (lance-spark
0.2.0, and lance-ray 0.2.0 with the pylance 2.0.1 the matrix pins). It would
also keep the 1.3.0 add-column behaviour for tables without blob columns, where
`lance.blob.legacy` is currently accepted and `lance.blob` rejected. If moving
to 2.2 is the intended direction, that's fine, but please write the reasoning
into the issue.
##########
lance/lance-rest-server/src/test/java/org/apache/gravitino/lance/integration/test/LanceSparkRESTServiceIT.java:
##########
@@ -287,6 +306,73 @@ public void testSelectFromEmptyTableViaSpark() {
Assertions.assertTrue(rows.isEmpty(), "Expected no rows for a newly
created empty table");
}
+ @Test
+ public void testSparkReadsTableCreatedThroughGravitino() {
+ // Creating through Gravitino's own API is the only path that reaches
+ // LanceTableOperations#createDataset, which leaves the Lance file format
version at
+ // lance-core's
+ // default whenever the table has no blob column - 2.2 as of lance-core
12.0.0, where earlier
+ // releases defaulted lower. Spark's CREATE TABLE goes through `declare`
instead, where
+ // Gravitino
+ // writes no dataset at all and the bundle writes it at its own version.
So this is the one
Review Comment:
nit: spotless left `lance-core's` and `Gravitino` on lines of their own.
Please re-wrap this comment by hand.
##########
docs/lance-rest-integration.md:
##########
@@ -53,6 +55,14 @@ The following table outlines the tested compatibility
between Gravitino versions
raising `ImportError: cannot import name
'LanceNamespaceStorageOptionsProvider' from 'lance'`. You can use
`lance-ray` 0.2.0 with Gravitino 1.3.0 by pinning `pylance` to 3.x or 4.x.
+- **Gravitino 2.0.0 embeds `lance-core` 12.0.0 and `lance-namespace-core`
0.11.1**, the pair
Review Comment:
This note says older clients keep working, and the matrix row above lists
lance-spark 0.2.0 and lance-ray 0.2.0 (pylance 2.0.1) without conditions. But a
lance-core 2.0.0 client cannot write to a table created through Gravitino's
table API. That limitation is only documented in
`lakehouse-generic-lance-table.md`. Could you add one sentence here along the
lines of "for tables created through the Gravitino table API, lance-core 2.0.0
clients (lance-spark 0.2.0, pylance 2.x) are read-only" and link the follow-up
issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]