This is an automated email from the ASF dual-hosted git repository.

github-merge-queue[bot] pushed a commit to branch 
gh-readonly-queue/dev/pr-13061-e1fe5c5ef22c71a483dffad3f34fe202774a2d61
in repository https://gitbox.apache.org/repos/asf/seatunnel.git

commit 535d4237dd4b6ae92b14a9cbd790acf8d81cc71d
Author: Sepuri Sai Krishna <[email protected]>
AuthorDate: Mon Nov 9 14:46:20 2026 +0000

    [Fix][Connector-V2] Create Lance BIGINT columns as int64 (#13061)
---
 docs/en/connectors/sink/Lance.md                   |  4 +-
 .../introduction/concepts/incompatible-changes.md  |  7 +++
 docs/zh/connectors/sink/Lance.md                   |  4 +-
 .../introduction/concepts/incompatible-changes.md  |  7 +++
 .../seatunnel/lance/catalog/LanceCatalog.java      |  8 ++-
 .../seatunnel/lance/data/LanceTypeMapper.java      |  8 ++-
 .../seatunnel/lance/utils/SchemaUtils.java         |  5 +-
 .../lance/namespace/LanceCatalogTest.java          |  6 ++
 .../lance/utils/FragmentConverterTest.java         | 57 +++++++++++++++++
 .../seatunnel/e2e/connector/lance/LanceIT.java     | 40 ++++++++++++
 .../test/resources/lance/fake_bigint_to_lance.conf | 71 ++++++++++++++++++++++
 11 files changed, 209 insertions(+), 8 deletions(-)

diff --git a/docs/en/connectors/sink/Lance.md b/docs/en/connectors/sink/Lance.md
index eac7f6d701..681cc3322f 100644
--- a/docs/en/connectors/sink/Lance.md
+++ b/docs/en/connectors/sink/Lance.md
@@ -130,7 +130,7 @@ replica becomes a bottleneck. See [Sink Common 
Options](../common-options/sink-c
 
 ## Data Type Mapping
 
-Lance uses the Apache Arrow type system. The sink creates the Lance schema 
from the incoming SeaTunnel schema. The current mapping narrows every integer 
SeaTunnel type (`TINYINT`, `SMALLINT`, `INT`, `BIGINT`) to Arrow `int32`, so 
`BIGINT` values outside the signed 32-bit range are truncated.
+Lance uses the Apache Arrow type system. The sink creates the Lance schema 
from the incoming SeaTunnel schema. `TINYINT`, `SMALLINT` and `INT` are written 
as Arrow `int32`, and `BIGINT` as Arrow `int64`. When the dataset already 
exists, the sink writes to its existing column types, so a `BIGINT` column that 
an earlier version created as `int32` still truncates values outside the signed 
32-bit range.
 
 | SeaTunnel Data Type | Lance / Arrow Data Type |
 |---------------------|-------------------------|
@@ -138,7 +138,7 @@ Lance uses the Apache Arrow type system. The sink creates 
the Lance schema from
 | TINYINT             | int32                   |
 | SMALLINT            | int32                   |
 | INT                 | int32                   |
-| BIGINT              | int32 (values outside the signed 32-bit range are 
truncated) |
+| BIGINT              | int64                   |
 | FLOAT               | float32                 |
 | DOUBLE              | float64                 |
 | DECIMAL             | decimal128              |
diff --git a/docs/en/introduction/concepts/incompatible-changes.md 
b/docs/en/introduction/concepts/incompatible-changes.md
index 61d47aa3e6..93bbd5f334 100644
--- a/docs/en/introduction/concepts/incompatible-changes.md
+++ b/docs/en/introduction/concepts/incompatible-changes.md
@@ -5,6 +5,13 @@ You need to check this document before you upgrade to related 
version.
 
 ## dev
 
+### Lance sink: BIGINT columns are created as int64
+
+- **Behavior change: a `BIGINT` column in a dataset the Lance sink creates is 
Arrow `int64`**
+  - **Description**: When the Lance sink created a dataset, it declared 
`BIGINT` columns as Arrow `int32` and wrote each value with 
`Number.intValue()`, so a value outside the signed 32-bit range was silently 
wrapped. For example, `3000000000` was stored as `-1294967296`, and the epoch 
millisecond `1739586083123` as `124328243`. The sink, and the catalog's 
`createTable`, now declare `BIGINT` as `int64` and store the value unchanged.
+  - **Impact**: A dataset created by this version has an `int64` column where 
an earlier version created `int32`. A dataset that already exists keeps its 
column types, because the sink writes to the schema it finds, so its `int32` 
column still wraps large values.
+  - **Migration Guide**: No configuration change is needed. If a consumer 
expects `int32` for these columns, update it to read `int64`. To store large 
values in a dataset created by an earlier version, recreate the dataset and 
write the rows again from the source.
+
 ### Databend source: a DECIMAL column is read as DECIMAL instead of STRING
 
 - **Behavior change: a Databend `DECIMAL(p, s)` column reaches the job as 
`DECIMAL(p, s)`**
diff --git a/docs/zh/connectors/sink/Lance.md b/docs/zh/connectors/sink/Lance.md
index 7115fb3ead..83ab68cbca 100644
--- a/docs/zh/connectors/sink/Lance.md
+++ b/docs/zh/connectors/sink/Lance.md
@@ -122,7 +122,7 @@ lance.write.storage.options = {
 
 ## 数据类型映射
 
-Lance 使用 Apache Arrow 类型系统。sink 会根据上游 SeaTunnel 表结构创建 Lance 
schema。当前映射会把所有整数类型(`TINYINT`、`SMALLINT`、`INT`、`BIGINT`)一律收窄为 Arrow 
`int32`,因此超出有符号 32 位范围的 `BIGINT` 值会被截断。
+Lance 使用 Apache Arrow 类型系统。sink 会根据上游 SeaTunnel 表结构创建 Lance 
schema。`TINYINT`、`SMALLINT` 和 `INT` 写为 Arrow `int32`,`BIGINT` 写为 Arrow 
`int64`。如果数据集已经存在,sink 会按已有列的类型写入,因此早期版本创建为 `int32` 的 `BIGINT` 列,仍会截断超出有符号 32 
位范围的值。
 
 | SeaTunnel 数据类型 | Lance / Arrow 数据类型 |
 |--------------------|------------------------|
@@ -130,7 +130,7 @@ Lance 使用 Apache Arrow 类型系统。sink 会根据上游 SeaTunnel 表结
 | TINYINT            | int32                  |
 | SMALLINT           | int32                  |
 | INT                | int32                  |
-| BIGINT             | int32(超出有符号 32 位范围的值会被截断) |
+| BIGINT             | int64                  |
 | FLOAT              | float32                |
 | DOUBLE             | float64                |
 | DECIMAL            | decimal128             |
diff --git a/docs/zh/introduction/concepts/incompatible-changes.md 
b/docs/zh/introduction/concepts/incompatible-changes.md
index a1fd24b43e..cde3e0c4b7 100644
--- a/docs/zh/introduction/concepts/incompatible-changes.md
+++ b/docs/zh/introduction/concepts/incompatible-changes.md
@@ -4,6 +4,13 @@
 
 ## dev
 
+### Lance Sink:BIGINT 列创建为 int64
+
+- **行为变更:Lance Sink 创建的数据集中,`BIGINT` 列为 Arrow `int64`**
+  - **描述**:Lance Sink 创建数据集时,会把 `BIGINT` 列声明为 Arrow `int32`,并用 
`Number.intValue()` 写入每个值,因此超出有符号 32 位范围的值会被静默回绕。例如 `3000000000` 被存为 
`-1294967296`,毫秒时间戳 `1739586083123` 被存为 `124328243`。现在 Sink 以及 Catalog 的 
`createTable` 都把 `BIGINT` 声明为 `int64`,值按原样存储。
+  - **影响**:本版本创建的数据集中该列为 `int64`,而早期版本创建的是 `int32`。已经存在的数据集保留原有的列类型,因为 Sink 
会按已有的 schema 写入,所以其中的 `int32` 列仍会回绕较大的值。
+  - **迁移指南**:无需修改配置。如果下游按 `int32` 读取这些列,请改为读取 
`int64`。若要在早期版本创建的数据集中存储较大的值,请重新创建数据集,并从数据源重新写入数据。
+
 ### Databend Source:DECIMAL 列按 DECIMAL 读取,而不再是 STRING
 
 - **行为变更:Databend 的 `DECIMAL(p, s)` 列在作业中的类型为 `DECIMAL(p, s)`**
diff --git 
a/seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/catalog/LanceCatalog.java
 
b/seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/catalog/LanceCatalog.java
index 70c10c27b1..d2a2bae633 100644
--- 
a/seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/catalog/LanceCatalog.java
+++ 
b/seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/catalog/LanceCatalog.java
@@ -447,7 +447,13 @@ public class LanceCatalog implements Catalog {
                     new com.lancedb.lance.namespace.model.JsonArrowDataType();
 
             if (arrowType instanceof 
org.apache.arrow.vector.types.pojo.ArrowType.Int) {
-                jsonType.setType("int32");
+                org.apache.arrow.vector.types.pojo.ArrowType.Int intType =
+                        (org.apache.arrow.vector.types.pojo.ArrowType.Int) 
arrowType;
+                if (intType.getBitWidth() == 64) {
+                    jsonType.setType(intType.getIsSigned() ? "int64" : 
"uint64");
+                } else {
+                    jsonType.setType("int32");
+                }
             } else if (arrowType instanceof 
org.apache.arrow.vector.types.pojo.ArrowType.Utf8) {
                 jsonType.setType("utf8");
             } else if (arrowType instanceof 
org.apache.arrow.vector.types.pojo.ArrowType.Bool) {
diff --git 
a/seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/data/LanceTypeMapper.java
 
b/seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/data/LanceTypeMapper.java
index 0f824b44cf..3d20469359 100644
--- 
a/seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/data/LanceTypeMapper.java
+++ 
b/seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/data/LanceTypeMapper.java
@@ -49,12 +49,13 @@ public class LanceTypeMapper {
             case "int8":
             case "int16":
             case "int32":
-            case "int64":
             case "uint8":
             case "uint16":
             case "uint32":
             case "uint64":
                 return BasicType.INT_TYPE;
+            case "int64":
+                return BasicType.LONG_TYPE;
             case "utf8":
             case "largeutf8":
             case "string":
@@ -91,11 +92,14 @@ public class LanceTypeMapper {
         switch (type.getSqlType()) {
             case TINYINT:
             case SMALLINT:
-            case BIGINT:
             case INT:
                 JsonArrowDataType intType = new JsonArrowDataType();
                 intType.setType("int32");
                 return intType;
+            case BIGINT:
+                JsonArrowDataType bigintType = new JsonArrowDataType();
+                bigintType.setType("int64");
+                return bigintType;
             case STRING:
                 JsonArrowDataType stringType = new JsonArrowDataType();
                 stringType.setType("utf8");
diff --git 
a/seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/utils/SchemaUtils.java
 
b/seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/utils/SchemaUtils.java
index ffe9eee501..93a8f37ad2 100644
--- 
a/seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/utils/SchemaUtils.java
+++ 
b/seatunnel-connectors-v2/connector-lance/src/main/java/org/apache/seatunnel/connectors/seatunnel/lance/utils/SchemaUtils.java
@@ -69,10 +69,13 @@ public class SchemaUtils {
                     case TINYINT:
                     case SMALLINT:
                     case INT:
-                    case BIGINT:
                         field = Field.nullable(fieldName, new 
ArrowType.Int(32, true));
                         fieldList.add(field);
                         break;
+                    case BIGINT:
+                        field = Field.nullable(fieldName, new 
ArrowType.Int(64, true));
+                        fieldList.add(field);
+                        break;
                     case FLOAT:
                     case DOUBLE:
                         field =
diff --git 
a/seatunnel-connectors-v2/connector-lance/src/test/java/org/apache/seatunnel/connectors/seatunnel/lance/namespace/LanceCatalogTest.java
 
b/seatunnel-connectors-v2/connector-lance/src/test/java/org/apache/seatunnel/connectors/seatunnel/lance/namespace/LanceCatalogTest.java
index b5d62f1806..0a2c31089d 100644
--- 
a/seatunnel-connectors-v2/connector-lance/src/test/java/org/apache/seatunnel/connectors/seatunnel/lance/namespace/LanceCatalogTest.java
+++ 
b/seatunnel-connectors-v2/connector-lance/src/test/java/org/apache/seatunnel/connectors/seatunnel/lance/namespace/LanceCatalogTest.java
@@ -117,6 +117,12 @@ public class LanceCatalogTest {
         // The getTable() should return the same table structure as created, 
including primary key
         // and comment
         Assertions.assertEquals(templateTable.toString(), table.toString());
+        // TableSchema.toString() leaves out the columns, so check the integer 
types directly. The
+        // directory namespace returns no schema, so these come from the 
dataset's own schema.
+        Assertions.assertEquals(
+                BasicType.INT_TYPE, 
table.getTableSchema().getColumn("integer_col").getDataType());
+        Assertions.assertEquals(
+                BasicType.LONG_TYPE, 
table.getTableSchema().getColumn("long_col").getDataType());
     }
 
     @Test
diff --git 
a/seatunnel-connectors-v2/connector-lance/src/test/java/org/apache/seatunnel/connectors/seatunnel/lance/utils/FragmentConverterTest.java
 
b/seatunnel-connectors-v2/connector-lance/src/test/java/org/apache/seatunnel/connectors/seatunnel/lance/utils/FragmentConverterTest.java
index 3e042d6733..e2d539d592 100644
--- 
a/seatunnel-connectors-v2/connector-lance/src/test/java/org/apache/seatunnel/connectors/seatunnel/lance/utils/FragmentConverterTest.java
+++ 
b/seatunnel-connectors-v2/connector-lance/src/test/java/org/apache/seatunnel/connectors/seatunnel/lance/utils/FragmentConverterTest.java
@@ -17,6 +17,8 @@
 
 package org.apache.seatunnel.connectors.seatunnel.lance.utils;
 
+import org.apache.seatunnel.api.table.catalog.PhysicalColumn;
+import org.apache.seatunnel.api.table.catalog.TableSchema;
 import org.apache.seatunnel.api.table.type.ArrayType;
 import org.apache.seatunnel.api.table.type.BasicType;
 import org.apache.seatunnel.api.table.type.LocalTimeType;
@@ -27,6 +29,7 @@ import org.apache.seatunnel.api.table.type.SeaTunnelRowType;
 import 
org.apache.seatunnel.connectors.seatunnel.lance.exception.LanceConnectorException;
 
 import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BigIntVector;
 import org.apache.arrow.vector.TimeStampMicroTZVector;
 import org.apache.arrow.vector.TimeStampMilliVector;
 import org.apache.arrow.vector.VarCharVector;
@@ -290,6 +293,60 @@ class FragmentConverterTest {
         }
     }
 
+    @Test
+    void shouldKeepEveryBitOfABigint() {
+        SeaTunnelRowType rowType =
+                new SeaTunnelRowType(
+                        new String[] {"c_bigint"},
+                        new SeaTunnelDataType<?>[] {BasicType.LONG_TYPE});
+        // Every value is outside the int32 range. The third is an epoch in 
milliseconds.
+        long[] values = {
+            3_000_000_000L, -3_000_000_000L, 1_739_586_083_123L, 
Long.MAX_VALUE, Long.MIN_VALUE
+        };
+        List<SeaTunnelRow> rows =
+                Arrays.asList(
+                        new SeaTunnelRow(new Object[] {values[0]}),
+                        new SeaTunnelRow(new Object[] {values[1]}),
+                        new SeaTunnelRow(new Object[] {values[2]}),
+                        new SeaTunnelRow(new Object[] {values[3]}),
+                        new SeaTunnelRow(new Object[] {values[4]}));
+        // The schema the sink declares when it creates the dataset itself.
+        Schema schema = SchemaUtils.convertSchema(rows.get(0), rowType);
+        Assertions.assertEquals(new ArrowType.Int(64, true), 
schema.getFields().get(0).getType());
+
+        try (RootAllocator allocator = new RootAllocator();
+                VectorSchemaRoot root =
+                        FragmentConverter.convertToVectorSchemaRoot(
+                                rows, rowType, schema, allocator)) {
+            BigIntVector vector = (BigIntVector) root.getVector("c_bigint");
+            for (int i = 0; i < values.length; i++) {
+                Assertions.assertEquals(values[i], vector.get(i));
+            }
+        }
+    }
+
+    @Test
+    void shouldDeclareABigintColumnAsInt64InTheCatalogSchema() {
+        TableSchema tableSchema =
+                TableSchema.builder()
+                        .column(
+                                PhysicalColumn.of(
+                                        "c_bigint",
+                                        BasicType.LONG_TYPE,
+                                        (Long) null,
+                                        true,
+                                        null,
+                                        null))
+                        .build();
+        Assertions.assertEquals(
+                "int64",
+                SchemaUtils.convertJsonArrowSchema(tableSchema)
+                        .getFields()
+                        .get(0)
+                        .getType()
+                        .getType());
+    }
+
     private static long epochMicros(LocalDateTime value) {
         Instant instant = value.atZone(ZoneId.systemDefault()).toInstant();
         return ChronoUnit.MICROS.between(Instant.EPOCH, instant);
diff --git 
a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-lance-e2e/src/test/java/org/apache/seatunnel/e2e/connector/lance/LanceIT.java
 
b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-lance-e2e/src/test/java/org/apache/seatunnel/e2e/connector/lance/LanceIT.java
index dddca4518d..ac2cb504c8 100644
--- 
a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-lance-e2e/src/test/java/org/apache/seatunnel/e2e/connector/lance/LanceIT.java
+++ 
b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-lance-e2e/src/test/java/org/apache/seatunnel/e2e/connector/lance/LanceIT.java
@@ -26,6 +26,7 @@ import 
org.apache.seatunnel.e2e.common.junit.TestContainerExtension;
 
 import org.apache.arrow.memory.BufferAllocator;
 import org.apache.arrow.memory.RootAllocator;
+import org.apache.arrow.vector.BigIntVector;
 import org.apache.arrow.vector.IntVector;
 import org.apache.arrow.vector.TimeStampVector;
 import org.apache.arrow.vector.VectorSchemaRoot;
@@ -68,6 +69,7 @@ public class LanceIT extends TestSuiteBase {
     private static final String DATASET_PATH = "/tmp/seatunnel_mnt/lanceTest/";
     private static final String TABLE_NAME = "lance_sink_table";
     private static final String TIMESTAMP_TABLE_NAME = "lance_timestamp_table";
+    private static final String BIGINT_TABLE_NAME = "lance_bigint_table";
 
     @TestContainerExtension
     protected final ContainerExtendedFactory extendedFactory =
@@ -157,6 +159,44 @@ public class LanceIT extends TestSuiteBase {
         return fractions;
     }
 
+    @TestTemplate
+    public void testBigintKeepsItsValue(TestContainer container)
+            throws IOException, InterruptedException {
+        Container.ExecResult writeResult = 
container.executeJob("/lance/fake_bigint_to_lance.conf");
+        Assertions.assertEquals(0, writeResult.getExitCode(), 
writeResult.getStderr());
+
+        Map<Integer, Long> expected = new HashMap<>();
+        expected.put(1, 3_000_000_000L);
+        expected.put(2, -3_000_000_000L);
+        expected.put(3, 1_739_586_083_123L);
+        expected.put(4, Long.MAX_VALUE);
+
+        given().ignoreExceptions()
+                .await()
+                .atMost(60000, TimeUnit.MILLISECONDS)
+                .untilAsserted(() -> Assertions.assertEquals(expected, 
readBigints()));
+    }
+
+    private Map<Integer, Long> readBigints() throws Exception {
+        Map<Integer, Long> values = new HashMap<>();
+        try (BufferAllocator allocator = new RootAllocator();
+                Dataset dataset = Dataset.open(DATASET_PATH + 
BIGINT_TABLE_NAME, allocator);
+                LanceScanner scanner = dataset.newScan();
+                ArrowReader reader = scanner.scanBatches()) {
+            while (reader.loadNextBatch()) {
+                VectorSchemaRoot root = reader.getVectorSchemaRoot();
+                IntVector ids = (IntVector) root.getVector("id");
+                BigIntVector bigints = (BigIntVector) 
root.getVector("c_bigint");
+                for (int i = 0; i < root.getRowCount(); i++) {
+                    // A duplicated row would otherwise overwrite the first 
one unnoticed.
+                    Assertions.assertNull(
+                            values.put(ids.get(i), bigints.get(i)), "duplicate 
id " + ids.get(i));
+                }
+            }
+        }
+        return values;
+    }
+
     private long loadLanceTableCount() {
         long count = 0;
         try {
diff --git 
a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-lance-e2e/src/test/resources/lance/fake_bigint_to_lance.conf
 
b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-lance-e2e/src/test/resources/lance/fake_bigint_to_lance.conf
new file mode 100644
index 0000000000..241d56a3a8
--- /dev/null
+++ 
b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-lance-e2e/src/test/resources/lance/fake_bigint_to_lance.conf
@@ -0,0 +1,71 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+env {
+  parallelism = 1
+  job.mode = "BATCH"
+
+  # You can set spark configuration here
+  spark.app.name = "SeaTunnel"
+  spark.executor.instances = 2
+  spark.executor.cores = 1
+  spark.executor.memory = "1g"
+  spark.master = local
+}
+
+source {
+  FakeSource {
+    schema = {
+      fields {
+        id = int
+        c_bigint = bigint
+      }
+    }
+    # Each value is outside the int32 range. The third is an epoch in 
milliseconds.
+    rows = [
+      {
+        kind = INSERT
+        fields = [1, 3000000000]
+      },
+      {
+        kind = INSERT
+        fields = [2, -3000000000]
+      },
+      {
+        kind = INSERT
+        fields = [3, 1739586083123]
+      },
+      {
+        kind = INSERT
+        fields = [4, 9223372036854775807]
+      }
+    ]
+    plugin_output = "fake"
+  }
+}
+
+transform {
+}
+
+sink {
+  Lance {
+    dataset_path = "/tmp/seatunnel_mnt/lanceTest/lance_bigint_table"
+    namespace_type = "dir"
+    namespace_id = "root"
+    table = "lance_bigint_table"
+  }
+}

Reply via email to