CryoThrust opened a new issue, #12745:
URL: https://github.com/apache/seatunnel/issues/12745
### Search before asking
I searched the open/closed Lance issues and PRs and found no report for this
Map schema mismatch. #12738 addresses fragment batching, and this failure also
occurs without that change.
### What happened
`FragmentConverter.writeMapToVector` rejects a valid Arrow Map field because
it expects `key` and `value` to be two direct children of the Map. Arrow
represents a Map with one non-null `entries` Struct child containing those two
fields.
A one-row regression added to the existing `FragmentConverterTest` on
current `dev` (`b0f6feddf9ed0dc1f5d485cfc69d387edfe6ac82`) fails with:
```text
java.lang.IllegalArgumentException: Map field must have key and value child
fields
```
The existing restored-schema test passes: 2 tests run, 1 pass, 1 error. On
#12738's head, both one-row and three-row Map controls fail at the same guard.
The array batch control passes. This is a pre-existing Map issue, not a
batching regression.
### Minimal reproduction
In `FragmentConverterTest`, construct the schema and call the existing
single-row converter:
```java
SeaTunnelRowType rowType = new SeaTunnelRowType(
new String[] {"attributes"},
new SeaTunnelDataType<?>[] {
new MapType<>(BasicType.STRING_TYPE, BasicType.INT_TYPE)
});
Field entries = new Field(
"entries",
FieldType.notNullable(new ArrowType.Struct()),
Arrays.asList(
Field.notNullable("key", new ArrowType.Utf8()),
Field.nullable("value", new ArrowType.Int(32, true))));
Schema schema = new Schema(Arrays.asList(new Field(
"attributes",
FieldType.nullable(new ArrowType.Map(false)),
Arrays.asList(entries))));
try (RootAllocator allocator = new RootAllocator();
VectorSchemaRoot root = FragmentConverter.convertToVectorSchemaRoot(
new SeaTunnelRow(new Object[] {Map.of("a", 1)}),
rowType, schema, allocator)) {
Assertions.assertEquals(1, root.getRowCount());
}
```
Verified with JDK 17 and the Lance module's reactor `verify`, selecting
`FragmentConverterTest`. On macOS, the test JVM arguments were overridden to
`-Dfile.encoding=UTF-8 --add-opens=java.base/java.nio=ALL-UNNAMED` to avoid the
separate unquoted `-Dos.name=Mac OS X` launcher problem. Spotless passed;
Checkstyle/license checks and live engine/Lance E2E were not run for this
diagnostic.
### Expected behavior and scope
For a valid Arrow Map schema, the writer should read the key/value fields
from the `entries` Struct and write the Map values. The regression should
verify actual keys/values, plus empty Maps, null rows and nullable values.
There is a related schema-creation gap: current `SchemaUtils.convertSchema`
constructs a Map without child fields. Supporting Map columns end to end
therefore needs the schema-creation and writer paths to agree; only changing
the writer guard would leave new dataset creation incomplete. I would keep this
separate from #12738, preserve scalar behavior, and add EN/ZH documentation and
an existing `LanceIT` scenario.
Is that the intended Map support contract for this connector? I can prepare
a focused follow-up once the scope is agreed. No production fix or full job
reproduction is claimed here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
