wgzhao opened a new pull request, #12846: URL: https://github.com/apache/seatunnel/pull/12846
Yes, I am contributing this change to Apache SeaTunnel. ### Purpose of this change Fixes #12745. The Lance sink could not write `MAP` columns. Three problems combined: - `SchemaUtils.convertSchema` created a `Map` field with no child fields, so a newly created dataset did not follow Arrow's canonical map layout. - `FragmentConverter.writeMapToVector` expected `key` and `value` as direct children of the map field, while canonical Arrow maps store them inside a non-nullable `entries` struct. - The map key/value write path used the list-element convenience methods, which append to the union data vector instead of the map's key/value children. Entries were misaligned and values were dropped. ### Changes - Build canonical Arrow Map fields from the SeaTunnel `MapType`: one non-nullable `entries` struct containing a non-nullable `key` and a nullable `value`. - Resolve map key/value fields from the canonical `entries` struct, while still accepting schemas that declare direct children. - Write map keys and values through Arrow's mode-aware typed writers, including typed null writes for nullable map values. - Extend `FragmentConverterTest` with generated-schema, restored-schema, multi-row, empty-map, null-map-row, nullable-value and read-back coverage. - Add a `map<string, int>` column to the existing Lance E2E job and verify that the persisted dataset schema uses the canonical `entries` layout. - Document the Map layout in the English and Chinese Lance sink docs. ### Compatibility - No config options, defaults, public APIs, or dependencies change. - Jobs that previously could not write Map columns now work; there is no previously working Map write behavior to regress. - Map keys remain non-nullable, matching Arrow's requirement. ### Validation - `./mvnw -pl seatunnel-connectors-v2/connector-lance -DskipTests verify` passed. - `connector-lance` unit tests: 39 tests, 0 failures. - `spotless:check` passed for `connector-lance` and `connector-lance-e2e`. - The E2E module compiles. The `LanceIT` scenario was not executed locally because Docker/Testcontainers is unavailable in this environment; CI will run it. ### Does this PR introduce any user-facing change? No configuration change. Map columns in the Lance sink now produce valid Arrow Map data instead of failing. ### How was this patch tested? - New unit tests cover single and multiple rows, empty maps, null map rows, nullable values, generated schemas and restored dataset schemas. - The Lance E2E job now writes a `map<string, int>` column and checks the resulting dataset schema. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
