XuQianJin-Stars opened a new pull request, #2040:
URL: https://github.com/apache/fluss/pull/2040
### Purpose
Linked issue: close #1978
This PR adds comprehensive support for Array type in the Flink connector,
enabling users to read and write array data between Flink and Fluss tables.
This addresses a key limitation in the connector's type support system.
### Brief change log
- **Core Array type support in common module**:
- Enhanced Arrow format readers and writers (`ArrowReader`, `ArrowWriter`,
`ArrowArrayWriter`, `ArrowFieldWriter`) to handle array serialization and
deserialization
- Updated `ArrowUtils` to support Array type schema conversion
- **Flink connector integration**:
- Implemented bidirectional type conversion in
`FlussRowToFlinkRowConverter` and `FlinkRowToFlussRowConverter` to support
Array types
- Added `FlinkAsFlussArray` wrapper class to convert Flink's `ArrayData`
to Fluss's `InternalArray`
- Updated `FlinkAsFlussRow` to support array field access
- **Comprehensive test coverage**:
- Added `FlinkArrayTypeITCase` as the base test class with comprehensive
integration tests covering:
- Arrays of primitive types (int, bigint, string, boolean, double)
- Arrays with null elements
- Nested arrays (array of arrays)
- Arrays in primary key tables
- Array operations and access
- Created version-specific test implementations for Flink 1.18, 1.19,
1.20, and 2.1
### Tests
**Unit Tests:**
- Arrow array writer and reader tests for serialization/deserialization
**Integration Tests:**
- `Flink118ArrayTypeITCase`: Array type support for Flink 1.18
- `Flink119ArrayTypeITCase`: Array type support for Flink 1.19
- `Flink120ArrayTypeITCase`: Array type support for Flink 1.20
- `Flink21ArrayTypeITCase`: Array type support for Flink 2.1
Test scenarios include:
- `testArrayOfPrimitiveTypesInLogTable`: Verifies arrays of int, bigint,
string, boolean, and double types
- `testArrayWithNullElements`: Ensures proper handling of null elements
within arrays and null arrays
- `testNestedArrays`: Tests multi-dimensional arrays (array of arrays)
- `testArrayInPrimaryKeyTable`: Validates array support in tables with
primary keys
- `testArrayWithAllDataTypes`: Comprehensive test with all supported data
types in arrays
- `testArrayAccessAndCardinality`: Tests array operations and element access
### API and Format
**API Changes:**
- No breaking API changes
- Array type now supported in Flink connector's type mapping
**Storage Format:**
- Enhanced Arrow format to support Array type serialization
- Backward compatible with existing storage formats
### Documentation
**Feature Introduction:**
- This change introduces Array type support as a new feature in the Flink
connector
- Users can now create tables with array columns and perform read/write
operations through Flink SQL
**Documentation Updates Needed:**
- Update `website/docs/engine-flink/getting-started.md` to reflect Array
type support in the type mapping table (change from "Not supported" to
"Supported")
- Add examples of using Array types in Flink SQL with Fluss tables
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
