XuQianJin-Stars opened a new pull request, #2266:
URL: https://github.com/apache/fluss/pull/2266
### Purpose
Linked issue: close #2252
This PR adds support for array type conversion between Fluss and Iceberg,
enabling tiering of tables with array columns to Iceberg lakehouse storage. The
implementation references the Paimon array type support from PR #2166.
### Brief change log
- Updated `FlussDataTypeToIcebergDataType` to convert Fluss `ARRAY` type to
Iceberg `LIST` type instead of throwing `UnsupportedOperationException`
- Created `FlussArrayAsIcebergList` adapter class to wrap Fluss
`InternalArray` as Java `List` for Iceberg, supporting:
- All primitive types (boolean, byte, short, int, long, float, double)
- Complex types (string, char, decimal, timestamp, timestamp_ltz, date,
time, binary, bytes)
- Nested arrays (array of arrays)
- Null handling for both array elements and null arrays
- Updated `FlussRowAsIcebergRecord` to handle array field conversion using
the new adapter
- Enhanced `IcebergConversions` to support bidirectional type conversion
(Iceberg LIST ↔ Fluss ARRAY)
- Updated documentation to include `ARRAY → LIST` in the Iceberg data type
mapping table
### Tests
**Unit Tests** (`FlussRowAsIcebergRecordTest` - 9 test cases):
- `testArrayWithIntElements` - Array of integers
- `testArrayWithStringElements` - Array of strings
- `testNestedArrayType` - Nested arrays (array of arrays)
- `testArrayWithAllPrimitiveTypes` - Arrays of all primitive types
- `testArrayWithDecimalElements` - Array of decimal values
- `testArrayWithTimestampElements` - Arrays of TIMESTAMP and TIMESTAMP_LTZ
- `testArrayWithNullElements` - Arrays with null elements
- `testNullArray` - Null array handling
- `testArrayWithBinaryElements` - Array of binary data
**Integration Tests** (`IcebergTieringTest`):
- `testTieringWriteTableWithArrayType` - Parameterized test (4 cases)
covering:
- Primary key tables with array columns (partitioned/non-partitioned)
- Log tables with array columns (partitioned/non-partitioned)
**Test Results**: All 92 tests pass (increased from 88 tests before this PR)
### API and Format
**API Changes**: None. No breaking changes to public APIs.
**Storage Format**: No changes to Fluss storage format. This only affects
the conversion layer between Fluss and Iceberg during tiering operations.
**Type Mapping**:
- Fluss `ARRAY<T>` → Iceberg `LIST<T>` (element type `T` is converted
recursively)
- Supports nullable and non-nullable arrays
- Maintains element type semantics through recursive conversion
### Documentation
**New Feature**: Yes, this introduces array type support for Iceberg tiering.
**Documentation Changes**:
- Updated `website/docs/streaming-lakehouse/integrate-data-lakes/iceberg.md`
- Added `ARRAY → LIST` mapping to the data type compatibility table
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
