supermacro opened a new pull request, #299:
URL: https://github.com/apache/arrow-js/pull/299
## Summary
This PR implements full support for the `LargeList` data type in Apache
Arrow JavaScript bindings. LargeList uses 64-bit offsets (BigInt64Array)
instead of 32-bit offsets, enabling list values larger than 2GB.
## Related Issues
Closes #70
## Implementation Details
### Core Type System
- Added `LargeList` enum value (`Type.LargeList = 21`)
- Implemented `LargeList<T>` class with `BigInt64Array` offset support
- Added `isLargeList()` type guard
- Implemented `LargeListDataProps` interface for data construction
### Visitor Pattern Implementation
Implemented `visitLargeList()` methods across all visitors:
- **GetVisitor** (`visitor/get.ts`): Handles vector value retrieval with
BigInt to number conversion
- **IteratorVisitor** (`visitor/iterator.ts`): Enables iteration support via
`Symbol.iterator`
- **IndexOfVisitor** (`visitor/indexof.ts`): Supports `indexOf()` and
`includes()` operations
- **TypeAssembler** (`visitor/typeassembler.ts`): FlatBuffers serialization
for IPC
- **JSONTypeAssembler** (`visitor/jsontypeassembler.ts`): JSON serialization
### IPC Support
- Added LargeList case in `ipc/metadata/message.ts` for reading IPC metadata
### Testing
- Implemented comprehensive test data generation in
`test/generate-test-data.ts`
- Added `generateLargeList()` function with proper BigInt64Array offset
generation
- Fixed `createVariableWidthOffsets64()` to handle float-to-BigInt
conversion
- Added LargeList to test suite in `test/unit/generated-data-tests.ts`
### Public API
- Exported `LargeList` in `src/Arrow.ts` and `src/Arrow.dom.ts`
## Test Plan
All existing tests continue to pass, plus new comprehensive LargeList tests
validate:
- ✅ Type creation and properties
- ✅ Data construction with `makeData()`
- ✅ Vector creation and value retrieval
- ✅ Empty lists and null value handling
- ✅ Iteration via `[Symbol.iterator]`
- ✅ `indexOf()` and `includes()` operations
- ✅ Vector slicing and concatenation
Run tests with:
```bash
npm test -- -t src
```
## Checklist
- [x] Implementation follows existing code patterns
- [x] All visitor methods implemented
- [x] IPC serialization/deserialization support added
- [x] Comprehensive tests added using existing test framework
- [x] All tests passing
- [x] Public API exports added
- [x] No breaking changes
## Notes
- This implementation focuses on reading/writing LargeList data from IPC
format
- Builder class for LargeList construction is not included (can be added in
future PR if needed)
- Implementation mirrors existing List support but with 64-bit offsets
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
