Karakatiza666 commented on PR #438: URL: https://github.com/apache/arrow-js/pull/438#issuecomment-4505897316
> Could you compare with https://github.com/apache/arrow-js/pull/299 and explain what's different? I started this PR by repeating the original #299 verbatim, and then building on it for full parity of every feature available for the `List` type, while considering feedback to the original PR. The scope of #299 - a LargeList type stub, not even quite an MVP: you could manually build a LargeList column and read individual values out of one already in memory. Anything beyond that — saving it to an Arrow file, loading it in IPC binary or JSON form, comparing two LargeList schemas, modifying values, or constructing one through the normal Builder API — was either silently broken or missing entirely. The scope of this PR - LargeList as a fully-featured column type, on par with List. You can build one, read and write values, compare schemas, iterate, slice, serialize it to the Arrow IPC binary or JSON form, and round-trip it back — all with safe handling of 64-bit offsets and clear errors (rather than silent corruption) if a value would exceed JavaScript's safe integer range. Existing List, LargeUtf8, and LargeBinary types also get a latent slicing-on-write bug fixed as a side effect. Added on top of the original PR - IPC write path — VectorAssembler.visitLargeList + generalized assembleListVector with bigIntToNumber coercion - IPC read path — VectorLoader.visitLargeList - JSON form — JSONVectorAssembler.visitLargeList (OFFSET via bigNumsToStrings) - Type comparison — TypeComparator.visitLargeList (widened compareList to List | LargeList) - Mutation — SetVisitor.visitLargeList (merged into setList via bigIntToNumber) - Iterator interface declaration — completes IteratorVisitor typing - LargeListBuilder — new src/builder/largelist.ts with BigInt() accumulation and bigIntToNumber for safe child-cursor narrowing - Builder plumbing — widened VariableWidthBuilder bound; GetBuilderCtor.visitLargeList; LargeList / LargeListBuilder mapped into TypeToDataType, TypeToBuilder, DataTypeToBuilder - Public API — exported LargeListBuilder from Arrow.ts / Arrow.dom.ts - Latent bug fix — rebaseValueOffsets now bigint-safe (also fixes silent breakage on LargeUtf8 / LargeBinary sliced writes) - "Get" path consolidation — getList / getLargeList merged with bigIntToNumber boundary coercion - Test generator robustness — shared generateListLike; createVariableWidthOffsets64 truncates min / max on entry so fractional stride doesn't RangeError in BigInt() - Test coverage — LargeListBuilder entry in builder-tests.ts; visitLargeList in BasicVisitor / FeatureVisitor and both describe matrices in visitor-tests.ts -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
