CurtHagenlocher commented on code in PR #450:
URL: https://github.com/apache/arrow-dotnet/pull/450#discussion_r4116206223
##########
src/Apache.Arrow/Arrays/ArrayDataConcatenator.cs:
##########
@@ -125,6 +127,200 @@ public void Visit(FixedWidthType type)
Result = new ArrayData(resolvedType, _totalLength,
_totalNullCount, 0, new ArrowBuffer[] { validityBuffer, valueBuffer });
}
+ public void Visit(NullType type)
+ {
+ foreach (ArrayData arrayData in _arrayDataList)
+ {
+ arrayData.EnsureDataType(type.TypeId);
+ }
+
+ // A null array has no buffers; every slot is null.
+ Result = new ArrayData(type, _totalLength, _totalLength, 0,
System.Array.Empty<ArrowBuffer>());
+ }
+
+ public void Visit(DictionaryType type)
+ {
+ CheckData(type, 2);
+ var indexType = (IntegerType)type.IndexType;
+
+ // Inputs with no rows don't contribute any dictionary entries.
+ var contributing = new List<ArrayData>(_arrayDataList.Count);
+ foreach (ArrayData arrayData in _arrayDataList)
+ {
+ var otherType = (DictionaryType)arrayData.DataType;
+ if (otherType.IndexType.TypeId != indexType.TypeId)
+ {
+ throw new ArgumentException(
+ $"Cannot concatenate dictionary arrays with
different index types: {indexType.Name} vs {otherType.IndexType.Name}");
Review Comment:
Good catch, fixed in e2d1b28. When different dictionaries are appended, the
result is now never ordered: even if every input is ordered, the appended
entries aren't necessarily in order, and nothing verifies that. When every
input shares one dictionary, nothing is appended, so the result is ordered only
if every input says so. That commit also moves the dictionary concatenation
ahead of the buffer allocations and releases the buffers if building the
indices throws, which covers the leak from the overview. It also adds a test
for dictionaries held in separate `ArrayData` objects over the same memory that
checks the result still works after the inputs are disposed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
