Hello All,
I am experiencing issues with finding documentation on .Net API of apache
arrow. Are there any examples or sample projects, that can be used for learning?
Currently I am trying to implement ability to append extra values to existing
primitives arrays, but I was not able to figure out a correct API for that.
The best way that I found was to create new PrimitiveArray, based on the values
in existing array (by coping values buffers).
To achieve this I created 2 new buffer builders (value and bitmap). Coping
existing values from the source array to new value buffer was quite
straightforward:
var valueBufferBuilder = new ArrowBuffer.Builder<T>();
var valueBufferSpan = sourceArray .ValueBuffer.Span.CastTo<T>();
if (!valueBufferSpan.IsEmpty)
valueBufferBuilder.Append(valueBufferSpan.Slice(0, Length));
after coping values from the source array, I was able to finally append a new
value
valueBufferBuilder.Append(value ?? default );
Unfortunately, I didn’t manage to find a similar way to achieve the same for
the bitmap buffer. However, I found a code that performs this task in
ArrayDataConcatenator class (not the most efficient one as it sets bit by bit
in a cycle and this can lead to high CPU usage especially on large arrays, so I
am not sure, that I should use it):
var validityBufferSpan = sourceArray .NullBitmapBuffer.Span;
for ( int i = 0; i < sourceArray .Length; i++)
{
validityBufferBuilder.Append(validityBufferSpan.IsEmpty ||
BitUtility.GetBit(validityBufferSpan, i));
}
validityBufferBuilder.Append(value.HasValue);
After both buffers were filled, I was able to construct primitive array by
writing:
var data = new ArrayData( sourceArray .Data.DataType, sourceArray.L ength
+ 1 , -1, 0, new ArrowBuffer[] { validityBufferBuilder.Build(),
valueBufferBuilder.Build() });
var targetA rray = ArrowArrayFactory.BuildArray(data);
Could you please tell if it’s a correct approach or there is a better way to do
it?
What do you think about extending .Net Apache Arrow API by adding extra
constructor to BitMapBuilder? This constructor will copy validity buffer and
avoid using GetBit/Append pair for each bit inside it, like:
public BitmapBuilder(Span<byte> initialValues, int validBits)
Best regards,
Alexey
--
Alexey Smirnov
