Hi,
I'm trying to wrap a buffer inside a DataArray container as suggested, but my
C++ skills are still a bit weak. I've read through files and tried numerous
inputs for the first parameter to ArrayData Make, but cannot get it to compile.
I'll keep working on this, but would appreciate any help offered.
Thanks,
Gene
--------
#include <cstdint>
#include <iostream>
#include <vector>
#include <arrow/api.h>
#include <arrow/type_fwd.h>
#include <arrow/array/data.h>
using arrow::BufferBuilder;
using arrow::Buffer;
int main(int argc, char** argv) {
arrow::Result<std::unique_ptr<Buffer>> maybe_buffer =
arrow::AllocateBuffer(121751207936);
if (!maybe_buffer.ok()) {
std::cout << "Error Allocating Buffer from Memory Pool\n";
// Could check /proc/meminfo for: MemAvailable: 128252780 kB
exit(1);
}
std::shared_ptr<arrow::Buffer> buffer = *std::move(maybe_buffer);
int64_t buf_size = buffer->size();
arrow::ArrayData adc;
adc.Make(??????, 1000, buffer, -1, 0);
// static std::shared_ptr<ArrayData> Make(std::shared_ptr<DataType> type,
int64_t length,
// std::vector<std::shared_ptr<Buffer>>
buffers,
// int64_t null_count =
kUnknownNullCount,
// int64_t offset = 0);
return EXIT_SUCCESS;
}
________________________________________
From: Antoine Pitrou [[email protected]]
Sent: Monday, September 20, 2021 6:00 AM
To: [email protected]
Subject: [Non-DoD Source] Re: C++ Arrays & Buffers
Hello Eugene,
On Mon, 20 Sep 2021 09:33:26 +0000
"Weber, Eugene F Jr CIV (USA)" <[email protected]> wrote:
>
> I've gone through the documentation but I'm still unclear about the usage of
> buffers with respect to arrays.
>
> "A Buffer encapsulates a pointer and data size .... Buffers are untyped: they
> simply denote a physical memory area"
>
> "The central type in Arrow is the class arrow::Array. An array represents a
> known-length sequence of values all having the same type. Internally, those
> values are represented by one or several buffers ...."
>
> The Array section then goes on to explain how to build Arrays with the
> ArrayBuilder base class, and concrete subclasses. There doesn't appear to be
> any need to allocate buffer space first. There is also a BufferBuilder class.
> But since there does not appear to be a way associate a created buffer with
> an array, I don't understand when explicit buffer creation would be used?
The C++ documentation is unfortunately incomplete. Using ArrayBuilder
subclasses is one way of creating arrays if you want to populate them
with logical values (e.g. int64_t values for a Int64Array).
But you can also create buffers directly and wrap them inside a
ArrayData container:
Caution-https://github.com/apache/arrow/blob/master/cpp/src/arrow/array/data.h#L73
(this is useful if e.g. you want your array to point to *existing*
memory)
Then you can call MakeArray to get an actual Array subclass from the
ArrayData container:
Caution-https://github.com/apache/arrow/blob/master/cpp/src/arrow/array/util.h#L38
Regards
Antoine.
