Hi Gene,
I didn't quite understand what your requirement is. But Let me try to give
an example of how buffers can be used to create an ArrayData obj. Maybe
that could help your case.
Say I want to create an Int64 array of 500 elements.
int elms = 500;
auto data_res = arrow::AllocateBuffer(elms * sizeof(int64_t));
if (!data_res.ok()) {
// handle error
}
auto data_buf = std::move(data_res).ValueOrDie();
// alternatively you can do this in a function that returns a Status
// ARROW_ASSIGN_OR_RAISE(auto data_buf, arrow::AllocateBuffer(elms
*sizeof(int64_t)));
// now, if we dont need a validity buffer (i.e. all elems are valid) we
could keep
// the validity buffer as nullptr. Else you could use AllocateBitmap for
that
auto array_data = ArrayData::Make(arrow::int64(), elms, {nullptr,
std::move(data_buf)},
/*nullcount =*/0,
/*offset=*/0);
If you want to allocate a buffer yourself, using malloc, you could use this
constructor in Buffer
https://github.com/apache/arrow/blob/master/cpp/src/arrow/buffer.h#L58
I hope this helps.
On Mon, Sep 27, 2021 at 9:28 AM Weber, Eugene F Jr CIV (USA) <
[email protected]> wrote:
> Hi,
>
> I'm trying to wrap a buffer inside a DataArray container as suggested, but
> my C++ skills are still a bit weak. I've read through files and tried
> numerous inputs for the first parameter to ArrayData Make, but cannot get
> it to compile. I'll keep working on this, but would appreciate any help
> offered.
>
> Thanks,
>
> Gene
> --------
>
> #include <cstdint>
> #include <iostream>
> #include <vector>
>
> #include <arrow/api.h>
> #include <arrow/type_fwd.h>
> #include <arrow/array/data.h>
>
> using arrow::BufferBuilder;
> using arrow::Buffer;
>
> int main(int argc, char** argv) {
>
> arrow::Result<std::unique_ptr<Buffer>> maybe_buffer =
> arrow::AllocateBuffer(121751207936);
> if (!maybe_buffer.ok()) {
> std::cout << "Error Allocating Buffer from Memory Pool\n";
> // Could check /proc/meminfo for: MemAvailable: 128252780 kB
> exit(1);
> }
>
> std::shared_ptr<arrow::Buffer> buffer = *std::move(maybe_buffer);
> int64_t buf_size = buffer->size();
>
> arrow::ArrayData adc;
> adc.Make(??????, 1000, buffer, -1, 0);
>
> // static std::shared_ptr<ArrayData> Make(std::shared_ptr<DataType> type,
> int64_t length,
> //
> std::vector<std::shared_ptr<Buffer>> buffers,
> // int64_t null_count =
> kUnknownNullCount,
> // int64_t offset = 0);
>
> return EXIT_SUCCESS;
> }
> ________________________________________
> From: Antoine Pitrou [[email protected]]
> Sent: Monday, September 20, 2021 6:00 AM
> To: [email protected]
> Subject: [Non-DoD Source] Re: C++ Arrays & Buffers
>
> Hello Eugene,
>
> On Mon, 20 Sep 2021 09:33:26 +0000
> "Weber, Eugene F Jr CIV (USA)" <[email protected]> wrote:
> >
> > I've gone through the documentation but I'm still unclear about the
> usage of buffers with respect to arrays.
> >
> > "A Buffer encapsulates a pointer and data size .... Buffers are untyped:
> they simply denote a physical memory area"
> >
> > "The central type in Arrow is the class arrow::Array. An array
> represents a known-length sequence of values all having the same type.
> Internally, those values are represented by one or several buffers ...."
> >
> > The Array section then goes on to explain how to build Arrays with the
> ArrayBuilder base class, and concrete subclasses. There doesn't appear to
> be any need to allocate buffer space first. There is also a BufferBuilder
> class. But since there does not appear to be a way associate a created
> buffer with an array, I don't understand when explicit buffer creation
> would be used?
>
> The C++ documentation is unfortunately incomplete. Using ArrayBuilder
> subclasses is one way of creating arrays if you want to populate them
> with logical values (e.g. int64_t values for a Int64Array).
> But you can also create buffers directly and wrap them inside a
> ArrayData container:
> Caution-
> https://github.com/apache/arrow/blob/master/cpp/src/arrow/array/data.h#L73
>
> (this is useful if e.g. you want your array to point to *existing*
> memory)
>
> Then you can call MakeArray to get an actual Array subclass from the
> ArrayData container:
> Caution-
> https://github.com/apache/arrow/blob/master/cpp/src/arrow/array/util.h#L38
>
> Regards
>
> Antoine.
--
Niranda Perera
https://niranda.dev/
@n1r44 <https://twitter.com/N1R44>
