This is an automated email from the ASF dual-hosted git repository.
sgilmore pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/main by this push:
new 2ca9ad2861 GH-41653: [MATLAB] Add new `arrow.c.Array` MATLAB class
which wraps a C Data Interface format `ArrowArray` C struct (#41655)
2ca9ad2861 is described below
commit 2ca9ad2861387a08244427eb1a2457c32a8ed31a
Author: Sarah Gilmore <[email protected]>
AuthorDate: Wed May 15 10:00:48 2024 -0400
GH-41653: [MATLAB] Add new `arrow.c.Array` MATLAB class which wraps a C
Data Interface format `ArrowArray` C struct (#41655)
### Rationale for this change
Now that the MATLAB interface has support for `arrow.tabular.RecordBatch`
and `arrow.array.Array`, we should add support for the [C Data
Interface](https://arrow.apache.org/docs/format/CDataInterface.html) format.
The C Data Interface is based around two C struct definitions: (1)
`ArrowArray` and (2) `ArrowSchema`.
We should start by adding a new MATLAB class (e.g. `arrow.c.Array`) which
wraps the underlying `ArrowArray` C struct.
Later, we can add another new MATLAB class (e.g. `arrow.c.Schema`) which
wraps the `ArrowSchema` C struct.
Once we have added these two MATLAB classes, we can then add import and
export functionality to share the Arrow memory between multiple language
runtimes running in the same process.
This would help enable workflows like sharing Arrow data between the MATLAB
Interface to Arrow and `pyarrow` running within the MATLAB process via the
[MATLAB interface to
Python](https://www.mathworks.com/help/matlab/call-python-libraries.html)).
### What changes are included in this PR?
1. Added a new C++ proxy class called `arrow::matlab::c::proxy::Array`
which wraps an `ArrowArray` `struct` pointer. This class is registered as the
proxy `arrow.c.proxy.Array` in order to make it accessible to MATLAB.
2. Added a new MATLAB class called `arrow.c.Array` that has an
`arrow.c.proxy.Array` instance. It has one public property named `Address`,
which is a scalar `uint64`. This property is the memory address of the
`ArrowArray` `struct` pointer owned by `arrow.c.proxy.Array`.
### Are these changes tested?
Yes.
1. Added a new test class called `test/arrow/c/tArray.m`.
2. @ kevingurney and I created a prototype for importing and exporting
arrow `Array`s via the C Data Interface format
[here](https://github.com/mathworks/arrow/tree/arrow-array-address). We were
able to share arrow `Array`s and `RecordBatch`es between mlarrow and pyarrow.
Our plan now is to submit the necessary MATLAB code incrementally.
### Are there any user-facing changes?
Yes. The `arrow.c.Array` class is user-facing. However, it's only intended
for "advanced" use-cases. In the future, we may add higher-level functionality
on top of the C Data Interface so that users don't need to interact with it
directly.
**NOTE:** On destruction, `arrow.c.proxy.Array` will check to see if the
`ArrowArray` has already been consumed by an importer. If not,
`arrow.c.proxy.Array`'s destructor will call the `release` callback on the
`ArrowArray` to avoid memory leaks. To the best of our knowledge, this is
similar to the how the [Arrow PyCapsule
Interface](https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html)
works.
### Future Directions
1. #41654
2. #41656
* GitHub Issue: #41653
Lead-authored-by: Sarah Gilmore <[email protected]>
Co-authored-by: Sarah Gilmore <[email protected]>
Co-authored-by: Kevin Gurney <[email protected]>
Signed-off-by: Sarah Gilmore <[email protected]>
---
matlab/src/cpp/arrow/matlab/c/proxy/array.cc | 49 ++++++++++++++++++++++
matlab/src/cpp/arrow/matlab/c/proxy/array.h | 41 ++++++++++++++++++
matlab/src/cpp/arrow/matlab/proxy/factory.cc | 2 +
matlab/src/matlab/+arrow/+c/Array.m | 37 ++++++++++++++++
matlab/test/arrow/c/tArray.m | 48 +++++++++++++++++++++
matlab/tools/cmake/BuildMatlabArrowInterface.cmake | 3 +-
6 files changed, 179 insertions(+), 1 deletion(-)
diff --git a/matlab/src/cpp/arrow/matlab/c/proxy/array.cc
b/matlab/src/cpp/arrow/matlab/c/proxy/array.cc
new file mode 100644
index 0000000000..a5f3418f1b
--- /dev/null
+++ b/matlab/src/cpp/arrow/matlab/c/proxy/array.cc
@@ -0,0 +1,49 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <cstddef>
+#include "arrow/c/abi.h"
+
+#include "arrow/matlab/c/proxy/array.h"
+
+#include "libmexclass/proxy/Proxy.h"
+
+namespace arrow::matlab::c::proxy {
+
+Array::Array() : arrowArray{} { REGISTER_METHOD(Array, getAddress); }
+
+Array::~Array() {
+ if (arrowArray.release != NULL) {
+ arrowArray.release(&arrowArray);
+ arrowArray.release = NULL;
+ }
+}
+
+libmexclass::proxy::MakeResult Array::make(
+ const libmexclass::proxy::FunctionArguments& constructor_arguments) {
+ return std::make_shared<Array>();
+}
+
+void Array::getAddress(libmexclass::proxy::method::Context& context) {
+ namespace mda = ::matlab::data;
+
+ mda::ArrayFactory factory;
+ auto address = reinterpret_cast<uint64_t>(&arrowArray);
+ context.outputs[0] = factory.createScalar(address);
+}
+
+} // namespace arrow::matlab::c::proxy
\ No newline at end of file
diff --git a/matlab/src/cpp/arrow/matlab/c/proxy/array.h
b/matlab/src/cpp/arrow/matlab/c/proxy/array.h
new file mode 100644
index 0000000000..b42b2dcd9c
--- /dev/null
+++ b/matlab/src/cpp/arrow/matlab/c/proxy/array.h
@@ -0,0 +1,41 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/c/abi.h"
+
+#include "libmexclass/proxy/Proxy.h"
+
+namespace arrow::matlab::c::proxy {
+
+class Array : public libmexclass::proxy::Proxy {
+ public:
+ Array();
+
+ ~Array();
+
+ static libmexclass::proxy::MakeResult make(
+ const libmexclass::proxy::FunctionArguments& constructor_arguments);
+
+ protected:
+ void getAddress(libmexclass::proxy::method::Context& context);
+
+ struct ArrowArray arrowArray;
+
+ // struct ArrowArray* arrowArray;
+};
+
+} // namespace arrow::matlab::c::proxy
\ No newline at end of file
diff --git a/matlab/src/cpp/arrow/matlab/proxy/factory.cc
b/matlab/src/cpp/arrow/matlab/proxy/factory.cc
index 23492f75de..cf13ed6aa5 100644
--- a/matlab/src/cpp/arrow/matlab/proxy/factory.cc
+++ b/matlab/src/cpp/arrow/matlab/proxy/factory.cc
@@ -25,6 +25,7 @@
#include "arrow/matlab/array/proxy/time64_array.h"
#include "arrow/matlab/array/proxy/timestamp_array.h"
#include "arrow/matlab/buffer/proxy/buffer.h"
+#include "arrow/matlab/c/proxy/array.h"
#include "arrow/matlab/error/error.h"
#include "arrow/matlab/io/csv/proxy/table_reader.h"
#include "arrow/matlab/io/csv/proxy/table_writer.h"
@@ -99,6 +100,7 @@ libmexclass::proxy::MakeResult Factory::make_proxy(
REGISTER_PROXY(arrow.io.feather.proxy.Reader ,
arrow::matlab::io::feather::proxy::Reader);
REGISTER_PROXY(arrow.io.csv.proxy.TableWriter ,
arrow::matlab::io::csv::proxy::TableWriter);
REGISTER_PROXY(arrow.io.csv.proxy.TableReader ,
arrow::matlab::io::csv::proxy::TableReader);
+ REGISTER_PROXY(arrow.c.proxy.Array ,
arrow::matlab::c::proxy::Array);
// clang-format on
return libmexclass::error::Error{error::UNKNOWN_PROXY_ERROR_ID,
diff --git a/matlab/src/matlab/+arrow/+c/Array.m
b/matlab/src/matlab/+arrow/+c/Array.m
new file mode 100644
index 0000000000..574fca9afe
--- /dev/null
+++ b/matlab/src/matlab/+arrow/+c/Array.m
@@ -0,0 +1,37 @@
+%ARRAY Wrapper for an Arrow C Data Interface format ArrowArray C struct
pointer.
+
+% Licensed to the Apache Software Foundation (ASF) under one or more
+% contributor license agreements. See the NOTICE file distributed with
+% this work for additional information regarding copyright ownership.
+% The ASF licenses this file to you under the Apache License, Version
+% 2.0 (the "License"); you may not use this file except in compliance
+% with the License. You may obtain a copy of the License at
+%
+% http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS,
+% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+% implied. See the License for the specific language governing
+% permissions and limitations under the License.
+classdef Array < matlab.mixin.Scalar
+
+ properties (Hidden, SetAccess=private, GetAccess=public)
+ Proxy
+ end
+
+ properties(Dependent, GetAccess=public, SetAccess=private)
+ Address(1, 1) uint64
+ end
+
+ methods
+ function obj = Array()
+ proxyName = "arrow.c.proxy.Array";
+ obj.Proxy = arrow.internal.proxy.create(proxyName);
+ end
+
+ function address = get.Address(obj)
+ address = obj.Proxy.getAddress();
+ end
+ end
+end
\ No newline at end of file
diff --git a/matlab/test/arrow/c/tArray.m b/matlab/test/arrow/c/tArray.m
new file mode 100644
index 0000000000..f8caf48065
--- /dev/null
+++ b/matlab/test/arrow/c/tArray.m
@@ -0,0 +1,48 @@
+%TARRAY Defines unit tests for arrow.c.Array.
+
+% Licensed to the Apache Software Foundation (ASF) under one or more
+% contributor license agreements. See the NOTICE file distributed with
+% this work for additional information regarding copyright ownership.
+% The ASF licenses this file to you under the Apache License, Version
+% 2.0 (the "License"); you may not use this file except in compliance
+% with the License. You may obtain a copy of the License at
+%
+% http://www.apache.org/licenses/LICENSE-2.0
+%
+% Unless required by applicable law or agreed to in writing, software
+% distributed under the License is distributed on an "AS IS" BASIS,
+% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+% implied. See the License for the specific language governing
+% permissions and limitations under the License.
+classdef tArray < matlab.unittest.TestCase
+
+ methods (Test)
+ function TestClassStructure(testCase)
+ array = arrow.c.Array();
+
+ % Verify array is an instance of arrow.c.Array.
+ testCase.verifyInstanceOf(array, "arrow.c.Array");
+
+ % Verify array has one public property named Address.
+ props = properties(array);
+ testCase.verifyEqual(props, {'Address'});
+ end
+
+ function TestAddressProperty(testCase)
+ array = arrow.c.Array();
+
+ % It's impossible to know what the value of Address will be.
+ % Just verify Address is a scalar uint64.
+ address = array.Address;
+ testCase.verifyInstanceOf(address, "uint64");
+ testCase.verifyTrue(isscalar(address));
+ end
+
+ function TestAddressNoSetter(testCase)
+ % Verify the Address property is read-only.
+ array = arrow.c.Array();
+ fcn = @() setfield(array, "Address", uint64(10));
+ testCase.verifyError(fcn, "MATLAB:class:SetProhibited");
+ end
+ end
+end
\ No newline at end of file
diff --git a/matlab/tools/cmake/BuildMatlabArrowInterface.cmake
b/matlab/tools/cmake/BuildMatlabArrowInterface.cmake
index e1641842ca..7a8cf8f403 100644
--- a/matlab/tools/cmake/BuildMatlabArrowInterface.cmake
+++ b/matlab/tools/cmake/BuildMatlabArrowInterface.cmake
@@ -75,7 +75,8 @@ set(MATLAB_ARROW_LIBMEXCLASS_CLIENT_PROXY_SOURCES
"${CMAKE_SOURCE_DIR}/src/cpp/a
"${CMAKE_SOURCE_DIR}/src/cpp/arrow/matlab/io/csv/proxy/table_writer.cc"
"${CMAKE_SOURCE_DIR}/src/cpp/arrow/matlab/io/csv/proxy/table_reader.cc"
"${CMAKE_SOURCE_DIR}/src/cpp/arrow/matlab/index/validate.cc"
-
"${CMAKE_SOURCE_DIR}/src/cpp/arrow/matlab/buffer/proxy/buffer.cc")
+
"${CMAKE_SOURCE_DIR}/src/cpp/arrow/matlab/buffer/proxy/buffer.cc"
+
"${CMAKE_SOURCE_DIR}/src/cpp/arrow/matlab/c/proxy/array.cc")
set(MATLAB_ARROW_LIBMEXCLASS_CLIENT_PROXY_FACTORY_INCLUDE_DIR
"${CMAKE_SOURCE_DIR}/src/cpp/arrow/matlab/proxy")
