Hi All, Following up on this mailing list discussion.
After several months of work, we now have a working version of libmexclass [1] to support development of the MATLAB Interface to Arrow. We've opened a pull request [2] for integrating libmexclass with the Arrow codebase. Making this a reality has involved a considerable amount of work by a number of MathWorkers including Fiona La, Sreehari Hegden, Sarah Gilmore, Jeremy Hughes, and many others. Thanks to everyone for their help in making this possible! Note that, libmexclass is still a relatively new project and continues to be tweaked and refined as we encounter new use cases. Please let us know if you have any questions. [1] https://github.com/mathworks/libmexclass [2] https://github.com/apache/arrow/pull/34563 Best Regards, Kevin Gurney ________________________________ From: Kevin Gurney <kgur...@mathworks.com> Sent: Friday, July 15, 2022 10:38 AM To: Sutou Kouhei <k...@clear-code.com>; dev@arrow.apache.org <dev@arrow.apache.org> Cc: Fiona La <fion...@mathworks.com>; Jeremy Hughes <jhug...@mathworks.com>; Nick Haddad <nhad...@mathworks.com> Subject: Re: [MATLAB] Integrating a framework for connecting MATLAB and C++ objects using MEX Hi Kou, Thank you for confirming the license compatibility! We will continue to keep the community updated as development efforts progress on libmexclass, and we prepare for integration with the upstream Arrow codebase. Best Regards, Kevin Gurney ________________________________ From: Sutou Kouhei <k...@clear-code.com> Sent: Wednesday, July 13, 2022 9:33 PM To: dev@arrow.apache.org <dev@arrow.apache.org> Cc: Kevin Gurney <kgur...@mathworks.com>; Fiona La <fion...@mathworks.com>; Jeremy Hughes <jhug...@mathworks.com>; Nick Haddad <nhad...@mathworks.com> Subject: Re: [MATLAB] Integrating a framework for connecting MATLAB and C++ objects using MEX Hi, Thanks for sharing progress! There is no problem with the BSD 3-Clause license because it's compatible with Apache License 2.0 and listed in Category A: https://www.apache.org/legal/resolved.html#category-a<https://www.apache.org/legal/resolved.html#category-a> > Category A: Licenses in Category A may be included in > Apache Software Foundation products. They are said to be > "Apache-like". Thanks, -- kou In <mn2pr05mb6496dfaad283ab5ce98713ecae...@mn2pr05mb6496.namprd05.prod.outlook.com> "Re: [MATLAB] Integrating a framework for connecting MATLAB and C++ objects using MEX" on Wed, 13 Jul 2022 19:20:40 +0000, Kevin Gurney <kgur...@mathworks.com> wrote: > Hi All, > > I am following up to close the loop on this. Apologies for the delay. We had > to work through some technical and procedural issues before releasing the > code. > > Updates: > > 1. We decided to release the code under the BSD 3-Clause [1], rather than the > BSD 2-Clause license. > > If there are any concerns about this licensing change, please let us know. > Our understanding is that BSD 3-Clause license should still be compatible > with the licensing of the upstream Arrow codebase. We apologize for the > confusion. > > 2. Initial code has been released under the MathWorks GitHub organization in > a repository named "libmexclass" [2]. > > We chose the name "libmexclass" because the project aims to enable users to > implement MATLAB classes in terms of calls to corresponding C++ classes using > MEX [3]. > > The code is under active development and is not yet ready for integration > with the Arrow codebase. However, we wanted to get the code on GitHub as soon > as possible so that anyone who is interested can feel free to follow > development progress. We welcome any contributions from Arrow community > members! > > Once the code has matured a bit more, we will work with the Arrow community > to update the build infrastructure for the MATLAB Interface to Arrow to make > use of libmexclass. Our hope is that using libmexclass will help unblock and > streamline development efforts for the MATLAB interface. > > Thank you again to the community for providing helpful feedback and enabling > us to move forward. > > [1] > https://github.com/mathworks/libmexclass/blob/main/LICENSE<https://github.com/mathworks/libmexclass/blob/main/LICENSE> > [2] > https://github.com/mathworks/libmexclass<https://github.com/mathworks/libmexclass> > [3] https://www.mathworks.com/help/matlab/call-mex-files-1.html > > Best Regards, > > Kevin Gurney > ________________________________ > From: Sutou Kouhei <k...@clear-code.com> > Sent: Sunday, June 12, 2022 10:12 PM > To: dev@arrow.apache.org <dev@arrow.apache.org> > Cc: Fiona La <fion...@mathworks.com>; Jeremy Hughes <jhug...@mathworks.com>; > Nick Haddad <nhad...@mathworks.com>; Kevin Gurney <kgur...@mathworks.com> > Subject: Re: [MATLAB] Integrating a framework for connecting MATLAB and C++ > objects using MEX > > +1 > > In > <mn2pr05mb6496c7164ff07547514ffaf7ae...@mn2pr05mb6496.namprd05.prod.outlook.com> > "Re: [MATLAB] Integrating a framework for connecting MATLAB and C++ objects > using MEX" on Fri, 10 Jun 2022 18:22:47 +0000, > Kevin Gurney <kgur...@mathworks.com> wrote: > >> Hi Kou, >> >> Thank you for helping to clear up our confusion. >> >>> How do we install the object dispatch layer to use it in >>> apache/arrow? I assumed that something like the following: >>> >>> ---- >>> $ git clone >>> https://github.com/mathworks/object-dispatch-layer.git<https://github.com/mathworks/object-dispatch-layer.git><https://github.com/mathworks/object-dispatch-layer.git<https://github.com/mathworks/object-dispatch-layer.git>> >>> $ cd object-dispatch-layer >>> $ cmake -S . -B build ... >>> $ cmake --build build >>> $ cmake --install build >>> $ git clone >>> https://github.com/apache/arrow.git<https://github.com/apache/arrow.git><https://github.com/apache/arrow.git<https://github.com/apache/arrow.git>> >>> $ cd apache/matlab >>> $ cmake -S . -B build # This find installed object-dispatch-layer >>> $ cmake --build build >>> $ cmake --install build >>> ---- >>> >>> My assumption is right? >> >> Your understanding is correct. Thanks for checking. >> >>> BTW, why do you want to use "git submodule" to use the >>> object dispatch layer? Why don't you install it separately >>> or build by externalproject_add() in CMake? >>> https://cmake.org/cmake/help/latest/module/ExternalProject.html<https://cmake.org/cmake/help/latest/module/ExternalProject.html><https://cmake.org/cmake/help/latest/module/ExternalProject.html<https://cmake.org/cmake/help/latest/module/ExternalProject.html>><https://cmake.org/cmake/help/latest/module/ExternalProject.html<https://cmake.org/cmake/help/latest/module/ExternalProject.html><https://cmake.org/cmake/help/latest/module/ExternalProject.html<https://cmake.org/cmake/help/latest/module/ExternalProject.html>>> >> >> After reflecting on your response, we realize that using a git submodule >> seems like a less than ideal solution. >> >> Initially, we were thinking that if code within the apache/arrow repository >> were to "depend" on some MATLAB files from the object dispatch layer, that >> we would need to "physically" (via vendoring or IP Clearance) or "virtually" >> (via git submodule) integrate this code into the apache/arrow source tree. >> However, since these files are only needed at build time / run time, this >> means that the object dispatch layer code does not necessarily need to be >> redistributable along with the rest of the code in the apache/arrow >> repository. >> >> It seems much clearer now that the object dispatch layer can be treated as a >> "pure" external "library" dependency, and thus, the code should not need to >> be present in the apache/arrow repository. Per your suggestion, at build / >> install time, it should be possible to copy any required MATLAB files or C++ >> header files to appropriate locations, so that they can be used by CMake and >> MATLAB. >> >> Using externalproject_add() is a great idea and seems more "in-model" than >> repeatedly bumping the version of a vendored copy of the object dispatch >> layer source or using a git submodule. >> >> To summarize, it sounds like a reasonable path forward would be to: >> >> 1. Develop the object dispatch layer in an external repository underneath >> the MathWorks GitHub organization, with a 2-Clause BSD license. >> 2. Use externalproject_add() to fetch and build the source code dynamically. >> >> Once the object dispatch layer is available on GitHub, I will follow up on >> this email thread with a link to the repository so that anyone in the >> community can track development progress, as well as contribute to the >> framework, if they are interested. >> >> If anyone has any objections to this approach, please let us know. >> >> Thank you! >> >> Kevin Gurney >> >> ________________________________ >> From: Sutou Kouhei <k...@clear-code.com> >> Sent: Friday, June 10, 2022 4:13 AM >> To: dev@arrow.apache.org <dev@arrow.apache.org> >> Cc: Kevin Gurney <kgur...@mathworks.com>; Fiona La <fion...@mathworks.com>; >> Jeremy Hughes <jhug...@mathworks.com>; Nick Haddad <nhad...@mathworks.com> >> Subject: Re: [MATLAB] Integrating a framework for connecting MATLAB and C++ >> objects using MEX >> >> Hi, >> >>> 1. A developer would author a custom MEX function that >>> uses C++ "building blocks" (i.e. classes and header files) >>> from the object dispatch layer "framework". They would >>> link their custom MEX function against a helper shared >>> library that is built from the source code of the object >>> dispatch layer and provides the symbols/implementation for >>> the aforementioned C++ "building blocks". >> >> How do we install the object dispatch layer to use it in >> apache/arrow? I assumed that something like the following: >> >> ---- >> $ git clone >> https://github.com/mathworks/object-dispatch-layer.git<https://github.com/mathworks/object-dispatch-layer.git><https://github.com/mathworks/object-dispatch-layer.git<https://github.com/mathworks/object-dispatch-layer.git>><https://github.com/mathworks/object-dispatch-layer.git<https://github.com/mathworks/object-dispatch-layer.git><https://github.com/mathworks/object-dispatch-layer.git<https://github.com/mathworks/object-dispatch-layer.git>>> >> $ cd object-dispatch-layer >> $ cmake -S . -B build ... >> $ cmake --build build >> $ cmake --install build >> $ git clone >> https://github.com/apache/arrow.git<https://github.com/apache/arrow.git><https://github.com/apache/arrow.git<https://github.com/apache/arrow.git>><https://github.com/apache/arrow.git<https://github.com/apache/arrow.git><https://github.com/apache/arrow.git<https://github.com/apache/arrow.git>>> >> $ cd apache/matlab >> $ cmake -S . -B build # This find installed object-dispatch-layer >> $ cmake --build build >> $ cmake --install build >> ---- >> >> My assumption is right? >> >>> Essentially, for a developer to use the object dispatch >>> layer, they will need to author a fair amount of custom >>> code which makes use of both MATLAB and C++ "building >>> blocks" from the "framework". >> >> I think that this is a normal library usage. For example, >> our S3 filesystem module implementation in C++ has about >> 2500 lines and uses classes provides by AWS SDK C++: >> https://github.com/apache/arrow/blob/master/cpp/src/arrow/filesystem/s3fs.cc<https://github.com/apache/arrow/blob/master/cpp/src/arrow/filesystem/s3fs.cc><https://github.com/apache/arrow/blob/master/cpp/src/arrow/filesystem/s3fs.cc<https://github.com/apache/arrow/blob/master/cpp/src/arrow/filesystem/s3fs.cc>><https://github.com/apache/arrow/blob/master/cpp/src/arrow/filesystem/s3fs.cc<https://github.com/apache/arrow/blob/master/cpp/src/arrow/filesystem/s3fs.cc><https://github.com/apache/arrow/blob/master/cpp/src/arrow/filesystem/s3fs.cc<https://github.com/apache/arrow/blob/master/cpp/src/arrow/filesystem/s3fs.cc>>> >> >> >>> If we had to go through the IP Clearance Process, would >>> that mean we would need to repeatedly clear the code every >>> time we wanted to sync up the git submodule with the >>> latest source code from the external repository? It seems >>> like this would quickly become impractical since we >>> anticipate the need to iterate frequently on the object >>> dispatch layer early on. >> >> If the object dispatch layer doesn't depend on Apache Arrow >> and is a general purpose framework, we can vendor it without >> IP clearance. >> e.g.: >> https://github.com/apache/arrow/tree/master/cpp/src/arrow/vendored/xxhash<https://github.com/apache/arrow/tree/master/cpp/src/arrow/vendored/xxhash><https://github.com/apache/arrow/tree/master/cpp/src/arrow/vendored/xxhash<https://github.com/apache/arrow/tree/master/cpp/src/arrow/vendored/xxhash>><https://github.com/apache/arrow/tree/master/cpp/src/arrow/vendored/xxhash<https://github.com/apache/arrow/tree/master/cpp/src/arrow/vendored/xxhash><https://github.com/apache/arrow/tree/master/cpp/src/arrow/vendored/xxhash<https://github.com/apache/arrow/tree/master/cpp/src/arrow/vendored/xxhash>>> >> >> BTW, why do you want to use "git submodule" to use the >> object dispatch layer? Why don't you install it separately >> or build by externalproject_add() in CMake? >> https://cmake.org/cmake/help/latest/module/ExternalProject.html<https://cmake.org/cmake/help/latest/module/ExternalProject.html><https://cmake.org/cmake/help/latest/module/ExternalProject.html<https://cmake.org/cmake/help/latest/module/ExternalProject.html>><https://cmake.org/cmake/help/latest/module/ExternalProject.html<https://cmake.org/cmake/help/latest/module/ExternalProject.html><https://cmake.org/cmake/help/latest/module/ExternalProject.html<https://cmake.org/cmake/help/latest/module/ExternalProject.html>>> >> >> >> Thanks, >> -- >> kou >> >> In >> <byapr05mb648755bee2e0caccb0f407d9ae...@byapr05mb6487.namprd05.prod.outlook.com> >> "Re: [MATLAB] Integrating a framework for connecting MATLAB and C++ objects >> using MEX" on Wed, 8 Jun 2022 15:59:13 +0000, >> Kevin Gurney <kgur...@mathworks.com> wrote: >> >>> Hi Kou, >>> >>> --- >>> >>> Note: I am replying to your email as a forward from Fiona (Cc'd) since your >>> original email was accidentally blocked by my email client). >>> >>> --- >>> >>> The way that we expected the object dispatch layer to be used by client >>> code is as follows: >>> >>> 1. A developer would author a custom MEX function that uses C++ "building >>> blocks" (i.e. classes and header files) from the object dispatch layer >>> "framework". They would link their custom MEX function against a helper >>> shared library that is built from the source code of the object dispatch >>> layer and provides the symbols/implementation for the aforementioned C++ >>> "building blocks". >>> >>> 2. The object dispatch layer expects the compiled MEX function to have a >>> specific name and be available on the MATLAB Search Path [1] so that it can >>> be used by the MATLAB side of the object dispatch layer. >>> >>> 3. Once the MEX function is available on the MATLAB Search Path, client >>> MATLAB code can use a set of MATLAB "building blocks" (i.e. classes), which >>> are part of the object dispatch layer "framework", to connect a MATLAB >>> class with a corresponding C++ class. >>> >>> Essentially, for a developer to use the object dispatch layer, they will >>> need to author a fair amount of custom code which makes use of both MATLAB >>> and C++ "building blocks" from the "framework". >>> >>> It's not clear to me whether the steps described above classify as "library >>> usage" with regard to the IP Clearance Process. >>> >>> If we had to go through the IP Clearance Process, would that mean we would >>> need to repeatedly clear the code every time we wanted to sync up the git >>> submodule with the latest source code from the external repository? It >>> seems like this would quickly become impractical since we anticipate the >>> need to iterate frequently on the object dispatch layer early on. >>> >>> It's quite possible that I am not answering your questions completely, so >>> please let me know if anything is unclear. My apologies in advance for any >>> confusion. >>> >>> [1] >>> https://www.mathworks.com/help/matlab/matlab_env/what-is-the-matlab-search-path.html >>> >>> Best, >>> >>> Kevin Gurney >>> >>> ________________________________ >>> From: Fiona La <fion...@mathworks.com> >>> Sent: Wednesday, June 8, 2022 11:24 AM >>> To: Kevin Gurney <kgur...@mathworks.com> >>> Subject: FW: [MATLAB] Integrating a framework for connecting MATLAB and C++ >>> objects using MEX >>> >>> >>> >>> >>> >>> >>> From: Sutou Kouhei <k...@clear-code.com> >>> Date: Tuesday, June 7, 2022 at 8:36 PM >>> To: dev@arrow.apache.org <dev@arrow.apache.org> >>> Cc: Fiona La <fion...@mathworks.com>, Jeremy Hughes >>> <jhug...@mathworks.com>, Nick Haddad <nhad...@mathworks.com> >>> Subject: Re: [MATLAB] Integrating a framework for connecting MATLAB and C++ >>> objects using MEX >>> >>> Hi, >>> >>> Can we use the object dispatch layer as a library? Or should >>> we copy (or submodule) the object dispatch layer to >>> apache/arrow? >>> >>> If we can use the object dispatch layer as a library, we can >>> just use it as an external library like GoogleTest. We don't >>> need IP clearance. You can use any Apache License 2.0 >>> compatible license for the object dispatch layer. >>> >>> Thanks, >>> -- >>> kou >>> >>> In >>> <mn2pr05mb6496cff053c60e54c133c93cae...@mn2pr05mb6496.namprd05.prod.outlook.com> >>> "[MATLAB] Integrating a framework for connecting MATLAB and C++ objects >>> using MEX" on Tue, 7 Jun 2022 18:10:43 +0000, >>> Kevin Gurney <kgur...@mathworks.com> wrote: >>> >>>> Hi All, >>>> >>>> I am reaching out to seek guidance from the community regarding a code >>>> integration puzzle. >>>> >>>> The architecture that we are currently pursuing for the MATLAB interface >>>> to Arrow [1] involves dispatching to the Arrow C++ libraries using MEX (a >>>> MATLAB facility for calling C/C++ code [2]). A major challenge with this >>>> approach has been keeping Arrow C++ objects (e.g. arrow::Array) alive in >>>> memory for the appropriate amount of time and making it easy to interface >>>> with them from MATLAB. >>>> >>>> MATLAB has a recommended solution for this problem [3]. However, we've >>>> been pursuing a MEX-based solution due to the pervasiveness of MEX and its >>>> familiarity to MATLAB users. Our hope is that using MEX will make it easy >>>> for others to contribute to the MATLAB interface. >>>> >>>> To help maintain the connection between MATLAB objects and C++, we've been >>>> experimenting with a MEX-based object dispatch layer. The primary goal of >>>> this work is to unblock development of the MATLAB interface to Arrow. >>>> However, this object dispatch layer is non-trivial and ultimately >>>> unrelated to the Arrow project's core mission. Therefore, submitting this >>>> code to the Arrow project doesn't seem like the optimal code integration >>>> strategy. >>>> >>>> We’ve been considering the possibility of creating a new open-source >>>> repository under the MathWorks GitHub organization [4] to host the object >>>> dispatch layer (a side effect of this approach is that it may help >>>> encourage reuse of this infrastructure in future open-source MATLAB >>>> projects). >>>> >>>> However, this approach would come with notable tradeoffs: >>>> >>>> 1. We would need to follow the ASF IP Clearance Process [5] to integrate >>>> this code into the Arrow project (it's possible we are mistaken about >>>> this). >>>> >>>> 2. It's not obvious how we should keep the code in sync. Would it be >>>> possible to use a git submodule [6] to "symlink" to the external repo? >>>> >>>> 3. What about licensing? Does the code need to be Apache licensed, or >>>> would it be possible to use another Apache-compatible license [7], like >>>> BSD? BSD is the default choice for new projects hosted under the MathWorks >>>> GitHub organization. >>>> >>>> Admittedly, we aren't sure what the best path forward is, so we appreciate >>>> the community's guidance. We welcome any suggestions. >>>> >>>> [1] >>>> https://github.com/apache/arrow/tree/master/matlab<https://github.com/apache/arrow/tree/master/matlab><https://github.com/apache/arrow/tree/master/matlab<https://github.com/apache/arrow/tree/master/matlab>><https://github.com/apache/arrow/tree/master/matlab<https://github.com/apache/arrow/tree/master/matlab><https://github.com/apache/arrow/tree/master/matlab<https://github.com/apache/arrow/tree/master/matlab>>><https://github.com/apache/arrow/tree/master/matlab<https://github.com/apache/arrow/tree/master/matlab><https://github.com/apache/arrow/tree/master/matlab<https://github.com/apache/arrow/tree/master/matlab>><https://github.com/apache/arrow/tree/master/matlab<https://github.com/apache/arrow/tree/master/matlab><https://github.com/apache/arrow/tree/master/matlab<https://github.com/apache/arrow/tree/master/matlab>>>> >>>> [2] https://www.mathworks.com/help/matlab/call-mex-files-1.html >>>> [3] >>>> https://www.mathworks.com/help/matlab/build-matlab-interface-to-c-library.html >>>> [4] >>>> https://github.com/mathworks<https://github.com/mathworks><https://github.com/mathworks<https://github.com/mathworks>><https://github.com/mathworks<https://github.com/mathworks><https://github.com/mathworks<https://github.com/mathworks>>><https://github.com/mathworks<https://github.com/mathworks><https://github.com/mathworks<https://github.com/mathworks>><https://github.com/mathworks<https://github.com/mathworks><https://github.com/mathworks<https://github.com/mathworks>>>> >>>> [5] >>>> https://incubator.apache.org/ip-clearance/<https://incubator.apache.org/ip-clearance><https://incubator.apache.org/ip-clearance<https://incubator.apache.org/ip-clearance>><https://incubator.apache.org/ip-clearance<https://incubator.apache.org/ip-clearance><https://incubator.apache.org/ip-clearance<https://incubator.apache.org/ip-clearance>>><https://incubator.apache.org/ip-clearance<https://incubator.apache.org/ip-clearance><https://incubator.apache.org/ip-clearance<https://incubator.apache.org/ip-clearance>><https://incubator.apache.org/ip-clearance<https://incubator.apache.org/ip-clearance><https://incubator.apache.org/ip-clearance<https://incubator.apache.org/ip-clearance>>>> >>>> [6] >>>> https://github.blog/2016-02-01-working-with-submodules/<https://github.blog/2016-02-01-working-with-submodules><https://github.blog/2016-02-01-working-with-submodules<https://github.blog/2016-02-01-working-with-submodules>><https://github.blog/2016-02-01-working-with-submodules<https://github.blog/2016-02-01-working-with-submodules><https://github.blog/2016-02-01-working-with-submodules<https://github.blog/2016-02-01-working-with-submodules>>><https://github.blog/2016-02-01-working-with-submodules<https://github.blog/2016-02-01-working-with-submodules><https://github.blog/2016-02-01-working-with-submodules<https://github.blog/2016-02-01-working-with-submodules>><https://github.blog/2016-02-01-working-with-submodules<https://github.blog/2016-02-01-working-with-submodules><https://github.blog/2016-02-01-working-with-submodules<https://github.blog/2016-02-01-working-with-submodules>>>> >>>> [7] >>>> https://www.apache.org/legal/resolved.html#category-a<https://www.apache.org/legal/resolved.html#category-a><https://www.apache.org/legal/resolved.html#category-a<https://www.apache.org/legal/resolved.html#category-a>><https://www.apache.org/legal/resolved.html#category-a<https://www.apache.org/legal/resolved.html#category-a><https://www.apache.org/legal/resolved.html#category-a<https://www.apache.org/legal/resolved.html#category-a>>><https://www.apache.org/legal/resolved.html#category-a<https://www.apache.org/legal/resolved.html#category-a><https://www.apache.org/legal/resolved.html#category-a<https://www.apache.org/legal/resolved.html#category-a>><https://www.apache.org/legal/resolved.html#category-a<https://www.apache.org/legal/resolved.html#category-a><https://www.apache.org/legal/resolved.html#category-a<https://www.apache.org/legal/resolved.html#category-a>>>> >>>> >>>> Thank you, >>>> >>>> Kevin Gurney
