Hi All,
Thanks again for all of the feedback we have received so far on the design doc
- it's been really helpful.
Fiona (Cc'd) and I took an initial pass at addressing the comments from Wes and
Antoine. @Wes and @Antoine - whenever you get a chance, we would appreciate it
if you could take a look over our responses to your feedback. Don't hesitate to
let us know if you have any additional questions or concerns.
Some notable changes we have made based on the feedback received so far:
1. Added a section about the ArrowC Data Interface and In-Process Arrow
Memory Sharing.
2. Modified the Out-of-Process Arrow Memory Sharing example to use the
Arrow IPC File Format rather than Feather V2.
3. Added a note that Plasma is informally deprecated, and it may not make
sense to invest effort in supporting it right now.
We continue to welcome anyone else in the community to add your thoughts and
comments to the design doc.
To keep things moving along, we would like to set a "soft" deadline for
feedback on this high level design doc for next **Tuesday, February 9th,
2021**. After this "soft" deadline, everyone is still free to raise concerns as
they come up, but we'll transition towards focusing on initial implementation
(i.e. we'll assume it is reasonable to start prototyping and opening pull
requests as appropriate).
Note: We realize that Word Online isn't turning out to be an ideal solution for
collaborating on design docs. The lack of support for including names with
comments is proving to be cumbersome (our apologies, again). In addition, for
long term archival purposes, it would be best to move the document to a better
location. Fiona and I believe that using Markdown to write future design docs
and user guides would be preferrable, as we can get all the benefits of normal
GitHub version control, as well as have the docs live right alongside the
MATLAB code. Commenting and general collaboration should be easier this way
too. Our plan is to migrate this initial design doc to Markdown after feedback
from the community has settled.
Thanks again to everyone for your help in getting this project off the ground!
Best Regards,
Kevin Gurney
________________________________
From: Kevin Gurney <kgur...@mathworks.com>
Sent: Tuesday, January 26, 2021 11:47 AM
To: dev <dev@arrow.apache.org>
Cc: Antoine Pitrou <anto...@python.org>; Jeremy Hughes <jhug...@mathworks.com>;
Nick Haddad <nhad...@mathworks.com>; Penny Anderson <pe...@mathworks.com>;
Fiona La <fion...@mathworks.com>; Tahsin Hassan <thas...@mathworks.com>; Yann
Debray <ydeb...@mathworks.com>; Wes McKinney <wesmck...@gmail.com>
Subject: Re: [MATLAB] Developing a MATLAB Interface for Apache Arrow
Hi Wes,
Thanks very much for taking the time to share your feedback!
Looking forward to incorporating more feedback from the community. Excited to
work together to move this project forward!
Best Regards,
Kevin Gurney
________________________________
From: Wes McKinney <wesmck...@gmail.com>
Sent: Monday, January 25, 2021 4:48 PM
To: dev <dev@arrow.apache.org>
Cc: Antoine Pitrou <anto...@python.org>; Jeremy Hughes <jhug...@mathworks.com>;
Nick Haddad <nhad...@mathworks.com>; Penny Anderson <pe...@mathworks.com>;
Fiona La <fion...@mathworks.com>; Tahsin Hassan <thas...@mathworks.com>; Yann
Debray <ydeb...@mathworks.com>
Subject: Re: [MATLAB] Developing a MATLAB Interface for Apache Arrow
hi Kevin -- I read through the document. It seems plenty reasonable to
me. Look forward to seeing the buildout.
Thanks
Wes
On Mon, Jan 25, 2021 at 3:10 PM Kevin Gurney <kgur...@mathworks.com> wrote:
>
> Hi Antoine,
>
> Thanks very much for taking a first pass over the document! I'll start
> working through the feedback you've provided soon.
>
> It's useful to have a variety of perspectives here, including from Arrow C++
> developers, like yourself. Also, I agree - having more MATLAB users provide
> feedback on the document would be great to help ensure we are covering all
> necessary requirements to make the interface as useful as possible.
>
> If anyone has any other ideas or suggestions, please don't hesitate to share
> your feedback on the document.
>
> Best Regards,
>
> Kevin Gurney
> ________________________________
> From: Antoine Pitrou <anto...@python.org>
> Sent: Monday, January 25, 2021 12:53 PM
> To: dev@arrow.apache.org <dev@arrow.apache.org>; Kevin Gurney
> <kgur...@mathworks.com>
> Cc: Jeremy Hughes <jhug...@mathworks.com>; Nick Haddad
> <nhad...@mathworks.com>; Penny Anderson <pe...@mathworks.com>; Fiona La
> <fion...@mathworks.com>; Tahsin Hassan <thas...@mathworks.com>; Yann Debray
> <ydeb...@mathworks.com>
> Subject: Re: [MATLAB] Developing a MATLAB Interface for Apache Arrow
>
>
> Hi Kevin,
>
> I've added some comments to the document. Bear in mind that I'm not a
> MATLAB user, so this is some outside view from a Arrow C++ developer.
>
> It would be nice if MATLAB users could comment on the document,
> especially the proposed APIs.
>
> Regards
>
> Antoine.
>
>
> Le 22/01/2021 à 23:19, Kevin Gurney a écrit :
> > Hi Antoine,
> >
> > Thanks for your input!
> >
> > As you pointed out, I am in fact familiar with the matlab/ directory! :-)
> > Several MathWorkers, including myself, helped contribute to this code a
> > while back. We are hoping to use it as a starting point as we build out a
> > more fully fledged MATLAB interface to Arrow memory.
> >
> > Based on your suggestion, I've included a link to a Word Online version of
> > the design document below:
> >
> > https://mathworks-my.sharepoint.com/:w:/p/kgurney/EcNXJh5S-HBCit-YNL6ZYnEB4Mv9ZPTVEs7a72SWlywIsg<https://mathworks-my.sharepoint.com/:w:/p/kgurney/EcNXJh5S-HBCit-YNL6ZYnEB4Mv9ZPTVEs7a72SWlywIsg><https://mathworks-my.sharepoint.com/:w:/p/kgurney/EcNXJh5S-HBCit-YNL6ZYnEB4Mv9ZPTVEs7a72SWlywIsg<https://mathworks-my.sharepoint.com/:w:/p/kgurney/EcNXJh5S-HBCit-YNL6ZYnEB4Mv9ZPTVEs7a72SWlywIsg>>
> >
> > As far as I can tell, this link should allow commenting by anyone.
> > Unfortunately, I'm not sure if the names of reviewers will be included when
> > they comment. If this turns out to be the case, it would be great if
> > reviewers could prefix their comments with something like [FirstName
> > LastName] so we can track feedback appropriately.
> >
> > Don't hesitate to let me know if you have any issues accessing or
> > commenting on the document. My apologies for the inconvenience in getting
> > this properly shared.
> >
> > Best Regards,
> >
> > Kevin Gurney
> > ________________________________
> > From: Antoine Pitrou <anto...@python.org>
> > Sent: Friday, January 22, 2021 11:28 AM
> > To: dev@arrow.apache.org <dev@arrow.apache.org>; Kevin Gurney
> > <kgur...@mathworks.com>
> > Cc: Jeremy Hughes <jhug...@mathworks.com>; Nick Haddad
> > <nhad...@mathworks.com>; Penny Anderson <pe...@mathworks.com>; Fiona La
> > <fion...@mathworks.com>; Tahsin Hassan <thas...@mathworks.com>; Yann Debray
> > <ydeb...@mathworks.com>
> > Subject: Re: [MATLAB] Developing a MATLAB Interface for Apache Arrow
> >
> >
> > Hello Kevin,
> >
> > You could use a Google Docs or similar to share the design document and
> > allow people to comment. Inside a Google Doc, you can use "File ->
> > Share" to create a sharable URL with specific permissions (such as
> > commenting but not editing).
> >
> > I was about to mention the matlab/ directory in the Arrow repository but
> > I see you're the main author, so you already know about it :-)
> >
> > Best regards
> >
> > Antoine.
> >
> >
> > Le 22/01/2021 à 16:05, Kevin Gurney a écrit :
> >> It seems like the mailing list stripped out the design doc I attached for
> >> some reason.
> >>
> >> Here is a link to the same document hosted online instead:
> >>
> >> https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4<https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4><https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4<https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4>><https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4<https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4><https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4<https://mathworks-my.sharepoint.com/:b:/p/kgurney/EU3Kdz0cubRJrkEyI1bNR88BKnH4S2siU2EHHNQwxTgHUg?e=wzLDx4>>>
> >>
> >> Note: This link is only a temporary solution (will expire on February 21,
> >> 2021). It would be ideal if we could move this to a better place like the
> >> Arrow Confluence Design Documents area.
> >>
> >> Thanks,
> >>
> >> Kevin
> >> ________________________________
> >> From: Kevin Gurney <kgur...@mathworks.com>
> >> Sent: Thursday, January 21, 2021 4:47 PM
> >> To: dev@arrow.apache.org <dev@arrow.apache.org>
> >> Cc: Jeremy Hughes <jhug...@mathworks.com>; Nick Haddad
> >> <nhad...@mathworks.com>; Penny Anderson <pe...@mathworks.com>; Fiona La
> >> <fion...@mathworks.com>; Tahsin Hassan <thas...@mathworks.com>; Yann
> >> Debray <ydeb...@mathworks.com>
> >> Subject: [MATLAB] Developing a MATLAB Interface for Apache Arrow
> >>
> >> Hello All,
> >>
> >> MathWorks is interested in collaborating with the rest of the Arrow
> >> community to build out a MATLAB interface to Arrow memory. We envision an
> >> interface analogous to the other language bindings, with packaged classes
> >> and functions like:
> >>
> >> * arrow.Array
> >> * arrow.TableReader
> >> * arrow.type.Float64
> >> * ...
> >>
> >> In the past, several MathWorkers worked with the Arrow community to
> >> develop a proof-of-concept MATLAB interface for reading/writing Feather V1
> >> files by leveraging the Arrow C++ libraries. Since then, the Arrow project
> >> has evolved considerably, and we'd like to work with the community to
> >> expand MATLAB's ability to interoperate with the broader Arrow ecosystem.
> >>
> >> Attached to this email is a lightweight design document which lays out a
> >> high-level direction for these development efforts. We welcome any and all
> >> feedback on this document.
> >>
> >> It would be great to move this design document to some place that is more
> >> easily accessible and publicly archived for all members of the Arrow
> >> community. At first glance, the Arrow Confluence Design Documents area
> >> (https://cwiki.apache.org/confluence/display/ARROW/Design+Documents<https://cwiki.apache.org/confluence/display/ARROW/Design+Documents><https://cwiki.apache.org/confluence/display/ARROW/Design+Documents<https://cwiki.apache.org/confluence/display/ARROW/Design+Documents>><https://cwiki.apache.org/confluence/display/ARROW/Design+Documents<https://cwiki.apache.org/confluence/display/ARROW/Design+Documents><https://cwiki.apache.org/confluence/display/ARROW/Design+Documents<https://cwiki.apache.org/confluence/display/ARROW/Design+Documents>>>)
> >> seems like the ideal place. However, if you have other suggestions of how
> >> best to collaborate on this document, please let me know.
> >>
> >> We are excited to work together with the rest of the Arrow community to
> >> make this a reality.
> >>
> >> Best Regards,
> >>
> >> Kevin Gurney
> >>
> >
