I agree that this problem could reasonably (and perhaps best) be solved with J code. The numpy binary spec seems straightforward and maps closely to the j binary rep. The structure level stuff (as opposed to data copying) is closely related to what is done for json.
This might be an interesting challenge to post to the programming forum. On Fri, Sep 25, 2020 at 12:10 PM Henry Rich <[email protected]> wrote: > Have you considered performing this function in J? It seems like it's > mostly data-copying. > > Henry Rich > > On 9/25/2020 11:52 AM, Raul Miller wrote: > > 3!:2 can construct an arbitrary nested J array and 3!:1 can serialize > > an arbitrary nested J array. The cost of using this mechanism is > > typically considerably lower than the cost of reading a file. > > > > (There's another potential approach, involving cdcallback, but the > > overhead and complexity of using that mechanism would not likely lead > > to a better result.) > > > > I hope this helps. > > > > > > -- > > Raul > > > > On Thu, Sep 24, 2020 at 4:51 PM 'Zhihao Yuan' via Source > > <[email protected]> wrote: > >> On Thu, Sep 24, 2020 at 12:44 PM Eric Iverson <[email protected] > > > >> wrote: > >> > >>> It should be possible to package it as an addon. The addon would > include > >>> shared libraries for the platforms of interest (windows/linux/macos) > and > >>> simple cover functions that did the cd calls to the libraries. This is > how > >>> Jd/lapack/etc provide extensions like this. > >> > >> You are essentially asking me to write a > >> new library in C, since there is no such > >> library from the beginning -- which may > >> be fine. But here are the problems I > >> encountered: > >> > >> 1. How do I allocate a J array before > >> knowing its type and shape? Because > >> DLL (15!:) interface wants J side to pass > >> such a thing into cd. The short answer is > >> I can't. I must use separated calls to > >> negotiate the type and shape, which > >> leads to the next question... > >> > >> 2. How to pass FILE* to DLL from J? > >> All I can access from J code is file number; > >> DLL doesn't know what that is. This implies > >> that I may have to pass only filenames and > >> open a file twice -- first to parse its header, > >> then to load its data. But there is no > >> guarantee that the file is not renamed > >> between the two calls, or corresponds to > >> the same file. From a C library's point of > >> view, such an API will be poorly designed. > >> > >> I'm leaning towards adding this > >> functionality as 3!:12 and 3!:13, converting > >> a J array to bytes in NPYand retrieving a > >> J array from bytes. > >> > >> -- > >> Zhihao > >> ---------------------------------------------------------------------- > >> For information about J forums see http://www.jsoftware.com/forums.htm > > ---------------------------------------------------------------------- > > For information about J forums see http://www.jsoftware.com/forums.htm > > > -- > This email has been checked for viruses by AVG. > https://www.avg.com > > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm > ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
