libzahl is not thread safe, and even if it was, it's not clear to me that it adequately supports enough architectures.
Meanwhile, libgmp's problems are addressable. I just have to use a different part of its API. (Also, on windows, we're using mpir rather than libgmp.) (J currently uses parts of the libgmp high level API, which performs memory allocations within the libgmp library routines, using callbacks whose implementation I supply. But it also exposes the low level routines used to build those high level routines, and those low level routines do not perform memory allocation, which means that we can manage the memory outside of the API.) ((The problem with libgmp's high level API is that if a memory allocation fails, it exits the program. So we came up with a workaround which reserves a memory pool, and limits arguments to certain routines, so successful memory allocations will succeed even under low memory conditions. That's not ideal, but it has been "good enough, so far". But libgmp supports another approach. It's a little more work, but not an excessive amount of work.)) I hope this makes sense, -- Raul On Tue, Oct 24, 2023 at 6:02 PM 'Viktor Grigorov' via Source <[email protected]> wrote: > > Have you explored other libraries? Ever since addition of GMP, I've been > thinking about libzahl[1], and how much work a rewrite using its function > would be required. I've used no extended integer number library, and haven't > had the need to, so I've not touched either, or their alternatives. > > [1] https://github.com/maandree/libzahl > > Oct 25, 2023, 00:21 by [email protected]: > > > Ah.. > > > > I was planning on using J's memory allocation routines (perhaps jtgafv?). > > > > Those libgmp low level functions are low-level functions which do not > > perform any memory allocation. For example: mpn_add_n adds two > > integers but requires memory allocation to be performed elsewhere. > > > > This is what would allow us (me) to use J's memory allocation routines. > > > > I hope this makes sense. > > > > And, I guess I should dive in and try something. If nothing else, to > > see what breaks... (I've made some small preparatory experiments, but > > I could obviously do a lot more.) > > > > -- > > Raul > > > > On Tue, Oct 24, 2023 at 1:44 PM Henry Rich <[email protected]> wrote: > > > >> > >> You referred to a page on low level functions. I don't see how it applies > >> to allocation. > >> > >> hhr > >> > >> On Tue, Oct 24, 2023, 4:15 PM Raul Miller <[email protected]> wrote: > >> > >> > I've lost track of the thread here. > >> > > >> > Could you expand your question? > >> > > >> > Thanks, > >> > > >> > -- > >> > Raul > >> > > >> > On Tue, Oct 24, 2023 at 11:08 AM Henry Rich <[email protected]> wrote: > >> > > > >> > > I don't see a memory allocator among the functions. How does this page > >> > help > >> > > with allocations? > >> > > > >> > > hhr > >> > > > >> > > On Mon, Oct 23, 2023, 6:10 PM Eric Iverson <[email protected]> > >> > wrote: > >> > > > >> > > > I think gmp has proven itself and the cleanup you suggest would be > >> > > > worthwhile. > >> > > > > >> > > > On Sun, Oct 15, 2023 at 11:12 AM Raul Miller <[email protected]> > >> > > > wrote: > >> > > > > >> > > > > Some time ago, Elijah Stone pointed out that using > >> > > > > https://gmplib.org/manual/Low_002dlevel-Functions we could use J's > >> > > > > memory management routines directly, without having to deal with > >> > > > > libgmp's "exit the program if the library can't allocate memory" > >> > > > > behavior. > >> > > > > > >> > > > > This seems like it would be a good thing for us. When completed, we > >> > > > > could eliminate the memory pool currently used when handling > >> > > > > extended > >> > > > > values, and we could also relax the current limit placed on the > >> > > > > magnitude of extended values. > >> > > > > > >> > > > > But changing everything all at once is a good way to never get > >> > started. > >> > > > > > >> > > > > So, it's worth thinking about how we could organize this kind of > >> > effort. > >> > > > > > >> > > > > Currently, the code is partitioned in three chunks: libgmp itself > >> > > > > (or > >> > > > > mpir on windows), the jgmp.h/jgmpinit.h glue, and macros defined in > >> > > > > jgmp.h which are used in most of the rest of the system. (There's > >> > also > >> > > > > a few direct calls to libgmp functions in k.c, v2.c, vq.c, vx.c and > >> > > > > wn.c) > >> > > > > > >> > > > > So, conceptually speaking, we could implement workalikes for these > >> > > > > macros (things like XaddXX() which rely on the lower level mpn_ > >> > > > > functions instead of the problematic mpz_ / mpq_ functions. (We > >> > > > > could > >> > > > > replace the direct calls with suitable macros, along the way. (Or, > >> > > > > if > >> > > > > there's cases where there's really a significant performance > >> > > > > advantage, we could replace them with suitable direct calls to the > >> > > > > memory management routines and mpn_ functions. But this seems > >> > > > > unlikely.)) > >> > > > > > >> > > > > The trick would be allowing XNUM and RAT values whose memory was > >> > > > > allocated via libgmp to coexist with XNUM and RAT values whose > >> > > > > memory > >> > > > > was allocated using J's memory manager. The details here are a bit > >> > > > > annoying, but fundamentally we've already provided for this. > >> > > > > > >> > > > > Basically, the distinction matters when we free the memory. And, > >> > > > > that > >> > > > > decision is based on FHRHISGMP==AFHRH(x) vs FHRHISGMP!=AFHRH(x) in > >> > > > > jgmp.h and jgmpinit.c > >> > > > > > >> > > > > ----------------------------------------------- > >> > > > > > >> > > > > So.. it seems to me that if we created a parallel glue rig -- maybe > >> > > > > jgmpn.h -- we could start migrating functionality to the mpn_* > >> > > > > family > >> > > > > of functions and J's "native" memory management. XaddXX() seems > >> > > > > like > >> > > > > the place to start. > >> > > > > > >> > > > > I would need to figure out how to deal with the "realloc" cases > >> > > > > where > >> > > > > the amount of memory required for a calculation (like multiply or > >> > > > > format) might be larger or smaller - perhaps significantly larger - > >> > > > > than the memory needed to represent the result. > >> > > > > > >> > > > > But, once started, the work could proceed gradually. As long as the > >> > > > > primitives continue to function, users mostly wouldn't notice the > >> > > > > changes. (And, ok, that sounds discouraging. But hopefully the end > >> > > > > result would be worth it.) > >> > > > > > >> > > > > Thoughts? > >> > > > > > >> > > > > My thought is that this kind of code cleanup seems worthwhile. > >> > > > > > >> > > > > Thanks, > >> > > > > > >> > > > > -- > >> > > > > Raul > >> > > > > > >> > ---------------------------------------------------------------------- > >> > > > > For information about J forums see > >> > http://www.jsoftware.com/forums.htm > >> > > > > > >> > > > ---------------------------------------------------------------------- > >> > > > For information about J forums see > >> > > > http://www.jsoftware.com/forums.htm > >> > > > > >> > > ---------------------------------------------------------------------- > >> > > For information about J forums see http://www.jsoftware.com/forums.htm > >> > ---------------------------------------------------------------------- > >> > For information about J forums see http://www.jsoftware.com/forums.htm > >> > > >> ---------------------------------------------------------------------- > >> For information about J forums see http://www.jsoftware.com/forums.htm > >> > > ---------------------------------------------------------------------- > > For information about J forums see http://www.jsoftware.com/forums.htm > > > > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
