2009/3/26 Olaf Till <[email protected]>: > On Thu, Mar 26, 2009 at 03:22:59PM +0100, Jaroslav Hajek wrote: >> On Thu, Mar 26, 2009 at 3:00 PM, Jaroslav Hajek <[email protected]> wrote: >> > 2009/3/26 Olaf Till <[email protected]>: >> >> On Thu, Mar 26, 2009 at 09:21:38AM +0100, Olaf Till wrote: >> >>> >> >>> ... >> >>> >> >>> Looking at the parcellfun stuff, I noticed that there are still more >> >>> functions for sending and receiving Octave variables in octave-forge: >> >>> fsave and fload. Some time ago there was a discussion in the >> >>> octave-maintainers list on save'ing and load'ing over streams. After >> >>> getting some hints, I posted psend and prcv, which can install/read >> >>> Octave variables to/from memory as well as directly returning/taking >> >>> their contents, and which read/write a binary header also (to care for >> >>> endian'ness, as I understood), and which distinguish eof at start of >> >>> reading from eof later. The discussion dried out, probably the >> >>> solution was not general enough and not obviously the right one for >> >>> Octave. For the now contributed code, the functionality of fsave and >> >>> fload would probably be sufficient, if only endian'ness was cared for >> >>> by writing/reading a header (varibles are sent between different >> >>> machines now). If this would be done, I could rewrite my code, and >> >>> then there is probably no reason to keep psend and prcv in the >> >>> package. >> >> >> >> Jaroslav, >> >> >> >> would you accept the attached small patches for fsave.cc and fload.cc, >> >> to make them robust for usage between different machines? >> >> >> > >> > I actually didn't intend fsave and fload to work across machines >> > (parcellfun uses them just for pipes). But I'm not against it. I want, >> > however, to avoid the overhead of sending the binary header with each >> > variable. >> > >> > I see several options: >> > 1. use explicit arch specs (like fwrite, fread) >> > 2. use explicit option to control whether headers are used (but then >> > you can as well use different functions) >> > 3. cache the stream pointer in both fload/fsave and only write the >> > header on first save / attempt to read it on first load. This is >> > possible because octave_stream has reference counting, ensuring a >> > cached object remains valid. >> > The price to pay is one dangling closed file (but no memory leaks), >> > which is easily acceptable. >> > >> >> Umm, I just realized this is not really that easy if you >> simultaneously use more streams. So the question is still open. I >> don't see any reasonably simple way to achieve it, other than >> maintaining a map of open stream numbers which already received the >> header. >> Specifying architecture explicitly is another option - or just leaving >> everything in the present state. > > Well, I didn't want to make things more difficult. But I thought there > may be more people wanting to save/load to/from streams, and something > general for all would be good. The names "fsave" and "fload" sound > very general. But maybe this is not so easy and we should use > different functions for different purposes. >
Yes, I agree with the motivation, but the price seems too high for parcellfun. I really want to avoid putting the binary header in front of every variable, because, say, if parcellfun sends mere scalars, it will increase the traffic several times. fload is actually a hybrid between fread, which requires explicit specification of arch, and load, which detects it. Similarly for fsave. My best thought right now is to add fwriteheader and freadheader to communicate the architecture first (if needed), then use that in fload. It would be nice to allow that information be somehow cached with the stream number, but I don't see a good way unless we modify Octave. > select.cc: current code is attached. > OK > _exit.cc: also attached, but maybe you expect too much, its only a > trivial one-liner. Yeah that's expected, I was just lazy to write it myself. But I think the function should be named __exit__ for consistency with other internal functions, that are intended for hackers' use. > functions for advisory locking: I think I postpone rewriting these and > for the present include them in the package. Still think, apart from > using Octave streams, they should be more general to be a solution for > all. > > > The question is still open wether I should make scheduling remote > function execution a seperate package or combine it maybe with > "parallel". Your choice. I agree with Soren; it seems to me that the "parallel" package is more or less orphaned (according to SVN log the author didn't contribute anything for 3 years), so if your package supersedes, at least approximately, the functionality of parallel and you're willing to maintain it, that seems to me enough to just replace the contents of "parallel". > Actually I think that "parallel" would also be a suitable > place for parcellfun, since users will first look there for such > functionality. > I considered that option, but decided to go with general which I maintain, partly because "parallel" seems orphaned. Also, parcellfun is more special in the sense that it should work more or less as a drop-in replacement for cellfun, out of the box, no setup, no manual launching of processes or similar, which, as I understand it, is not true for the other packages. But of course its applicability is limited to multicore machines, and probably just Linux-like. regards -- RNDr. Jaroslav Hajek computing expert & GNU Octave developer Aeronautical Research and Test Institute (VZLU) Prague, Czech Republic url: www.highegg.matfyz.cz ------------------------------------------------------------------------------ _______________________________________________ Octave-dev mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/octave-dev
