https://stackoverflow.com/questions/40015439/why-does-map-return-a-map-object-instead-of-a-list-in-python-3
map apparently has avoided being obsoleted by generators due to it's ability to accept multiple iterators: >>> list(map(min, [1,2,3,4], [0,10,0,10])) [0,2,0,4] The extra step is admittedly a small syntactic annoyance. I note that if python were to implicitly convert the map object to a list wherever there would otherwise be an error, then the user no longer cares (or knows) that this map object existed. (This would possibly be wasteful in python, where so much less happens per statement). For J, where so much more is happening per statement, we could transparently replace all intermediate results by generators, except where a full noun is required. To translate the python statement to j, one would write (<./ 1 2 3 4 , 0 10 0 10), however we lose lots of time to python here due to calculating the intermediate catenation (unless we are fortunate enough that the interpreter has a Special Combination (SC) for this) However the SC system is generally a suspicious concept, in all cases where it doesn't apply a completely different algorithm, (eg. if +/i.>:y were to calculate -:(]*<:)y). I suspect we can do much better by using generators internally (so no change whatsoever to J lang): These generators are always converted to arrays when needed, and 100% invisible to the user!). Then cases like (x>y)i.1 (which the SC 'x i.&1@:> y' is unable to recognize), can nevertheless be executed optimally, and we additionally solve the 'f.' problem in a clean way. For everyone who may be of the opinion that this is not worth it, I would like to direct your attention to the existing jsources, where vast amounts of effort for far smaller optimizations are apparent. for example p.c, where the parser itself tries to emplace arrays and avoid unnecessary copying. It is still unable to do this consistently however, as simple memory tests like 7!:2 'i.1000x' ,: '>: i.1000x' prove. (and yes I like using x for my benchmarks due to how much slower they are) another example is. jt.h, where a JFE's state information structure has been lovingly crafted to optimize it's use of cpu cache or again the countless optimizations in the primitives, like n&i. hash tables, the immensely flexible /: etc.. I believe the worst offending performance sinkholes are first of all extended precision calculations, and secondly the often avoidable eager evaluation. With that being said, you are absolutely right, the issue is the effort necessary to implement all of this. Concretely, we would need a 'generator' C datatype (Note sparse arrays are a special case) To be honest, at this stage I feel that I will just implement it myself, as soon as I have the time to take a closer look at the sources. My immediate plan (perhaps a naive one), is to simply redefine the AR macro to force a physical array given a generator. In this way everything continues to work as usual as I experiment with incrementally hooking up primitives to the generator as a situational improvement on traversing a physical array. I wish however to make sure this is the best approach and additionally examine the problem of integrating sparse arrays into this more general generator. An additional issue to clear up is how this applies to rank n>1 arrays. I can only hope that the added complexity is not utimately deemed too great, although of course since this may obsolete many SC's and greatly speed up verb trains, as well as by corollary making i._ (and at least linear combinations) immediately possible, I believe it should be well worth it., J4 ________________________________ From: Source <source-boun...@forums.jsoftware.com> on behalf of Alex Shroyer <ashro...@gmail.com> Sent: Wednesday, February 28, 2018 2:38:49 PM To: sou...@jsoftware.com Subject: Re: [Jsource] Propositions One case I was thinking of where laziness makes things more complicated in python is 'map'. In (eager) J, I can say 2 * 3 4 5 and get back 6 8 10. But in python, map([3,4,5], lambda x:2*x) returns an iterator object (I think). To eagerly evaluate and return the data, I need to wrap the whole thing in a call to 'list()'. That extra step forces a particular style of composition (can't seamlessly mix and match eager and lazy, you have to wrap and unwrap) and is more complicated than either lazy-only or eager-only. That said, the"idiom recognition" of J has a similar effect of reducing extra work at run-time. Although it seems to require more effort from J implementors. -----Original Message----- From: "Raul Miller" <rauldmil...@gmail.com> Sent: 2/28/2018 8:09 AM To: "Source forum" <sou...@jsoftware.com> Subject: Re: [Jsource] Propositions Stalled processing: i.1e7 takes a finite amount of memory - either you have it, or you do not. But lazy evaluations encourage you to work with expressions which are more like i._ and if you are tackling non-trivial problems it becomes trivial set the machine to work on problems that do not fit in memory. These can still be solved, but only by shifting your resource tradeoffs to favor less memory at the expense of increased computational time. But since the design of the system hides those choices from the programmer it can become difficult to talk about. Yak shaving: http://sethgodin.typepad.com/seths_blog/2005/03/dont_shave_that.html - this is basically a term for high investment in low priority efforts and low investment in high priority efforts. You have to have some tolerance for such things, because prioritization is based on estimation and guess work. It's ok to do things for fun, but take this too far and you get solutions looking for [irrelevant] problems and [critical] problems going without solutions. (A rule of thumb might be the 10% threshold. If you are not spending at least 10% of your efforts on things which have some generic utility, you should strongly consider rethinking what you are doing.) That said: if you want to build a version of the J interpreter which implements range handling like you've been suggesting, no one is stopping you. I personally am dubious of the value of that effort, but I can see the attraction of the effort, and if you write such a thing and need help debugging it, I might be inclined to help out at that point (this would be a low priority task for me, but I do spend time on low priority tasks). Finally: infinities are mostly impressive for people who do not try to use them, and instead are a great source of errors and slow systems for people who do make the attempt. As for the prefix of an infinite list thing - the attraction of working with a system which lets you say i.1e7 but not actually do that work is that later on you can go n{.1e7 and get the effect of i.n. But you can achieve the same thing more directly by performing i.n directly. There's a subtle complication here that you might want to use a value of n which is greater than 1e7 and that that result would be different from i.n but if that was the priority it might be that (* 1e7&>) is a better way of describing the useful part of the system. By focusing on creating a system which represents a subset of useful functions as ranges you've also built a system which does not support other functions. But what is the point of that? -- Raul On Tue, Feb 27, 2018 at 10:17 PM, james faure <james.fa...@epitech.eu> wrote: > 'Potential problems though include stalled processing and yak shaving' > > You need to elaborate on this, I don't see what this has to do with anything. > > 'but I do not draw the conclusion that this means that working with prefixes > of descriptions of infinite arrays is a particularly good idea' > > and this as well, after all, a prefix of an infinite list is no longer > infinite > > > Additionally, infinite arrays are of course not helpful for performing > calculations on data sets, they are mainly a theoretical exercise, although I > am of the opinion that they would be very helpful for solving problems > involving searches on infinite sequences, since 'while.' and '^:', are slow, > and having to guess an upperbound (here is the arbitrary prefix of an > infinite list) is just ugly. The physical impossiblity of an infinite array > has never been an issue for mathematics, and i.&1@:whatever@:i._ would be the > idiomatic solution. > > > Finally, a non negligible aspect of such infinite lists are how impressive > they are. J must impress newcomers or they will not be willing to spend the > considerable effort required to even get anything done in the first place. I > have no idea how many people choose to learn J nowadays, but I suspect > nowhere near enough. > > ________________________________ > From: Source <source-boun...@forums.jsoftware.com> on behalf of Raul Miller > <rauldmil...@gmail.com> > Sent: Wednesday, February 28, 2018 3:44:04 AM > To: Source forum > Subject: Re: [Jsource] Propositions > > On Tue, Feb 27, 2018 at 9:15 PM, james faure <james.fa...@epitech.eu> wrote: >> Funny that you would say that, since It seems to me that the more >> lightweight the description of >> an array is, the easier it is to operate on the whole thing at once. > > Potential problems though include stalled processing and yak shaving. > Both of these have to do with hitting the problem at the wrong level > of abstraction. > > In my experience, working with J on large data sets, it's extremely > useful to run through the concepts on very small sets of data, first, > so that you can verify that things are working right and that you > understand the data. Working with very large sets of data can be quite > a burden (finding a conceptual error a month into the calculations is > quite different from finding an error an hour in or a few seconds in - > at that point you need to think things through and decide what you can > salvage). > > This agrees with the abstraction you brought up that "the more > lightweight the description of an array is, the easier it is to > operate on the whole thing at once" but I do not draw the conclusion > that this means that working with prefixes of descriptions of infinite > arrays is a particularly good idea. > > Thanks, > > -- > Raul > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm > ---------------------------------------------------------------------- > For information about J forums see http://www.jsoftware.com/forums.htm ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm ---------------------------------------------------------------------- For information about J forums see http://www.jsoftware.com/forums.htm
