On Wed, Oct 14, 2020 at 11:53 AM Oddo Da <oddodao...@gmail.com> wrote: > I did not use Spark or Scala as measures of greatness but they are evolution, > at least people are trying ;). Not all evolution is in the positive > direction, of course. But I do think that the world of software engineering > has moved/changed for better since 1990s. Yes, we built software just fine in > the 1990s and we built it fine in the 1960s but that is like saying we drove > cars just fine in the 1930s, why do we need new cars.
I don't see it that way. to me things like hadoop/Spark/etc were designed to solve a specific problem other paradigms couldn't (or rather shouldn't). it's not evolutionary, it's something new. your analogy of cars is nice, but i believe it's more reflective of MPI progressing from MPI1 -> MPI2 -> MPI3 than how the entire industry has moved. cars have gotten better since the 1930's, but they still have four wheels and a motor i'll agree that in some respects software engineering has gotten better in the last 20yrs, but it's subjective. there are a lot of things that have gotten better and there are a lot of things that are much worse. but i'm not sure you can apply that statement to HPC. HPC code doesn't churn like business code or even more volatile cloud code. HPC code is usually written to solve something specific and gets incremental updates over time. usually that something specific hasn't changed the last 20yrs (think physics/chemistry) the models we use to describe or solve the problems likely have, but the underlying code is probably basically the same with tweaks along the way to fit the new model. re-writing the code in a more modern language because the language allows you to use more natural way to describe the problem doesn't make it less complex, it just hoists the problem onto someone else (ie the code under the code). and you're throwing away 20yrs of coding knowledge and stability. technical debt is probably the biggest preventer in new language adoption. the ML/AI frameworks weren't born in the MPI world, but are adopting it because no one wants to or sees a reason to re-write that code. there's nothing that stops something like pytorch from hooking the low level RDMA libraries and sending messages around the network, but MPI/charm/upc/shmem already does it so why reinvent the wheel > I stated in the original post that I am coming back into the field and that > it feels like not much has changed since 20 years ago. I asked to be schooled > or corrected. I suppose it would be alright for you to tell me specifically > what has changed for the better (or has been a paradigm shift) and how widely > have these changes been adopted. For example, I consider things like chapel > evolutionary changes in the right direction even if it may not be adopted > widely - at least someone is trying ;). What you seem to be saying is that we > don't need anything new, C and MPI are all that is necessary and we are happy > with it? i'm not sure i'd go as far as to imply that. there are definitely things that could be fixed with C and MPI that many people are unhappy with. but there's already a huge investment in the platform as it sits and good change I believe must be done slowly. chapel is a good example that supports your statements and i would also consider this an evolution to MPI. but it goes back to technical debt. to re-write something in chapel is non-trivial and may not be worth the time. writing something new and choosing chapel is really left up to the developer. i have some chapel users here and there, but they're a minority. and since chapel is largely only found on cray machines its exposure is low i'm not sure the philosophical debate you're looking for is one that can take place. like vim vs emacs or init5 vs systemd. everything exists and it usually boils down to personal choice. i run a fairly large hpc center and "user written" C/MPI code really only represents <20% of my workload. but that's subjective. i'd bet if the beowulf list did a poll you'd find heavy slants based on user base. if you feel the industry hasn't moved, maybe thats just where you are working, what you're doing, or who you're working with rather than a representation of the hpc industry. i still think you're trying to compare two things that shouldn't be compared. MPI isn't a programming language, it's a library. if you want to debate programming language evolution, that's a totally separate discussion from one that includes MPI/Spark/Etc _______________________________________________ Beowulf mailing list, Beowulf@beowulf.org sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit https://beowulf.org/cgi-bin/mailman/listinfo/beowulf
