Continuations in our language are expressed very differently than has existed before (e.g. like in Scheme). They fit intuitively within the overall paradigm our language implements. Far from being complex or hard to comprehend, continuations are the key construct that enables us to avoid mountains of complexity that exist otherwise in distributed systems. I know this from personal experience building distributed systems in the past. The degree to which continuations help write asynchronous, reactive, and parallel code is huge. It would be clear if you saw the language in action, but we're keeping it under wraps for now.
We use deps.edn just for dependency resolution via lein-tools-deps.plugin/resolve-dependencies-with-deps-edn. We use lein as a test runner and for launching repls from the command line. Our build's not too complicated, but it's not ideal. That's right. Since tests run sequentially, using with-redefs is safe. We parallelize our build by running it as multiple Jenkins stages, where each stage runs a subset of the tests sequentially in separate processes. Any with-redefs we do are at the beginning of tests and thus global to its execution. with-redefs + no-op functions is extremely lightweight. We do use component overrides for other things, but that requires: a) the component system map is accessible where you want to capture events, b) non-trivial overhead in adding a new component and threading it to the spot in the code you care about. For events we want to capture in low-level implementation functions the with-redefs + no-op functions approach is simple, easy, doesn't affect production, and avoids bloating our component system maps with tons of things which are irrelevant to production. The only testing method we use internally that might be relevant to users of our product is deterministic simulation. We're unlikely to expose that for v1, but we may expose it in the future. For v1 we'll be exposing an "in process cluster" abstraction for testing which runs things in parallel on multiple threads. On Wednesday, June 9, 2021 at 10:57:40 AM UTC-10 Leif wrote: > Hi Nathan. Interesting post. Here are some questions that I wrote down > while > reading it. > > First-class continuations + distributed programming sounds like a > nightmare that > would wake me up in a cold sweat. Like if a void pointer was bitten by a > radioactive goto. Does your language have restrictions that make such a > pairing > comprehensible to average programmers like myself? I assume so, so my real > question is: what distributed programming languages / paradigms is your > language > inspired by? > > The usage of leiningen *and* deps.edn is interesting. How are they > combined? > Do you handle all deps with deps.edn, and use lein as a task runner, or...? > (After reading the post comments, it seems somewhat more complicated. > Complicated build processes: I am shocked. Shocked, I tell you.) > > Using with-redefs + concurrency has bitten me before. Is it not a problem > because your tests are high-level and run sequentially, so the per-test > redefs > don't interfere with each other? (I think this is mostly answered by the > "Deterministic simulation" section, but answer if you'd like.) > > The "no-op functions as a structured event log" pattern is interesting. > Did you > try out any other methods, like implementing the event logging as separate > components, or directly redef'ing the side-effecting code? If so, what > are the > pros & cons you considered? > > Are these testing methods just for internal use, or are you planning to > make > them easy to use by the end-users of your language? I have a lot of > questions > about your language, actually, but I'll just wait for your demo. > > It *is* an ambitious project, and I hope it succeeds. (Since most > software > nowadays becomes part of "distributed systems" and they are, frankly, a > PITA.) > > Cheers, > Leif > > > On Friday, June 4, 2021 at 7:53:43 PM UTC-6 brent....@gmail.com wrote: > >> Haven't read the article yet but thanks for the work, will definitely >> give it a read asap. After hearing (already a while ago) that Storm moved >> to Java, I'm happy to hear Clojure is still actively used (and even >> instrumental) in your current work. >> >> On Thursday, June 3, 2021 at 11:42:36 PM UTC-4 natha...@gmail.com wrote: >> >>> Probably not in the near future. Once we're close to being ready for >>> production usage, we'll be releasing a preview post demonstrating how our >>> product works through an example of using it to replicate a widely-used >>> service in a comparatively minuscule amount of code. Though we're already >>> able to build and run arbitrary applications at scale, we still have a lot >>> of work to do on operational features to enable the fault-tolerance >>> necessary for production usage. >>> >>> On Thursday, June 3, 2021 at 5:24:00 PM UTC-10 Robert P. Levy wrote: >>> >>>> Great post on the technical choices made in developing this platform. >>>> Do you plan on writing a post that describes in detail the system >>>> architecture that is the bread and butter of the product/platform itself? >>>> >>>> The website intriguingly states, "Red Planet Labs is pioneering a >>>> radically new kind of software tool. It's not just for the initial >>>> construction of an application, but also encapsulates deployment, >>>> monitoring, and maintenance. It implements the first truly cohesive model >>>> for building software applications – a set of abstractions that can build >>>> any application at any scale with greatly reduced engineering cost." >>>> >>>> It seems like a full article expanding on the infrastructure-level >>>> design, and the approach to generalizing and abstracting infrastructure, >>>> would be very interesting. >>>> >>>> >>>> On Thu, Jun 3, 2021 at 7:12 PM Nathan Marz <natha...@gmail.com> wrote: >>>> >>>>> Derek – we have a bunch of open-source on our Github >>>>> <https://github.com/redplanetlabs>. I'd like to release our new >>>>> language one day, but that won't be for a long time. When I release it I >>>>> want to do it the right way – extensive documentation, academic papers, >>>>> and >>>>> a commitment to supporting the language in broader usage. At our early >>>>> stage we just don't have the bandwidth for that as we're working hard to >>>>> get our first product out the door. Plus at the moment out language is >>>>> our >>>>> "secret weapon" :) >>>>> >>>>> Leandro – I started working on this codebase well before spec was >>>>> released. Had spec existed when I started I would have explored it more >>>>> thoroughly, but Schema has met our needs very gracefully. As for >>>>> deterministic simulation, it's orthogonal to techniques like test.check. >>>>> I >>>>> suggest you check our our earlier blog post >>>>> <https://tech.redplanetlabs.com/2021/03/17/where-were-going-we-dont-need-threads-simulating-distributed-systems/> >>>>> >>>>> on the subject. >>>>> >>>>> On Thursday, June 3, 2021 at 3:09:28 PM UTC-10 ldoc...@gmail.com >>>>> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Thu, 3 Jun 2021, 15:06 natha...@gmail.com, <natha...@gmail.com> >>>>>> wrote: >>>>>> >>>>>> Please give the post a read, and I'm happy to answer any questions. >>>>>>> >>>>>> >>>>>> Nice article, Nathan. Definitely, it would be nice to see the code :-) >>>>>> >>>>>> Just curious... >>>>>> >>>>>> 1) You mention using Schema for data definition and validation. Did >>>>>> you consider using other options for this, such as clojure.spec? What's >>>>>> your experience with it/them? >>>>>> >>>>>> 2) You mention using "deterministic simulation". Did you consider >>>>>> using other options for this, such as test.check? What's your experience >>>>>> with it/them? >>>>>> >>>>>> Best, >>>>>> Leandro >>>>>> >>>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Clojure" group. >>>>> To post to this group, send email to clo...@googlegroups.com >>>>> Note that posts from new members are moderated - please be patient >>>>> with your first post. >>>>> To unsubscribe from this group, send email to >>>>> clojure+u...@googlegroups.com >>>>> For more options, visit this group at >>>>> http://groups.google.com/group/clojure?hl=en >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Clojure" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to clojure+u...@googlegroups.com. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/clojure/70353370-6dd2-4ddc-af82-2fce32ee6bden%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/clojure/70353370-6dd2-4ddc-af82-2fce32ee6bden%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/clojure/75b98a55-39cd-4b5c-8f17-4b0c0390b655n%40googlegroups.com.
