Well, the answer to this is pretty long... I'd really like to keep things short, but I'm afraid I can't explain, why I think that this project is relevant in a short summary.
I'm not naive enough to think that two people could pull off something big like this. But that's why you do things Open Source right? If the project is interesting enough, other people will ultimately join and help on the way. I just hope we can offer a cool prototype soon, so that it gets interesting enough for other people to join. But there are certainly things, that I feel more uncomfortable re-implementing. One of them is Github. Github + git are so advanced and well established, that it might just seem really, really stupid to re-implement it. But if you think about it, Github is just not up to the task. What we want to implement, is something very closely tailored to the language. And that is not just because I randomly like Julia. My ideas for this started at a point, where I was very frustrated with Open Source libraries. I have a long list about things, that I'm quite frustrated about: 1. It's always a big pain to find well documented libraries 2. It's very difficult to compare similar libraries in terms of performance and functionality in a broader scope 3. You do most of this research work with google, stack overflow, or github, which is not really specialized for this task 4. Mostly things don't work well together, because everyone re-implements basic types, or exchange formats are not available or just suck 5. it is very hard to extend libraries, as the structure is often quite rigid 6. you always need to decide between languages, sacrificing either performance or development speed 7. In the end, after you went through all these struggles, you mostly end up with missing dependencies and re-implementing a lot of things I must admit, I start feeling less strongly about a few of these points, as things get easier with experience. But that still means, that the bar is quite high for beginners. So I started thinking about the root of these problems. My first conclusion was, that most of the languages are not a good fit for Open Source development. Especially Object Oriented languages seem to make things hard. Introducing Classes and inheritance means introducing a lot of rigidity and complexity, which makes it hard to extend anything, without breaking other peoples code. Or things are so intertwined, that you just can't figure out where to start changing things. Well to keep things short, I concluded, that what one needs for a language are six things: 1. Speed! Nothing is a bigger turn off, than being constantly reminded, that the implementation would be 500x faster in another language 2. A very easy but concise type system, that just offers datatypes but isn't bound to any functionality 3. Functions, that are basically just symbols and the right implementation gets called via pattern matching 4. Good ways to easily define new syntax, to keep the language up to date 5. A very good low level support, to keep things up to date with new hardware 6. High-Level features - Obvious, If you don't want to scare everyone away I searched long for a language fulfilling this and finally settled for Julia. Multiple dispatch is not completely what I had in mind, but works pretty well and is probably a lot more performant than what I wanted to have. I wanted to have arrays and dicts, and than do deep pattern matching on these to find the right implementation. With this, a type would be just a pattern matching rule. (type Person MATCH [:name => String, :surname => String]) Well, having a language that fulfills these requirements, still doesn't settle the problems that I have with Open Source. But a performant high level language already does one big thing: It doesn't turn off high performance nerds, but also doesn't scare away beginners. This is a huge win for an Open Source user base. The other thing, that gets introduced by multiple dispatch is, that reusing and extending code becomes a lot easier. You don't need to inherit from any class, you don't need to hack into anything, you just simply create a new function and suddenly, you extended some package, without making things awkward. This doesn't work too well for types in Julia yet, as you can't really extend anything, without breaking the whole thing and/or producing redundant code. That is where a more advanced multiple dispatch would work a lot better. For example with what I had in mind, you could extend a type by just supplying a new pattern matching rule. The old functions would still work, as they can find all the old fields, and a new function needing the more specific type, would just require one more field. But besides these problems, I think Julia is a good fit for finally settling a lot of problems that many Open Source Libraries have today. Now, one just needs good system, for organizing the resources, produced by the Open Source community, to fully harvest Julia's advantages. I must admit, that github is quite a good fit for doing this and I'm always impressed by how sophisticated github already is. But I came to the conclusion, that I want something more tailored to multiple dispatch and going away from source files. I want to have all resources, not just the code, in one connected graph. And I want to do global queries like: - give me functions that have the arguments (::Image, ::Matrix) - search ALL the docs for: "xxxxx" - is there example code for this function? (or where is this function used) - etc... Also, I want to have things more atomic. It's not really practical to have functions in a file or a class. You always need to justify, where to put a function, and then you need to make sure, that it's available at another place. Sometimes, it is very difficult to justify, where to put a certain function. Just look at the fft function, or the voronoi function. They are functions, which are best described, by their signature plus a few tags, but belong to a lot of places. Putting them into a certain module/class/file just can't represent their many use cases. And suddenly, you need to include a completely unrelated module into your program, just because it's the only one having the right fft function. Also, if you treat functions more like entities on their own, I hope it will become a lot more common to make tests per function and not per module. This obviously is not trivial sometimes. Best example are the OpenGL functions, which rely on a previously generated context, or the effects of calling a gl function is very difficult to test. But I think that shouldn't stop us to pursue a more atomic treatment of libraries, as you gain one big advantage. It becomes very easy to re-implement a function. You just upload a newer version, and if it survives the tests for the function and is for example faster, voila, you can accept it as a newer version. Then you can discuss and rate the function. And this, without having to deal with any forks, merges, the rest of the module, etc... You don't really have a place for that in Github, as all the functionalities are defined for repositories. Same goes for bug reports. If functions are an entity themselves in the system, you can just file in bug reports for single functions, which keeps things more concentrated on the location, where the error really happens. Also, when all the resources are connected in a big graph, it becomes trivial to, for example, build a minimal version of Julia. You just query all the dependencies( which could be any type of data) for your function calls. This surely just works well, if Julia reduces C dependencies, or has them more atomic as well. I don't want to go into even greater detail here. So lets just assume that you have all this magic running. It then becomes obvious, that a good IDE is needed to utilize all these functionalities. I could do this with already existent IDE's, that's definitely true. But I think IDE's need to change a lot in the future. I'd just feel very uncomfortable with extending an existing IDE, because I would constantly think: Well, this needs to be completely redone soon. I'm just very frustrated, with how you browse code, packages, documentations, example codes, etc... And the biggest frustration is, why don't the IDE's make more use of the dynamic features of the languages? I want to write code like, for example, with Eclipse, and then run the program like with a REPL, having all the functions of the language available at runtime. Then I could just manipulate and display variables at runtime, like I'm used to. This would make debugging so much easier. Especially high level debugging, where you just don't get a mathematical equation right, without getting any error. This is basically the reason, why I want to build up an IDE with the functionalities from a native graphic library. So I imagine the IDE more as a bundle of functions for displaying and manipulating the Julia data types together with a sophisticated search interface, which itself just returns Julia data types (which can be also binary streams or whatnot). Just look at all the beginner questions, that don't get how to create a multi-dimensional array in Julia. With a good interactive view on an Array, this can become much easier. Well, I think I shouldn't make this email much longer, so lets come to a conclusion: I'm not very afraid of re-implementing things, as I think there a lot of thinks that are in need of improvement. I'm much more afraid, that we don't have enough experience to judge things correctly, or lay down the needed ground-work. That's why we are here in the mailing list, to get feedback from more experienced developers. Also the thing that is most redundant to what I have in mind is actually the new Wolfram Language, which tackles basically most of the things I have mentioned. When it got announced I actually thought about just dropping all my plans. But it seems, that the license is quite restrictive, things rely on their cloud, and the core isn't really open for developers. So I hope, that our efforts are still relevant! Best, Simon
