[julia-users] Re: Too many packages?
For your interest: of your core packages: numpy, scipy, sympy, matplotlib, and f2py numpy functionlity should be all in Base. Sympy has a julia version https://github.com/jverzani/SymPy.jl, wrapping pyCall, and it is great. Similarly so does matplotlib https://github.com/stevengj/PyPlot.jl, though I've not used it, personally it was a relief to get a way from matplot lib and onto something less explicit and more intutive like GadFly https://github.com/dcjones/Gadfly.jl I've not used f2py, but I suspect core ccall http://julia.readthedocs.org/en/latest/manual/calling-c-and-fortran-code/will do it. Scipy is huge and vast, and I constantly forgetting what is in there. The most useful parts of Scipy, tend to be in the Managed Julia Github Groups, or including the Core Group https://github.com/JuliaLang The managed groups (self managed that is) can be found on the communities page http://julialang.org/community/ Cutting it down to just the things that I believe are in Scipy: (Some overlap with sklearn) - JuliaStats https://github.com/JuliaStats – Statistics http://www.juliastats.org/ - JuliaOpt https://github.com/JuliaOpt – Optimization http://www.juliaopt.org/ - BioJulia https://github.com/BioJulia - Biology - JuliaAstro https://github.com/JuliaAstro – Astronomy - JuliaQuantum https://github.com/JuliaQuantum – Julia libraries for quantum science and technology http://juliaquantum.github.io/ - JuliaSparse https://github.com/JuliaSparse – Sparse matrix solvers - JuliaDiff https://github.com/JuliaDiff/ – Differentiation tools http://www.juliadiff.org/ - JuliaDSP https://github.com/JuliaDSP – Digital signal processing - JuliaGraphs https://github.com/JuliaGraphs – Graph Theory and Implementation As Matt Bauman points out, the julia devs / major contributors tend to be major players in one or more communities. Packages within the communities tend to be of higher confidence in there reliability. More certain to be maintained. You can normally trust a package from one of the communities. So they are always my first stop when looking for a package to do something. My second stop right now tends to by using PyCall (for sklearn and NLTK). Hope that helps.
Re: [julia-users] Re: Too many packages?
I'm a newcomer as OP. And the most problematic to me is not a huge number of packages but the need to manually evaluate numerous alternatives while searching for desired functionality. Some kind of quality or popularity rating would simplify things tremendously. Or maybe some standard criteria for production-ready packages could be established and packages meet it should be clearly marked as such. There are communities developing thousands of interrelated packages (like drupal.org) which manage to tackle the issue using variation of techniques I mentioned. --Dmitry On 12.07.2015 04:09, John Myles White wrote: I think most of have the opposite desire: we're trying to move more functionality out of the core language and into packages. -- John
Re: [julia-users] Re: Too many packages?
When evaluating a library on Github or similar, I usually look at the following: 1. are there many open issues? How old are they? 2. are there recent commits? 3. is there a reasonable README file with examples etc? 4. are there unit tests (when applicable?) While these are imperfect proxies for the quality/maturity of the library, they can be used to form a reasonable first impression before investing more time. Best, Tamas On Tue, Jul 14 2015, Dmitry Vyal dmitryv...@gmail.com wrote: I'm a newcomer as OP. And the most problematic to me is not a huge number of packages but the need to manually evaluate numerous alternatives while searching for desired functionality. Some kind of quality or popularity rating would simplify things tremendously. Or maybe some standard criteria for production-ready packages could be established and packages meet it should be clearly marked as such. There are communities developing thousands of interrelated packages (like drupal.org) which manage to tackle the issue using variation of techniques I mentioned. --Dmitry On 12.07.2015 04:09, John Myles White wrote: I think most of have the opposite desire: we're trying to move more functionality out of the core language and into packages. -- John
[julia-users] Re: Too many packages?
On Sunday, 12 July 2015 22:47:42 UTC+2, Tony Kelman wrote: I think there's a big differences between developing core features in packages and shipping them with the default version and having optional third party packages implementing core features. Like what, exactly? If the complaint is about ease of installation of packages, then that's a known and acknowledged bug (set of bugs) that people are thinking about how to do a better job of. We could always use more help making things better. If there's a bunch of official packages that are shipped with default version it's like having no packages, it's just a way for the devs to organize their work internally that doesn't concern the user too much. On the other hand for third party packages the user has to find them, install them, debug them, worry about long term maintenance, etc. In reality it's a bit more fuzzy than that, so maybe my distinction isn't so relevant. For plotting I think it would be better to have any plotting than none, even though not everybody will agree on the best choice for the one. The least dependencies seems the most important criteria to me, as long as you can draw lines, points and surfaces with decent performances. The high level interface doesn't matter that much in my opinion.
[julia-users] Re: Too many packages?
Packages: I fall into the camp of tiny base, curated packages. It would be great to have an absolutely minimal core julia, and then lots of build recipes for those people that want a matlab-like experience. For example, JuliaStats could be responsible for compiling a best of breed list of key stats and plotting packages for the stats recipe, etc. Then on installation you can say what recipes you'd like to include (maybe it defaults to most/all of the major recipes so people don't even need to think about it?). Plotting: I agree that simple plotting with minimal dependencies is the way to go for a standard package. I don't think Gadfly fits that very well in it's current form... there needs to be simple ways to plot and there should not be a strong (or any?) dependency on DataFrames. I'm certainly not recommending that this become standard, but look at the readme of https://github.com/tbreloff/Qwt.jl as what I think the basic plotting interface should look like. On Monday, July 13, 2015 at 5:12:53 AM UTC-4, yu...@altern.org wrote: On Sunday, 12 July 2015 22:47:42 UTC+2, Tony Kelman wrote: I think there's a big differences between developing core features in packages and shipping them with the default version and having optional third party packages implementing core features. Like what, exactly? If the complaint is about ease of installation of packages, then that's a known and acknowledged bug (set of bugs) that people are thinking about how to do a better job of. We could always use more help making things better. If there's a bunch of official packages that are shipped with default version it's like having no packages, it's just a way for the devs to organize their work internally that doesn't concern the user too much. On the other hand for third party packages the user has to find them, install them, debug them, worry about long term maintenance, etc. In reality it's a bit more fuzzy than that, so maybe my distinction isn't so relevant. For plotting I think it would be better to have any plotting than none, even though not everybody will agree on the best choice for the one. The least dependencies seems the most important criteria to me, as long as you can draw lines, points and surfaces with decent performances. The high level interface doesn't matter that much in my opinion.
Re: [julia-users] Re: Too many packages?
Note there's also an open issue for requiring a higher overall standard for officially registered packages in the JuliaLang/METADATA.jl package repository. It's a big issue with a lot of work required to get to the proposal, but it would lead to (hopefully) instilling more confidence in users knowing that anything they add through `Pkg.add()` would meet some acceptable level of quality and robustness. -Jacob On Mon, Jul 13, 2015 at 11:11 AM, Christoph Ortner christophortn...@gmail.com wrote: I seem to be in the minority too many packages camp. I would prefer stable updates of julia version which means that key functionality should be included in core, e.g. BLAS, sparse solvers, eig, eigs, basic plotting and so on and so forth. But at some point there was an idea of having core and Stdlib, which I think is equally acceptable. Christoph
[julia-users] Re: Too many packages?
I seem to be in the minority too many packages camp. I would prefer stable updates of julia version which means that key functionality should be included in core, e.g. BLAS, sparse solvers, eig, eigs, basic plotting and so on and so forth. But at some point there was an idea of having core and Stdlib, which I think is equally acceptable. Christoph
Re: [julia-users] Re: Too many packages?
I echo Tom Breloff's sentiment that Pkg.add() should simply be expanded to allow various repos of repos in a kwarg argument. This is an open issue here https://github.com/JuliaLang/julia/issues/11914. The default repo of repos would be METADATA or maybe it's possible new name of CuratedPackages, and third parties would be welcome to add their own repo of repos which I'm sure could be coded into the method (i.e., JewelBox would map to https://github.com/JuliaJulia/JewelBox.jl) or alternatively the url can be provided as a string. On Monday, July 13, 2015 at 2:57:14 PM UTC-4, Jacob Quinn wrote: Note there's also an open issue for requiring a higher overall standard for officially registered packages in the JuliaLang/METADATA.jl package repository. It's a big issue with a lot of work required to get to the proposal, but it would lead to (hopefully) instilling more confidence in users knowing that anything they add through `Pkg.add()` would meet some acceptable level of quality and robustness. -Jacob On Mon, Jul 13, 2015 at 11:11 AM, Christoph Ortner christop...@gmail.com javascript: wrote: I seem to be in the minority too many packages camp. I would prefer stable updates of julia version which means that key functionality should be included in core, e.g. BLAS, sparse solvers, eig, eigs, basic plotting and so on and so forth. But at some point there was an idea of having core and Stdlib, which I think is equally acceptable. Christoph
[julia-users] Re: Too many packages?
I think there's a big differences between developing core features in packages and shipping them with the default version and having optional third party packages implementing core features. Like what, exactly? If the complaint is about ease of installation of packages, then that's a known and acknowledged bug (set of bugs) that people are thinking about how to do a better job of. We could always use more help making things better. The only thing I really would like to have included by default it plotting tools, because they are so essential for a lot of things. I don't think you're going to find a single plotting tool that satisfies everyone, unfortunately. Not everyone likes grammar-of-graphics style plotting, or dependencies on Tk or IPython/Jupyter or OpenGL or a web browser to have a usable plotting package. Once we figure out the right way to bundle packages and make larger distribution versions of Julia we can include a few different choices.
[julia-users] Re: Too many packages?
We can probably pick a default, and it'll probably be Gadfly, but last I checked Gadfly's support for running outside of IJulia is still a bit lacking. To distribute a working version of IJulia you have to get into the business of redistributing an entire Python installation, which could be quite a rabbit hole. On Sunday, July 12, 2015 at 1:59:21 PM UTC-7, Cedric St-Jean wrote: On Sunday, July 12, 2015 at 4:47:42 PM UTC-4, Tony Kelman wrote: The only thing I really would like to have included by default it plotting tools, because they are so essential for a lot of things. I don't think you're going to find a single plotting tool that satisfies everyone, unfortunately. Not everyone likes grammar-of-graphics style plotting, or dependencies on Tk or IPython/Jupyter or OpenGL or a web browser to have a usable plotting package. Once we figure out the right way to bundle packages and make larger distribution versions of Julia we can include a few different choices. I'm a Julia beginner who is still using pyplot to avoid making a decision. Choice is overwhelming when starting out with a new language, having a basic distribution that includes sensible defaults (eg. Enthought's scipy distribution) will help a lot.
[julia-users] Re: Too many packages?
On Sunday, July 12, 2015 at 4:47:42 PM UTC-4, Tony Kelman wrote: The only thing I really would like to have included by default it plotting tools, because they are so essential for a lot of things. I don't think you're going to find a single plotting tool that satisfies everyone, unfortunately. Not everyone likes grammar-of-graphics style plotting, or dependencies on Tk or IPython/Jupyter or OpenGL or a web browser to have a usable plotting package. Once we figure out the right way to bundle packages and make larger distribution versions of Julia we can include a few different choices. I'm a Julia beginner who is still using pyplot to avoid making a decision. Choice is overwhelming when starting out with a new language, having a basic distribution that includes sensible defaults (eg. Enthought's scipy distribution) will help a lot.
Re: [julia-users] Re: Too many packages?
In terms of dependencies and size, Gaston is probably minimal; it depends on gnuplot only, which is a small binary and readily distributed on Linux, OS X and windows. It offers basic features only (but has 3-D plotting), but this may be an advantage for a default package. It is also well documented (at least, I like to think so :) -- mb On Sun, Jul 12, 2015 at 5:17 PM, Tony Kelman t...@kelman.net wrote: We can probably pick a default, and it'll probably be Gadfly, but last I checked Gadfly's support for running outside of IJulia is still a bit lacking. To distribute a working version of IJulia you have to get into the business of redistributing an entire Python installation, which could be quite a rabbit hole. On Sunday, July 12, 2015 at 1:59:21 PM UTC-7, Cedric St-Jean wrote: On Sunday, July 12, 2015 at 4:47:42 PM UTC-4, Tony Kelman wrote: The only thing I really would like to have included by default it plotting tools, because they are so essential for a lot of things. I don't think you're going to find a single plotting tool that satisfies everyone, unfortunately. Not everyone likes grammar-of-graphics style plotting, or dependencies on Tk or IPython/Jupyter or OpenGL or a web browser to have a usable plotting package. Once we figure out the right way to bundle packages and make larger distribution versions of Julia we can include a few different choices. I'm a Julia beginner who is still using pyplot to avoid making a decision. Choice is overwhelming when starting out with a new language, having a basic distribution that includes sensible defaults (eg. Enthought's scipy distribution) will help a lot.
[julia-users] Re: Too many packages?
As John and Matt said, a huge portion of the standard library is written in Julia itself, so there's no real technical need for it to be developed within the same repository. In fact developing technical features as packages rather than as part of base allows getting features to users in a stable way on a time scale that the package developer has control over, rather than being subject to the readiness of features in the core language and compiler that obviously have to be done in the base repository. I can also find some Fortran/C code, and include in Julia, and have all these functionality, but then what is the advantage of using Julia, as opposed to, say, python? Using Julia, your wrapper glue code will be shorter, simpler to understand, more efficient, entirely in the high-level language, and yet map more directly to the underlying library's interface. You just have to be able to point to a standard C or Fortran shared library and you can use that API directly, even interactively from the REPL. No need to invoke a separate compiler or build system at install time just for the language-specific binding code. Depending on the library, there will usually be less marshalling and worrying about cross-language data structure copying. But since Julia is a language specifically for scientific computation That's not quite fair. Julia is a general-purpose language that happens to be designed to be very good at scientific computing tasks. (And most, but not all, of the early-adopter user and package developer community have come from that domain.) There are pieces included by default in the standard library right now that you would normally find in packages in the likes of NumPy or SciPy in other languages, but this may not be the case forever as the technical problems that have stood in the way of decoupling that development are gradually being solved.
[julia-users] Re: Too many packages?
I think there's a big differences between developing core features in packages and shipping them with the default version and having optional third party packages implementing core features. Personally I also find the huge amount of packages to be slightly annoying, but it's also clear the Julia team cannot do everything like Matlab would. The only thing I really would like to have included by default it plotting tools, because they are so essential for a lot of things.
[julia-users] Re: Too many packages?
It is worth differentiating what core Julia - the language and its standard library - includes as a default, and what different distributions of Julia may include to provide a good user experience. I personally have been wanting to make a distribution that includes a few key packages that I like and plotting for a while! I think with 0.4, we will be able to start doing this. -viral On Sunday, July 12, 2015 at 7:43:28 PM UTC+5:30, yu...@altern.org wrote: I think there's a big differences between developing core features in packages and shipping them with the default version and having optional third party packages implementing core features. Personally I also find the huge amount of packages to be slightly annoying, but it's also clear the Julia team cannot do everything like Matlab would. The only thing I really would like to have included by default it plotting tools, because they are so essential for a lot of things.
Re: [julia-users] Re: Too many packages?
As someone who remembers the days of the first packages, I also want to throw out the fact that the title of this issue should probably be viewed as a triumph :-). --Tim On Sunday, July 12, 2015 10:59:10 AM Viral Shah wrote: It is worth differentiating what core Julia - the language and its standard library - includes as a default, and what different distributions of Julia may include to provide a good user experience. I personally have been wanting to make a distribution that includes a few key packages that I like and plotting for a while! I think with 0.4, we will be able to start doing this. -viral On Sunday, July 12, 2015 at 7:43:28 PM UTC+5:30, yu...@altern.org wrote: I think there's a big differences between developing core features in packages and shipping them with the default version and having optional third party packages implementing core features. Personally I also find the huge amount of packages to be slightly annoying, but it's also clear the Julia team cannot do everything like Matlab would. The only thing I really would like to have included by default it plotting tools, because they are so essential for a lot of things.
[julia-users] Re: Too many packages?
Thank you! I apparently just made a wrong assumption that Julia was a language for scientific computing only. Once I think of it as a general purpose language, the current structure makes total sense, just as it does for python. On Sunday, July 12, 2015 at 4:47:04 AM UTC-4, Tony Kelman wrote: As John and Matt said, a huge portion of the standard library is written in Julia itself, so there's no real technical need for it to be developed within the same repository. In fact developing technical features as packages rather than as part of base allows getting features to users in a stable way on a time scale that the package developer has control over, rather than being subject to the readiness of features in the core language and compiler that obviously have to be done in the base repository. I can also find some Fortran/C code, and include in Julia, and have all these functionality, but then what is the advantage of using Julia, as opposed to, say, python? Using Julia, your wrapper glue code will be shorter, simpler to understand, more efficient, entirely in the high-level language, and yet map more directly to the underlying library's interface. You just have to be able to point to a standard C or Fortran shared library and you can use that API directly, even interactively from the REPL. No need to invoke a separate compiler or build system at install time just for the language-specific binding code. Depending on the library, there will usually be less marshalling and worrying about cross-language data structure copying. But since Julia is a language specifically for scientific computation That's not quite fair. Julia is a general-purpose language that happens to be designed to be very good at scientific computing tasks. (And most, but not all, of the early-adopter user and package developer community have come from that domain.) There are pieces included by default in the standard library right now that you would normally find in packages in the likes of NumPy or SciPy in other languages, but this may not be the case forever as the technical problems that have stood in the way of decoupling that development are gradually being solved.
[julia-users] Re: Too many packages?
I think most of have the opposite desire: we're trying to move more functionality out of the core language and into packages. -- John On Sunday, July 12, 2015 at 3:03:31 AM UTC+2, Burak Budanur wrote: I heard a lot about Julia language over the last year and last week had a conversation with a colleague, who attended Juliacon and was quite impressed. We talked about possibly moving some of our fluid dynamics projects to Julia, so that for a new student who is joining the project it would be much easier to start without going through learning c++ and/or fortran. I am a physicist and most of my day job is some form of scientific computing. My current default working environment is python (numpy, scipy, sympy, matplotlib) + fortran (f2py) when some part of my code needs to speed up. Yesterday I decided to start a a new, relatively easy project as a simple example for an upcoming paper. So I thought this might be a good occasion to start learning Julia language to code a simple dynamical systems toolbox in it, which might be useful for other people as well. Basic functionality I need from the language are these: - Symbolic differentiation (for computation of Jacobians) - Numerical integration of ODEs (a general purpose integrator, such as lsoda from odepack, wrapped in scipy.integrate.odeint) - Linear algebra functions - Interpolation - Plotting in 2D and 3D After reading The Julia Express and parts of the documentation, I thought that such a project is not a good investment, at least for now. The reason is all the functionality I listed above are provided by external packages, partially excluding linear algebra functions. I'm aware that I can use specific packages for all the functionality I mentioned above, but each such package is maintained by different people, and they can change or become obsolete. I can also find some Fortran/C code, and include in Julia, and have all these functionality, but then what is the advantage of using Julia, as opposed to, say, python? In a more general sense, I am a little bit turned off by the presence of an external package for almost every task I need to do. I can understand this kind of structure in python as it is a general purpose language. But since Julia is a language specifically for scientific computation, I'd be happy to have something like the basic functionality of MATLAB in the main language. I understand that Julia is under development and there is a lot to change and to be added, but I am wondering what is the Julia's future directions regarding these issues? I did some search, but could not find an answer to this question, so I apologize if this was already answered elsewhere. I heard a lot about Julia language over the last year and last week had a conversation with a colleague, who attended Juliacon and was quite impressed. We talked about possibly moving some of our fluid dynamics projects to Julia, so that for a new student who is joining the project it would be much easier to start without going through learning c++ and/or fortran. I am a physicist and most of my day job is some form of scientific computing. My current default working environment is python (numpy, scipy, sympy, matplotlib) + fortran (f2py) when some part of my code needs to speed up. Yesterday I decided to start a a new, relatively easy project as a simple example for an upcoming paper. So I thought this might be a good occasion to start learning Julia language to code a simple dynamical systems toolbox in it, which might be useful for other people as well. Basic functionality I need from the language are these: - Symbolic differentiation (for computation of Jacobians) - Numerical integration of ODEs (a general purpose integrator, such as lsoda from odepack, wrapped in scipy.integrate.odeint) - Linear algebra functions - Interpolation - Plotting in 2D and 3D After reading The Julia Express and parts of the documentation, I thought that such a project is not a good investment, at least for now. The reason is all the functionality I listed above are provided by external packages, partially excluding linear algebra functions. I'm aware that I can use specific packages for all the functionality I mentioned above, but each such package is maintained by different people, and they can change or become obsolete. I can also find some Fortran/C code, and include in Julia, and have all these functionality, but then what is the advantage of using Julia, as opposed to, say, python? In a more general sense, I am a little bit turned off by the presence of an external package for almost every task I need to do. I can understand this kind of structure in python as it is a general purpose language. But since Julia is a language specifically for scientific computation, I'd be happy to have something like the basic functionality of MATLAB in the main
[julia-users] Re: Too many packages?
One of the most powerful features of Julia is that packages (and user code in general) aren't at a major disadvantage to code included in Julia itself. 0.4 promises to make this even better with precompilation and documentation. Many of the folks who have spearheaded the big packages are also contributors to Julia itself. And many of the core developers help out with package development. So the lines between user, package developer, and Julia developer are all a little blurry. Also note that many packages aren't owned by just one person - organizations have formed to help ensure packages are well-supported. I think the general direction is towards moving more functionality out of base, but to also provide a default set of preinstalled packages. That way the core language remains small, but the default user experience is rich and featureful.
