[julia-users] What is the deal with macros that do not return expressions?

[Lyndon White]

I was generally under the impression that a macro always had the return a 
`Expr`
and that doing otherwise would make the compiler yell at me.

Apparently I was wrong, at least about the second part:
https://github.com/ChrisRackauckas/ParallelDataTransfer.jl/issues/3



macro ex_timestwo(x)
:($(esc(x))*2)
end



@ex_timestwo(4)
8

macroexpand(:(@ex_timestwo(4)))
:(4 * 2)

@ex_timestwo(a)
LoadError: UndefVarError: a not defined


macroexpand(:(@ex_timestwo(a)))
:(a * 2)



?

VS: not returning an expression, and just doing a thing:

macro n_timestwo(x)
 x*2
end


@n_timestwo(4)
8


macroexpand(:(@n_timestwo(4)))
8


@n_timestwo(a)
LoadError: MethodError: no method matching *(::Symbol, ::Int64)


macroexpand(:(@n_timestwo(a)))
:($(Expr(:error, MethodError(*,(:a,2)




Is this a feature?
Or is it just undefined behavior?

[julia-users] [ANN]: Deoplete-Julia: Fairly decent syntax completions in Neovim

[Lyndon White]

Ok, I've had this kicking around for months.
It has taken me until now to get around to actually packaging it up so that 
anyone could use it.

Check it out: https://github.com/JuliaEditorSupport/deoplete-julia

It actually works pretty well (even if I can't write reasonable code in 
demos):
https://asciinema.org/a/688g8iyhj1idrtz8ooptr6iso

Neovim is very close to vim in functionality -- it can 
run all the same extensions etc.
If you run vim, you should not really even notice the change to neovim, as 
a used.
As a plug-in developer though, things have gotten way better.
Writing a smart syntax completer for regular vim was historically nearly 
impossible, because vim didn't support background tasks.
Neovim supports background takes out of the box, and these are used by Deoplete 
completion engine,
which this plugin extends.


Syntax completion is loaded from what is exported from any modules you 
`using`,
as well as from `Base` and `Core`.
It also spits out help text, as it goes (as you can see in the demo)


The way it works is a little bit (maybe a lot evil).
`jltag` loads up every module you `using` and generates a semantic tagfile 
(technically a legal exuberant-ctags file even, I think).
It uses reflection to dump out all kinds of useful info, like docstrings 
and constant values and a bunch of other stuff.
It caches this in a file, which is then loaded by a small chunck of python 
code, that runs in deoplete.
If your dependency modules change the cache is regenerated.
`jltag`  and the files it generated could be ported to be used with another 
syntax completion engine.
I think you could with only a little work, get normal Vim YouCompleteMe 
syntax completion working, by 
telling it to load from these tag files. but YCM has a terrifying codebase, 
and overall architecture.
You could probably get a bunch of other syntax completers that accept tag 
files working too.
(or you could use the ctagger from julialang/julia/contrib 
 to do that)


Anyway,
if you are a Vim user, I suggest grabbing Neovim, and Deoplete and then 
firing this up.
Let me know what you think, and raise some issues over in the repo.

Re: [julia-users] Does Julia 0.5 leak memory?

[Lyndon White]

On Sunday, 18 September 2016 21:36:24 UTC+8, K leo wrote:
>
> I am also wondering what information I should look into.
>

Learn to use valgrind -- it is a life skill (like knowing how to do a tax 
return :-P)
http://valgrind.org/

Julia specific instructions are at:
http://docs.julialang.org/en/release-0.5/devdocs/valgrind/

Valgrind's whole purpose is to tell you if a program is leaking memory.

[julia-users] Re: ANN: DelegationMacros v0.0.3

[Lyndon White]

Nice work.
I spotted https://github.com/Jeffrey-Sarnoff/Delegate.jl
last week and was pretty keen on it.
Yet to use it but really looks like it will
solve a lot of my irritation with Composition as a Inheritance.

What is the relationship between Delegate.jl and DelegateMacros.jl?
Is Delegate.jl deprecated?
Is there non-overlapping features


On Wednesday, 7 September 2016 06:14:22 UTC+8, Jeffrey Sarnoff wrote:
>
> This offers updated macros to assist with delegating into/through fields 
> of a defined type.  
>
> Two macros apply a function to field value(s) yielding the functional 
> result; 
> and two others do that and then rewrap the result in the source type.
>
> The exports are:
>@delegate_oneField, 
>@delegate_oneField_fromTwoVars,
>@delegate_oneField_asType, 
>@delegate_oneField_fromTwoVars_asType
>
> They work so:
>
> import Base:string, show, (<), (<=), abs, (-), (+), (*)
>
> immutable MyInt16
>value::Int16
> end
>
> three = MyInt16(3)
> seven = MyInt16(7)
>
>
> @delegate_oneField( MyInt16, value, [string, show])
>
> string(three) == "3"  # true
> show(seven)   # 7
>
>
> @delegate_oneField_fromTwoVars( MyInt16, value, [ (<), (<=) ] )
>
> three <  seven# true
> seven <= seven# true
>
>
> @delegate_oneField_asType( MyInt16, value, [abs, (-)])
>
> abs(three) == three   # true
> -seven == MyInt16(-7) # true
>
>
> @delegate_oneField_fromTwoVars_asType( MyInt16, value, [ (+), (*) ] )
>
> three + seven == MyInt16(3+7) # true
> three * seven == MyInt16(3*7) # true
>
>
>
>
>
>

[julia-users] Re: Saving and Loading data (when JLD is not suitable)

[Lyndon White]

On Wednesday, 31 August 2016 12:47:37 UTC+8, Lyndon White wrote:
>
>
> So I have one of the models, saved using their custom format (
> https://github.com/sbos/AdaGram.jl/blob/master/src/util.jl#L94)
> It is *6Gb*, and takes ages to load and save.
> I switch to JLD, *13Gb* even worse.
> I used Base.serialize, Its just *13Mb* -- That is a full 3 orders of 
> magnitude different.
> And it is fast to read and write from disk.
>
>
Actually, the figure for Base,Serialize is off there.
Turns out there was a ShareArray that did not serialize to disk.
As so that 13 MB file can't be deserialised.
I thought it was a bit extreme.

But the point stands, that in general there is a large category of types 
for which Base.Serialise is much much faster than JLD.

[julia-users] Saving and Loading data (when JLD is not suitable)

[Lyndon White]

I like JLD. I really do.

But I have a problem that it can't solve -- not due to a limitation of JLD, 
but due to HDF5.
HDF5 is (AFAIK) designed with rectangular arrays in mind -- everything else 
is secondary.
If your the type you are trying to serialize is more or less a  Structure 
of Arrays, your fine.
If it is an array of structures, you are not fine.
If it is a linked list/tree, you are *really* not fine.


So I have been working with some models from AdaGrams.jl 
(https://github.com/sbos/AdaGram)
Very cool stuff in my area. 

So I have one of the models, saved using their custom format 
(https://github.com/sbos/AdaGram.jl/blob/master/src/util.jl#L94)
It is *6Gb*, and takes ages to load and save.
I switch to JLD, *13Gb* even worse.
I used Base.serialize, Its just *13Mb* -- That is a full 3 orders of 
magnitude different.
And it is fast to read and write from disk.

This happens to me again and again.
Apparently the types in tend to need, do not make JLD happy.



There are 3 problems with using Base,serialize as a data storage format.

   1. *It is not stable* -- the format is evolving as the language evolves, 
   and it breaks the ability to load files
   2. *It is only usable from Julia* -- vs JLD which is, in the endm fancy 
   HDF5, anything can read it after a little work
   3. *It is not safe from a security perspective*  -- Maliciously crafted 
   ".jsz" files can allow arbitrary code execution to occur during the 
   deserialize step.



Points 2, and 3 are also true of Python's Pickle.


I haven't actually heard of anyone doing Point 3, but it seems really 
likely.
Deserialisation is a standard vector of attack, and this is not a security 
hardened part of the codebase
Here is how it can be done using Pickle 



Point 2 is pretty serious, it leads to bad open science.


Point 1 though is what we tend to focus on, at least from what I've 
observed on SO, Github issues, Juliausers, and in the docs.

I have been bit by point 1, when I fully knew I was doing the wrong thing 
by using Base.serialize as a storage format, but did it anyway,
because it change things from hours to seconds.
I updated the Julia nightly and then couldn't load any of my files.
So I ended up having to clone a version earlier, and then wrote a script to 
resave everything in to JLD.

One thing that would go a way towards solving points 2, and 3 would be a 
standard script distributed with JLD (or julia),
that does convert ".jsz" into JLD files; in a sandboxed enviroment.

Solving point 1, would be to have a script/function (possibly as part of 
compat) that can converts between version of Base.Serialize.
This might be bit tricky, given changes to Base.Serialize are often tied to 
deep changes in how the language works internally.
In ways that mean that the Base.Serialize code from before the commit the 
changed it, can not even be run after that commit.
Such a script may need to do what I did and check out two versions of julia 
and convert to an intermediary format.


The other option would be to try and create a new data storage package 
based on the same logic as Base.Serialize,
but without it breaking when the language changes. 
This would be hard, since I suspect what makes it fast, is also what makes 
it incompatible with changes: 

that the representation on disk is very close to the representation in 
memory.
Still, even if it is 100x worse that Base.Serialize, it would still (For my 
kinda data) be 10x faster that JLD.



I had hoped Protobuf.jl might be a package that I could use as an 
alternative:
But the Protobuf spec doesn't really make it good for general data storage
See https://github.com/tanmaykm/ProtoBuf.jl/issues/73



So what to do?

Right now, I try to use Base.Serialize internally, and to convert to JLD 
when I am done messing with it, and also before updating julia.

[julia-users] Re: How to set JULIA_NUM_THREADS for remote workers?

[Lyndon White]

On a per machine basis, some `profile` initialisations files run even when 
your login shell is used (ie even if a command is specified in the SSH 
parameter, i.e. even when starting julia workers),
where as others only run if you start a shell script or an interactive 
shell (rc) 
If your default shell (on the remote machines) is bash,
then setting the environment variable JULIA_NUM_THREADS in your 
`~./bash_profile` rather than in `~./.bashrc` will make it work.

See
http://stackoverflow.com/questions/415403/whats-the-difference-between-bashrc-bash-profile-and-environment

Another hack, if you really want to specify it from julia would be to make 
a function that reads your `~/.ssh/enviroment` file, and then appends 
`export JULIA_NUM_THREADS=$n`,
then calls addproc, and then when does replaced the file with its original 
content.
(Use `tryfinally`)

But these are just hacks til the issue is done.

---



On Wednesday, 17 August 2016 18:52:33 UTC+8, Oliver Schulz wrote:
>
> Sure, but *how* do I preset JULIA_NUM_THREADS in the environment of the 
> workers? It'll still be application dependent, so it can be in a .profile 
> or so (which probably won't be run anyway for the worker processes).
>
> On Wednesday, August 17, 2016 at 12:00:16 PM UTC+2, Jeffrey Sarnoff wrote:
>>
>> Hi Oliver,
>>
>> I omitted two letters "the environment" should have been "their 
>> environment":
>> "and in [each of] *their* [remote] enviroment[s] JULIA_NUM_THREADS had 
>> been preset [in each remote environment before each remote Julia had been 
>> started]..
>>
>> from in the REPL ?addprocs
>> " Note that workers do not run a .juliarc.jl startup script, nor do they
>>   synchronize their global state (such as global variables, new method 
>> definitions,
>>   and loaded modules) with any of the other running processes."
>>
>>  so it seems environment variables are one of those nonsynced things
>>
>> -- Jeffrey
>>
>>
>> On Wednesday, August 17, 2016 at 5:17:53 AM UTC-4, Oliver Schulz wrote:
>>>
>>> Hi Jeff,
>>>
>>> > If your remote workers are remotely local invocations of Julia and in 
>>> the environment JULIA_NUM_THREADS has been preset, then the remote workers 
>>> will be using that many threads
>>>
>>> I tried it, and at least when the workers are started via SSH (using 
>>> addprocs([host1, ...])), that doesn't seem to be the case, 
>>> JULIA_NUM_THREADS 
>>> doesn't seem to be passed on. It would actually be very helpful to be 
>>> able to forward (or explicitly set) environment variables for remote 
>>> workers.
>>>
>>> Cheers,
>>>
>>> Oliver
>>>
>>>
>>> On Wednesday, August 17, 2016 at 12:16:11 AM UTC+2, Jeffrey Sarnoff 
>>> wrote:

 Hi Oliver,
 As I understand it:
 JULIA_NUM_THREADS is an environment variable read by the local 
 invocation of Julia.  It is not a run-time passable value. If your remote 
 workers are remotely local invocations of Julia and in the environment 
 JULIA_NUM_THREADS has been preset, then the remote workers will be using 
 that many threads (if the have the cores).


 On Tuesday, August 16, 2016 at 11:52:20 AM UTC-4, Oliver Schulz wrote:
>
> I guess the answer is "no", then?
>
> On Monday, August 1, 2016 at 3:26:17 PM UTC+2, Oliver Schulz wrote:
>>
>> Is it possible to pass on or explicitly set JULIA_NUM_THREADS for 
>> remote workers started via
>>
>> addprocs([host1, ...])
>>
>> ?
>>
>>

Re: [julia-users] I would like to display links to new Julia Questions from StackOverflow in the Gitter Sidebar

[Lyndon White]

thanks.
It is working

On Sunday, 10 July 2016 12:25:52 UTC+8, Jacob Quinn wrote:
>
> Just sent it to you.
>
> On Sat, Jul 9, 2016 at 10:11 PM, Lyndon White <oxin...@ucc.asn.au 
> > wrote:
>
>>
>> Hi all,
>> We were discussing this in the gitter chat 
>> <https://gitter.im/JuliaLang/julia/>.
>> That it would be cool if everytime someone asked a Julia question on 
>> Stack Overflow, it would appear in the Activity sidebar.
>>
>>
>> *They way, when questions are asked on stackoverflow, people hanging 
>> around on gitter would see it, and could answer it.Thus making the 
>> community more welcoming by people who are stuck getting help sooner,and so 
>> improving the adoption of the language.*
>>
>>
>> So I threw together some code to make that happen: 
>> https://github.com/oxinabox/GitterBots.jl
>> You can see it in action on my own gitter channel: 
>> https://gitter.im/oxinabox/JuliaStackOverflowWatcher
>>
>> <http://i.stack.imgur.com/WG7OL.png>
>>
>>
>> Its just a script running in a loop on my computer, 
>> each minute it checks the stackover flow JuliaLang RSS feed,
>> and then posts them to a Gitter Custom Integration activity notifier.
>>
>> I would like to set it up to run in the main channel.
>> The Activity bar in the main channel is currently empty -- unused.
>>
>> I'm happy to host it, more or less forever, on the same server I use to 
>> host the bot that links the IRC to gitter 
>> <https://groups.google.com/forum/#!topic/julia-users/ImKYzqHXA90>
>> But unlike that bot, I can not do this without permission form a channel 
>> admin.
>> Which by default for gitter are people with commit access to the Julia 
>> repository.
>>
>> I need a channel admin to give me a webhook URL.
>>
>> This can be gotten by clicking:
>> Room Settings -> Integrations -> *Custom*
>> Then copying the URL (It should look like 
>> *https://webhooks.gitter.im/e/adb87a00ca31a22272dc 
>> <https://webhooks.gitter.im/e/adb87a00ca31a22272dc>*) and clicking done.
>> and sending it to me in an private message on gitter 
>> or an email (though that is unencripted plain text, so could I guess be 
>> snooped.  Certainly not be sending to this mailing list as that would leave 
>> it open for anyone to hook to)
>>
>> Alternatively, the repo for the bot that checks stack overflow could be 
>> cloned by a channel admin, and then they could run it themselves.
>> And so I wouldn't need to be given the webhook URL.
>>
>>
>> What do people think?
>> People who were on gitter at the time I brought it up, and showed the 
>> demonstration were in favor, I think.
>>
>> Regards
>> Lyndon White
>>
>>
>>
>>
>

[julia-users] I would like to display links to new Julia Questions from StackOverflow in the Gitter Sidebar

[Lyndon White]

Hi all,
We were discussing this in the gitter chat 
<https://gitter.im/JuliaLang/julia/>.
That it would be cool if everytime someone asked a Julia question on Stack 
Overflow, it would appear in the Activity sidebar.


*They way, when questions are asked on stackoverflow, people hanging around 
on gitter would see it, and could answer it.Thus making the community more 
welcoming by people who are stuck getting help sooner,and so improving the 
adoption of the language.*


So I threw together some code to make that 
happen: https://github.com/oxinabox/GitterBots.jl
You can see it in action on my own gitter 
channel: https://gitter.im/oxinabox/JuliaStackOverflowWatcher

<http://i.stack.imgur.com/WG7OL.png>


Its just a script running in a loop on my computer, 
each minute it checks the stackover flow JuliaLang RSS feed,
and then posts them to a Gitter Custom Integration activity notifier.

I would like to set it up to run in the main channel.
The Activity bar in the main channel is currently empty -- unused.

I'm happy to host it, more or less forever, on the same server I use to host 
the bot that links the IRC to gitter 
<https://groups.google.com/forum/#!topic/julia-users/ImKYzqHXA90>
But unlike that bot, I can not do this without permission form a channel 
admin.
Which by default for gitter are people with commit access to the Julia 
repository.

I need a channel admin to give me a webhook URL.

This can be gotten by clicking:
Room Settings -> Integrations -> *Custom*
Then copying the URL (It should look like 
*https://webhooks.gitter.im/e/adb87a00ca31a22272dc*) and clicking done.
and sending it to me in an private message on gitter 
or an email (though that is unencripted plain text, so could I guess be 
snooped.  Certainly not be sending to this mailing list as that would leave 
it open for anyone to hook to)

Alternatively, the repo for the bot that checks stack overflow could be 
cloned by a channel admin, and then they could run it themselves.
And so I wouldn't need to be given the webhook URL.


What do people think?
People who were on gitter at the time I brought it up, and showed the 
demonstration were in favor, I think.

Regards
Lyndon White

[julia-users] Re: Master list of constants

[Lyndon White]

You can use the `names` function to search a module for all constants it 
defines.
By evaluating the names to get the values, then checking if they are 
Irrational (formerly known as MathConst?)
By doing this, I conclude that the  ASCII name for the Euler-Mascheroni 
constant (γ) is eulergamma


 


julia> for name_sym in names(Base)
  value = eval(name_sym)
  if typeof(value)<: Irrational
println(name_sym, "\t", value)
  end
   end


catalan catalan = 0.9159655941772...
e   e = 2.7182818284590...
eu  e = 2.7182818284590...
eulergamma  γ = 0.5772156649015...
golden  φ = 1.6180339887498...
pi  π = 3.1415926535897...
γ   γ = 0.5772156649015...
π   π = 3.1415926535897...
φ   φ = 1.6180339887498...

[julia-users] Re: Julia community stewards

[Lyndon White]

 
Excellent  quality people.
Good of them to step up to formally take on the nontechnical roles -- ontop 
of the very significant technical roles most (all?) are already playing.


As such, can we new get a formal response from one of these Stewards to 
Bridging the IRC and Gitter 
 (https://groups.google.com/forum/#!topic/julia-users/ImKYzqHXA90)

Further to that, 
the IRC, (and thus the gitter, due to the bridge) was spammed 3 times in 
the last week (Which I think is abnormally high).
It would be nice to have someone on IRC and Gitter with the ability to kick 
spammers, or at least set up a bot to catch flooding etc.
The chat channels do seem rather lonely of stewardship, as it were.

[julia-users] Re: VS code extension

[Lyndon White]

I would be happy to 
transfer https://github.com/oxinabox/julia-vim-completions to an Org.
Though probably worth waiting til it is wrapped up into a form people who 
are not me can use.





On Wednesday, 22 June 2016 06:26:52 UTC+8, David Anthoff wrote:
>
> Hi all,
>
>  
>
> I’ve created a new github repo for a VS code extension 
> https://github.com/davidanthoff/julia-vscode and it is published here 
> https://marketplace.visualstudio.com/items?itemName=julialang.language-julia. 
> Be5invis will delete the old julia extension from the marketplace because 
> he is no longer maintaining it. At this point my extension has zero 
> additional features over his old one, except that there is a public github 
> repo where in theory people could contribute.
>
>  
>
> Two questions:
>
> 1) could we move the github repo under some official julia organization? I 
> know most other editor plugins are under julialang, but I also remember 
> talk about creating something like a juliaeditor org or something like 
> that? In any case, I would like to move the repo to something more 
> official. I’m happy to maintain it for the foreseeable future, so no fear 
> on that front.
>
> 2) I used the julia logo. I hope that is ok from a copyright point of 
> view? I took it from the repo for the julia homepage, and that whole repo 
> seemed to be under an MIT license, but just wanted to be sure.
>
>  
>
> And if anyone wants to add stuff to the extension, PRs are welcome! 
> Especially a debug adapter would of course be fantastic.
>
>  
>
> Cheers,
>
> David 
>
>  
>
> --
>
> David Anthoff
>
> University of California, Berkeley
>
>  
>
> http://www.david-anthoff.com
>
>  
>

[julia-users] Re: Let's Bridge the IRC and Gitter

[Lyndon White]

>
> Alright, I have reset this up.
>>
> There has been a fair bit of shouting by new usered on IRC of the 
"Hell, anyone here" kind.
So I figure the need.

This is now running on a microVM (1core, 700MB of RAM) 
from https://nectar.org.au/
Which is from the  Australian Government's  National Collaborative Research 
Infrastructure Strategy.
Which I have an allocation on for my research.

Julia is an essential tool for my research, so building its community I 
think is in scope.
At some point a NeCTAR administrator may disagree, and tell me to stop it.
In which case I will rehost it elsewhere.
 
It is running in the NCI zone, so if the NCI zone goes down for 
maintainance, this will go down too.
And it probably wont come back up on its own, (its just running in tmux).
So poke me if that happens and I don't notice (I should notice though).

Re: [julia-users] Re: Private Forks of Julia Repositories

[Lyndon White]

I am doing this right now, via Loadpath.
Set the loadpath in your `.juliarc.jl` so that an extra folder (where you 
are working) is also in your loadpath.
Control it with Git, (not the julia package mananger).
Everything is fine and normal ,it is just another git repo.

I have no interest in pushing my changes back upstream, as my modified 
package is not really compatible with orignal intent,
so I don't have multiple remotes.
(Well I do for other reasons)


On Friday, 3 June 2016 14:18:09 UTC+8, Mauro wrote:
>
> On Fri, 2016-06-03 at 07:58, Chris Rackauckas  > wrote: 
> > I think I will need both versions available, since the majority of the 
> work 
> > is public, while the private work will tend to sit around longer (i.e. 
> > waiting to hear back from reviewers). So I'd want to be able to easily 
> work 
> > with the public repository, but basically switch over to a private 
> branch 
> > every once in awhile. 
>
> Well, then just checkout the branch you need at the time, easy. 
>
> Alternatively, have two different folders for the two branches and set 
> the LOAD_PATH depending on what you want to do.  If the REQUIREments are 
> different, in particular, different versions, it might be more tricky. 
> I think then you'd need two ~/.julia/v0.* folders. 
>
> > On Thursday, June 2, 2016 at 9:21:13 PM UTC-7, Curtis Vogt wrote: 
> >> 
> >> If you don't need to have both versions of the package available at the 
> >> same time then I would recommend using a single Git repo with multiple 
> >> remotes. With this setup you can push to your private remote for 
> >> experiments and later push to the public remote when your ready to 
> share 
> >> your work. 
> >> 
> >> Some reading material on Git remotes: 
> >> https://git-scm.com/book/en/v2/Git-Basics-Working-with-Remotes 
> >> 
>

Re: [julia-users] Can we make Julia develop itself?

[Lyndon White]

And on that
take a look at all the interlanguage operations.
I have some old notes, I wrote:


Julia is well know for its excellent Foreign Function Interface (FFI). What 
you might not know is that it is more than C, Fortran and Python.
 There are many more. In theory we could daisy chain off of those 
languages, to get even more. Eg using Pycall and oct2py to call octave

##.*[C, Fortran, 
Rust...](http://julia.readthedocs.org/en/latest/manual/calling-c-and-fortran-code/)*
 C and Fortran are infact supported within the core system. No 3rd Party 
Libary needed. It also appears to be able to call 
[Rust](http://paul.woolcock.us/posts/rust-perl-julia-ffi.html). Presumably 
it can be used to call any number of other compiled languages, so long as 
the follow certain conventions.

*## [PyCall](https://github.com/stevengj/PyCall.jl) *
Probably the most well known Julia FFI.
Its great for NLTK, sklearn etc.
Also useful for things like Pickle

Several popular julia libraries are build around python libraries Eg:


   -  - [SymPy](https://github.com/jverzani/SymPy.jl) is 
   [SymPy](http://www.sympy.org) 
   -  - [PyPlot](https://github.com/stevengj/PyPlot.jl) is 
   [Matplotlib](http://matplotlib.org/) 



*##[Mathematica](https://github.com/one-more-minute/Mathematica.jl)*

 Far less well known. I am yet to test it, but it looks real neat. It does 
require Mathematica, and my liscense has expired. 
The [Mathlink Component can be found 
here.](https://github.com/one-more-minute/MathLink.jl).

* ##[Matlab](https://github.com/JuliaLang/MATLAB.jl) *
Like the Mathematica above requires the commercial Matlab. Does not appear 
to support octave. 

*##[JavaCall](http://aviks.github.io/JavaCall.jl/) *

This seems like a strange notion, Java doesn't seem like it has much use by 
the scientific community tht is julia's target audience. However, both 
Matlab and Mathematica have support for calling Java so there must be some 
use. CoreNLP and a few other things like that, some good datamining tools. 
Could be very useful depending on field.

*##[C++](https://github.com/Keno/CXX.jl) CXX *
used to require a custom build of Julia 0.4 with LLVM 3.7.
I think that in no longer true.


A few more have probably shown up there.



On cracking wise: 
It is not that there is no such that as *inane* questions, it is that there 
is no such thing as *stupid *questions.
A stupid question (in this context) is one that demonstrates your ignorance 
of the answer.
An inane question is one that demonstrates your ignorance of the question.


On Friday, 3 June 2016 07:24:54 UTC+8, Isaiah wrote:
>
> I'm going to assume that there is a terminology barrier here...
>
> My interpretation of the question is: can we directly translate programs 
> from other languages?
>
> This is called source to source translation or compilation (
> https://en.m.wikipedia.org/wiki/Source-to-source_compiler), and the 
> answer is "it's complicated".
>
> Kevin, hopefully that terminology will give you a better start on looking 
> for answers -- stackoverflow probably has better treatments than I can give 
> regarding why this is difficult.
>
> On Thursday, June 2, 2016, Kevin Liu  
> wrote:
>
>> Is there an answer somewhere here? 
>>
>> On Thursday, June 2, 2016 at 7:54:04 PM UTC-3, Kristoffer Carlsson wrote:
>>>
>>> Someone taught you wrong.
>>
>>

[julia-users] Re: Does anyone use BBEditTextWrangler-julia.plist?

[Lyndon White]

I am not.

Re: Generating lists exports automatically, a simple call to `names(Base)` 
would probably do it.
I'm doing more complex lists of exports in  
https://github.com/oxinabox/julia-vim-completions

Julia reflection is strong enough, for this kinda thing.

[julia-users] Re: Question about passing/using module as argument

[Lyndon White]

It is useful, some methods in Base, eg `names`, take modules as parameters.
So 1, is yes it is intended -- if for purposes of reflection if nothing 
else.

[julia-users] Re: Let's Bridge the IRC and Gitter

[Lyndon White]

Transcribing a few comments from IRC:


> 11:51 < acetoline> there's a bot that connects IRC to Gitter?
> 11:51 < Lyndon> I set one up last night.
> 11:51 < acetoline> oh cool
> 11:51 < acetoline> how does it work
> 11:52 < Lyndon> I didn't make it, but basically it just has a gitter 
> account
> and an IRC "account" and just reposts things that people 
> say.
> So that IRC and Gitter have mutual communication
> 11:52 < acetoline> cool
> 11:53 < Lyndon> I think it is a good idea given the small size of the 
> community
> for it not to be in two  "channels"
> 11:53 < acetoline> yeah
> 11:54 < acetoline> realisitically, though, most julia 'communication' 
> happens
>on github, not gitter or irc
> 11:54 < acetoline> I mean even the julia google groups pages look pretty 
> deserted
>
...

> 13:28 < jballanc> Lyndon: I could definitely see the value in merging 
> freenode
>   and gitter
> 13:29 < jballanc> Currently my solution of connecting to both (via the 
> Gitter
>   IRC gateway) is...suboptimal, shall we say?

 








On Monday, 23 May 2016 08:17:00 UTC+8, Lyndon White wrote:
>
> Hi all,
> So it should be generally known that Julia has a IRC Channel: 
> http://webchat.freenode.net/?channels=julia
> and a Gitter https://gitter.im/JuliaLang/julia
>
> The Gitter is (AFAICT) moderately active, with a mix of new-comers and 
> experienced users.
> The IRC is not.
>
> As of the last week or so I think I was the only person posting who had 
> used julia for more than a couple of months.
> Almost all posts have been people saying "I just started julia, how do i 
> X",
> and generally me responding with "By doing Y", or sometimes me saying 
> "idk, I've never Xed, try Stackoverflow"
>
> Anyway, I don't particularly mind answering questions.
> But my enthusiasm for IRC goes up and down, sometimes I stop using it for 
> months.
>
> I suggest that we should connect the Gitter and the IRC into a single 
> instant-messaging/chat "Room".
>
> Via some form of IRC-Gitter bridge.
>
> On a quick look for how to do this:
>
>
>  - Sameroom 
> <https://sameroom.io/blog/how-to-bridge-existing-irc-channels-and-rooms-on-gitter/>:
>  
> Commerical, but says "we're happy to provide chat interop to open source 
> communities."
>  - Gitter IRC-bridge <https://github.com/gitterHQ/irc-bridge>: Made by 
> the people who make Gitter (I think), doesn't bridge to existing channel 
> (AFAICT), but creates a new channel -- which we could use (and deprecate 
> Freenode), or we could hack a IRC-IRC brdige to connect freenode to it (or 
> maybe even do it properly with IRC's peering stuff, but I can't recall if 
> that is possible, particularly without intervention from a Freenode Oper)
>  - IRC2Gitter <https://github.com/shyim/Irc2Gitter>: This seems simple, 
> and open source. Looks like a chuck of node.js that you can  just stick on 
> a computer and leave running and have it do the bridging. It seem like it 
> is in current development (or at least maintance) since it got a push last 
> week.
>
> There are other more complicated routes, like I bet you can bridge GItter 
> to Slack and I know you can bridge Slack to IRC.
> There is also the simpler route of just terminating the IRC channel, and 
> telling people not to use it and to just use gitter.
>
> Having an inactive IRC does not make us look good.
>
>
>

[julia-users] Re: Let's Bridge the IRC and Gitter

[Lyndon White]

Alright for purpose of testing,
I have created an instance of gitter-irc-bot 
 on Heroku.
The program was suggested by ScottPJones.

It shows up on IRC as *GitterBot*.
and on gitter as *oxinabot *with a little robot icon saying IRC. See github 
profile .

I believe it will kill itself in about 17 hours, then possibly rise back 
from the dead 6 hours later, or maybe not.
I'll just kill it manually not long after that if it does rise up.

If this works out well, then I (or someone else) can recreate it on a VM, 
rather than running on Heroku.
And make it a github account that is called something good (Eg *JuliaBot*).

[julia-users] Let's Bridge the IRC and Gitter

[Lyndon White]

Hi all,
So it should be generally known that Julia has a IRC Channel: 
http://webchat.freenode.net/?channels=julia
and a Gitter https://gitter.im/JuliaLang/julia

The Gitter is (AFAICT) moderately active, with a mix of new-comers and 
experienced users.
The IRC is not.

As of the last week or so I think I was the only person posting who had 
used julia for more than a couple of months.
Almost all posts have been people saying "I just started julia, how do i X",
and generally me responding with "By doing Y", or sometimes me saying "idk, 
I've never Xed, try Stackoverflow"

Anyway, I don't particularly mind answering questions.
But my enthusiasm for IRC goes up and down, sometimes I stop using it for 
months.

I suggest that we should connect the Gitter and the IRC into a single 
instant-messaging/chat "Room".

Via some form of IRC-Gitter bridge.

On a quick look for how to do this:


 - Sameroom 
:
 
Commerical, but says "we're happy to provide chat interop to open source 
communities."
 - Gitter IRC-bridge : Made by the 
people who make Gitter (I think), doesn't bridge to existing channel 
(AFAICT), but creates a new channel -- which we could use (and deprecate 
Freenode), or we could hack a IRC-IRC brdige to connect freenode to it (or 
maybe even do it properly with IRC's peering stuff, but I can't recall if 
that is possible, particularly without intervention from a Freenode Oper)
 - IRC2Gitter : This seems simple, and 
open source. Looks like a chuck of node.js that you can  just stick on a 
computer and leave running and have it do the bridging. It seem like it is 
in current development (or at least maintance) since it got a push last 
week.

There are other more complicated routes, like I bet you can bridge GItter 
to Slack and I know you can bridge Slack to IRC.
There is also the simpler route of just terminating the IRC channel, and 
telling people not to use it and to just use gitter.

Having an inactive IRC does not make us look good.

[julia-users] Re: Julia Utopia: Share your tips and tricks to efficient coding in Julia

[Lyndon White]

*My tip: Use your `.juliarc.jl` to add your current work to your path.*So 
much less annoying than constantly put `push!(LOAD_PATH, directory)` at the 
top of various files.

For example I have:

function add_to_path(directory)
if isdir(directory)
push!(LOAD_PATH, directory)
end
end

add_to_path("/mnt_volume/phd/prototypes/SenseSplittingWord2Vec/src/")
add_to_path("/mnt/phd/prototypes/SenseSplittingWord2Vec/src/")

So if the folder with my current project can be found, it is added to the 
path.
If not then, not.




* managing your cluster with `-L` rather than with `--machine-file`.*
Which I am less sure is a good idea.
See this StackOverflow question 

  

[julia-users] Re: ANN: JuMP 0.13 released (or, JuMP dumps camelCase)

[Lyndon White]

>
> I like new clean syntax.
>

I threw together a simple substitution script,
to apply the changes.
 
You can use it online on https://repl.it/CL6c/6
Just change the original text in the left, for what ever is being updated.
Then run it and copy from the right.


I am tempted to go through all StackOverflow posts about julia containing 
the word JuMP and update them.

http://stackoverflow.com/search?q=+%5Bjulia-lang%5D+"JuMP;
Its only about 35 questions.

[julia-users] Re: What are the best ways to define scenarios?

[Lyndon White]

Define a type (possibly immutable) per sensation, with the constants
Create a function that operates on that type that carries out the scenario.

loop (or map (or pmap)) through an array of the aforementioned type, 
applying the aforementioned function.


On Friday, 29 April 2016 09:15:40 UTC+8, K leo wrote:
>
> Each scenario has the same set of constants but with different values.  I 
> would like to loop through all these scenarios.  What are the best ways to 
> handle this? 

[julia-users] Re: When using julia-vim should basic syntax completion work?

[Lyndon White]

Ok, finally worked it out.
Its not ideal, as it will only give local completions if there is no syntax 
completions but it is better than nothing.
I'ld rather it was the other way round, only give syntax if there is not 
local  completions. but can't seem to make that work.


"""Completions
autocmd Filetype *
\   if  == "" |
\   setlocal omnifunc=syntaxcomplete#Complete |
\   call SuperTabChain(, "") |
\   call SuperTabSetDefaultCompletionType("") |
\   endif


After first loading Julia-vim plugin etc.

[julia-users] When using julia-vim should basic syntax completion work?

[Lyndon White]

As I understand it, VIM should be working out what completions to use based 
on the syntax highlighting file.

So I should at least get basic stuff like `using` and `function` 
autocompleting?

I am using vim 7.4 with super tab.
I never got my head around using completions, I just followed install 
instructions with what ever plugin I was using, and it normally just worked.

After a hour of messing with my .vimrc I have deleted all the config 
setting related to completion.
LaTeX to Unicode completions work fine, when I press tab.
As does completion of any word which appeared already in my file

But none of the language keywords complete.
Ideally I would also want the functions defined in Base to autocomplete too.
But neither works

[julia-users] Re: Should Iterators be nonmutating?

[Lyndon White]

I'm not sure changing this behavior is possible,
at least not without making IOStream clonable.
And I suspect there are filesystem constraints against this.

Well there is a scary way, when the state just holds the current file 
position,
then seeks to it when next is called,
reads a line,
then seeks back to where it was before.
So it looks like we are not mutating the underlying object.
(cos we undo our changes)
This would not be threadsafe though.


On Saturday, 23 April 2016 05:22:11 UTC+8, Dan wrote:
>
> Looks like it might be appropriate to make the `stream` in `Eachline` be 
> the state of the iterator (instead of the `nada` returned by `next`).
>
> Opening an issue (perhaps with a PR of a fix) could be the next step. 
> Unless someone finds some other logic for keeping `Eachline` the way it is.
>
> On Friday, April 22, 2016 at 7:09:13 AM UTC+3, Lyndon White wrote:
>>
>> I am writing some iterators that work with `IO`.
>>
>> I thought, so that my iterators would not mutate,
>> I would `deepcopy` the IO item, and store it as part of my state.
>> (then only my state would mutate, and no-one cares probably (unless they 
>> were trying to `tee` for free))
>>
>> But `deepcopying` IO items apparently does not actually cause them to 
>> become separate.
>> At least not for IOStreams.
>>
>>
>>
>> io=open("./data/text8/text8","r")
>> ia = deepcopy(io)
>> >IOStream()
>>
>>
>> position(io)
>> >0
>>
>> position(seekend(io))
>> >1
>>
>> position(ia)
>> >1
>>
>>
>>
>> I was under the assumption that iterators never mutated, as there is no 
>> `!` in `next`.
>> (then again there is also no `!` in `seekend`)
>>
>> So I was a bit stuck for how to implement.
>>
>> So I thought I would see how `Eachline` is implemented. 
>> <https://github.com/JuliaLang/julia/blob/master/base/io.jl#L356>
>> It is mutating.
>> So maybe my assumptions are wrong, and iterators can be mutating?
>> They would be a lot easier to write that way, I guess.
>>
>> But julia does not have a `tee` function 
>> <https://docs.python.org/2/library/itertools.html#itertools.tee>
>>
>>
>>
>>

Re: [julia-users] Any downside of using a symbol as an interned string?

[Lyndon White]

@Tamas thanks
Can you clarify a Dict of strings to what? Ints?
Loose a lot of interpretablity that way.


On Friday, 22 April 2016 19:08:11 UTC+8, Tamas Papp wrote:
>
> I would be more concerned about style than speed -- symbols as strings 
> is an ancient Lisp technique in NLP, but IMO a Dict of strings would be 
> better style. 
>
>
>

[julia-users] Any downside of using a symbol as an interned string?

[Lyndon White]

When tokenizing large files, 
it is normal to end up with many many multiples of the same string.

Normal julia strings are not interned.
Which means if you accumulate a large list of tokens,
you end up duplicating a lot of strings, which uses unnesc memory.

When you are tokenizing documents that are multiple gigabytes long,
this really adds up.


`symbols` *are  *interned.
Is there any downsides to using them, when an interned string is required?

I tried testing them for it a while ago, and got Huge improvments in memory 
use, and thus also in speed (allocating memory is expensive).

There are not `convert` methods defined for switching between symbols and 
strings but
`string(::Symbol)` and `symbol(::AbstractString)` work.

[julia-users] Re: Function check without @assert

[Lyndon White]

*TL;DR; Assert is no slower than exception throwing&  the problem is people 
expect it to be a `nop` if optimisation is turned on. and a `nop` is much 
faster than an assert*(*I haven't actually tested this),


Its not that assert is slow.
it is that traditionally (and we are talking going back to a very long 
time),
`assert` statements are removed by the optimizer. (Eg. It is one of the 
only things done by `cpython -o`)
Because `asserts` are error messages targeted at the developer.
In theory they will never be triggered at run time, in "shipped" code.

This is different from Exceptions.
Exceptions may be thrown (and not caught) in shipped code.
Its bad, but sometimes unavoidable,
and it is better that the exception is thrown (and not caught) than to 
allow the program to run in an unknown state (potentially leading to data 
corruption, or infinate looping or all kinds of problems).

vs if the exception was instead an assert, then in the shipped version they 
are not their and so the program runs in unknown state.

Currently running `julia --optimise` does not actually remove them.
This is generally considered a bad thing.
It break the expectation that they will be gone in the optimised version.
This is very bad, if you have an `assert` in a key inner loop.



However, some people are writing code, that is using asserts to check that 
a function is being used correctly.
Because they are lazy (aren't we all? I know I do this).
If this code is ship in optimized form (without asserts), then the users of 
there code (if it is a library) are going to be be able to get into 
unintended states -- by accident.

Now what people should do, is as in your example check conditions and throw 
meaning full exceptions.
Your way works, as does using an if statement (i'ld use an if statement, 
but that is just me).
But as a hack for those of us who are lazy, a @check macro was proposed,
which would not be removed by the optimizer.

[julia-users] Should Iterators be nonmutating?

[Lyndon White]

I am writing some iterators that work with `IO`.

I thought, so that my iterators would not mutate,
I would `deepcopy` the IO item, and store it as part of my state.
(then only my state would mutate, and no-one cares probably (unless they 
were trying to `tee` for free))

But `deepcopying` IO items apparently does not actually cause them to 
become separate.
At least not for IOStreams.



io=open("./data/text8/text8","r")
ia = deepcopy(io)
>IOStream()


position(io)
>0

position(seekend(io))
>1

position(ia)
>1



I was under the assumption that iterators never mutated, as there is no `!` 
in `next`.
(then again there is also no `!` in `seekend`)

So I was a bit stuck for how to implement.

So I thought I would see how `Eachline` is implemented. 

It is mutating.
So maybe my assumptions are wrong, and iterators can be mutating?
They would be a lot easier to write that way, I guess.

But julia does not have a `tee` function 

[julia-users] Re: Packaging Julia project for Open Respoducable Science

[Lyndon White]

I just realized that not only am I dependent on Julia packages.
but via PyCall, I am also dependent on python packages.
How utterly terrifying packaging this will be.

I suspect I will end up failing to meet the goal of a package anyone can 
just click and run.
But I can at least release all the code in a nice view-able form.

[julia-users] Packaging Julia project for Open Respoducable Science

[Lyndon White]

I've been working in IJulia for all of my experimental needs.
I am now submitting the paper and I wish to submit my code and data  along 
with it.

I also probably should include a copy of julia.
Since my code works in Julia 0.5, but I am confidant it doesn't work in any 
prior versions (such as the one in most package managers?)
and I guess I need to include the dependencies.

Are there guidelines for packaging up a julia project for reproducible 
science?

[julia-users] Is the Actor model for parallelism a good paradigm for Julia

[Lyndon White]

Hi,

So I do a lot of batch processing, for machine learning.
I have a lot of RAM, 45Gb, and 12 cores.

My normal method for parallel processing is to replicate any common shared 
read-only memory across all workers, using @everywhere.
Then process my data with pmap, or a variation of my own pmapreduce.

On my work yesterday, I couldnot replicate my common shared memory across 
all workers, as it was too large (17Gb).
So I left it on processor 1, and told the workers to do a remotecall_fetch 
to retrieve it.
This seems to have worked very well, as the workers quickly get out of sync 
so beg from processor 1 at different times.
And processor 1 is normally unused til the worker are all done anyway.

I was thinking about it after I went home, and realised that this was a 
very rough approximation of the Actor model
(If I am recalling the Actor 
model correctly).

What I am thinking,
is I could have 1 worker per service -- where a service is combination of 
data and functions that operate on it.
Which is more than 1 worker per core.
When ever a worker needs that data, it remotecall_fetches the function on 
the services worker.
(Potentially it does smarter things than a remotecall_fetch, so it is 
unblocking.)


Is this sensible?

[julia-users] Why does using behave the way it does with regards to remote workers?

[Lyndon White]

Hi all, a discussion on the IRC channel prompts me to ask.
>From a design point of view why does `using` behave the way it does to 
remote workers?

Assume some module called `FooModule` which exports `bar`

`using FooModule`
locally loads the module, and brings it into scope -- so running locally 
`bar` works, as does running locally `FooModule.bar`
on all workers it loads the module, but does not bring it into scope so 
running on worker 2 (for example) `bar` give an unknown function error, but 
running `FooModule.bar` works

`@everywhere using FooModule` causes it to be loaded locally and thus on 
all workers, and also to be loaded on all workers and brought into scope.
Thus you get warnings about replacing modules as is loaded twice, but 
everything is in scope, so running on worker 2 (again for example) `bar` 
works.

Why have this behavior?
Wouldn't it be cleaner is using only acted locally?



Discussion log from IRC follows for further context:

Regards
Frames



 

> Day changed to 26 Nov 2015
> 05:37 < Travisty> Is there an easy way to use a single file to load code 
> for
>   multiple workers /and/ act as a driver script? Currently 
> I
>   have been putting an “@everywhere begin … end” block in 
> the
>   script to make sure that all the workers have the 
> appropriate
>   functions defined.
> 05:38 < Travisty> But when I say “using SomeModule” inside the @everywhere
>   block, I get a bunch of warnings about the module being
>   replaced, which makes me think this might not be the 
> desired
>   approach
> 08:39 < Frames> @Travisty it took me quiet a while to work out how to do 
> using
> everywhere
> 08:40 < Frames> So let me explain, that the knowledge may be shared
> 08:40 < Travisty> Frames: sure, thanks
> 08:40 < Travisty> (I should mention that what I’ve done works, but it 
> gives the
>   annoying warning about the modules being replaced)
> 08:42 < Frames> If you run `using FooModule` on the main window, then
> `FooModule` is loaded both locally and in all workers, and
> furthermore locally (only) all exported functions are 
> brought
> into scope. (continued in next messagee...)
> 08:43 < Frames> So if `FooModule` exports `bar`, locally you can run 
> `bar`, but
> on all workers you much run `FooModule.bar` (which also 
> works
> locally)
> 08:47 < Frames> On the other hand if you run `@everywhere using
> `FooModule.bar`, then the stuff that happens when you run
> `using FooModule` happens (as it runs locally and on all
> workers), but also Simultainiously *!* it tiggers `using
> FooModule` as a local command on the remote workers bring 
> it
> into scope so now `bar` can run everywhere.  (But this 
> does not
> trigger all other remote workers to again get the remote 
> load
> but
> 08:47 < Frames> do not bring into scope, as under the default cluster 
> manager
> topology workers can't see each other)
> 08:49 < Frames> The inportant footnote *!* is that this can trigger a 
> (mostly
> harmless) Race condition with the modules all trying to
> precompile and cache the module at the same time. This 
> breaks
> some modules (yet to work out which or exactly what goes 
> on)
> 08:51 < Travisty> Frames: Yeah, I read about the behaviour of using
> 08:51 < Travisty> is there an argument for it behaving this way?
> 08:51 < Travisty> I mean, importing the module on the workers but not 
> bringing
>   the exported names into the namespace?
> 08:52 < Travisty> Is there a way to get the workers to also import the
>   namespace of the module?
> 08:52 < Frames> The best way I have for doing this is to first run `import
> FooModule` then `@everywhere using FooModule`, which 
> causes it
> to be loaded locally, then it is not recompiled again as 
> it is
> already compiled locally, then `@everywhere using 
> FooModule`
> brings it into scope everywhere. Thus you get no warnings 
> as it
> has Not being loaded, as using doesn't trigger locally thus
> does not trigger the remote loads in the
> 08:52 < Frames> workers (WWhich may itself be a bug_
> 08:52 < Frames> )
> 08:53 < Travisty> ah
> 08:53 < Travisty> so import on the driver and then @everywhere using
> 08:53 < Frames> --- I am really not sure why it is that way.
> 08:53 < Frames> ^Yes. Import then @everwhere using.
> 08:54 < Frames> I was trying to make a macro to do it, but it does not 
> seem to
> like.
> 08:54 < Frames> Shall I copy and post this whole discussion to the mailing 
> list?
> 08:54 < Travisty> 

[julia-users] Re: Too many packages?

[Lyndon White]

For your interest:
of your core packages: numpy, scipy, sympy, matplotlib, and f2py

numpy functionlity should be all in Base.
Sympy has a julia version https://github.com/jverzani/SymPy.jl, wrapping 
pyCall, and it is great.
Similarly so does matplotlib https://github.com/stevengj/PyPlot.jl, 
though I've not used it, personally it was a relief to get a way from 
matplot lib and onto something less explicit and more intutive like GadFly 
https://github.com/dcjones/Gadfly.jl
I've not used f2py, but I suspect core ccall 
http://julia.readthedocs.org/en/latest/manual/calling-c-and-fortran-code/will 
do it.

Scipy is huge and vast, and I constantly forgetting what is in there.
The most useful parts of Scipy, tend to be in the Managed Julia Github 
Groups, or including the Core Group https://github.com/JuliaLang

The managed groups (self managed that is) can be found on the communities 
page http://julialang.org/community/
Cutting it down to just the things that I believe are in Scipy: (Some 
overlap with sklearn)

   - JuliaStats https://github.com/JuliaStats – Statistics 
   http://www.juliastats.org/
   - JuliaOpt https://github.com/JuliaOpt – Optimization 
   http://www.juliaopt.org/
   - BioJulia https://github.com/BioJulia - Biology
   - JuliaAstro https://github.com/JuliaAstro – Astronomy
   - JuliaQuantum https://github.com/JuliaQuantum – Julia libraries for 
   quantum science and technology http://juliaquantum.github.io/
   - JuliaSparse https://github.com/JuliaSparse – Sparse matrix solvers
   - JuliaDiff https://github.com/JuliaDiff/ – Differentiation tools 
   http://www.juliadiff.org/
   - JuliaDSP https://github.com/JuliaDSP – Digital signal processing
   - JuliaGraphs https://github.com/JuliaGraphs – Graph Theory and 
   Implementation

As Matt Bauman points out, the julia devs / major contributors tend to be 
major players in one or more communities.
Packages within the communities tend to be of higher confidence in there 
reliability.
More certain to be maintained. You can normally trust a package from one of 
the communities.
So they are always my first stop when looking for a package to do something.
My second stop right now tends to by using PyCall (for sklearn and NLTK).


Hope that helps.

[julia-users] Re: preferred syntax for creating empty vectors?

[Lyndon White]

Nice thing about: Int[] is that it is visially similar to a empty Array 
comprehension,
eg Int[round(Int, x) for x in xs]

  


On Friday, 19 June 2015 23:49:01 UTC+8, Seth wrote:



 I note that Int[] works in 0.3 and 0.4, but Vector{Int}() and 
 Array{Int,1}() were introduced at some point in 0.4 (but this syntax 
 hasn't been backported or Compat'ed for 0.3 - I have an issue open for 
 that). Going forward, what's the recommended way to do this? (Am I missing 
 another constructor?)

[julia-users] Re: Julia equivalent for real threads

[Lyndon White]

You can do it with the parallel stuff.
Without using sharedmemory, and with using processes instead of threads.
Its not particularly nice but here is how it goes: (Tested in 0.39)


first add a second process (before the @everywhere calls):

pidB = addprocs(1)[1]



A mockup of your blocking kernal function

@everywhere function fake_kernal_fun()
sleep(5*rand()+5)
rand(1:100)
end

@everywhere function B_blocking_call_to_kernal(kernal_fun_ret::RemoteRef)
@assert kernal_fun_ret.where==myid()

while true
ret = fake_kernal_fun()
put!(kernal_fun_ret, ret)
end
end


A Function to do the reading:
We will take! the value from the RemoteRef if it is ready, or if it isn't 
we will continue using the old value.


function read_kernal_fun(kernal_fun_ret::RemoteRef)
latest_value = take!(kernal_fun_ret) #initialise it -- this will wait 
til we have at least the first result
for ii in 1:20
latest_value = isready(kernal_fun_ret) ? take!(kernal_fun_ret) : 
latest_value
println(ii,'-', latest_value)
sleep(3)
end
end


Putting them together:
shared_kernal_fun_ret = RemoteRef(pidB)
remotecall(pidB, B_blocking_call_to_kernal,shared_kernal_fun_ret) #Never 
use the remoteref returned by this remotecall, it will never be valid (as 
the called function does not ever return)
read_kernal_fun(shared_kernal_fun_ret)


That works fine for me 
Outputs:

1-47
2-47
3-47
4-46
5-46
6-19
7-19
8-19
9-31
10-31
11-31
12-37
13-37
14-37
15-13
16-13
17-63
18-63
19-48
20-48



I think it is fine, and don't suffer from race conditions in a problematic 
way.

Your other option I can think of it to write that section in C and call it 
with ccall. I have no experience wih that though.

[julia-users] Re: Never use Containers of abstract types? Aways Union with containers of subtypes, or use containers of Typed Parameterized containers?

[Lyndon White]

Oh, wait you meant that Matrix{Union(...)} should have same performance as 
Matrix{Number}?
Yes, I also assume that (I guess I should test).

But Union{Matrix{Float64}, Matrix{Number},...}
is fast.

---



On Friday, 12 June 2015 08:47:12 UTC+8, Lyndon White wrote:



 On Thursday, 11 June 2015 21:26:25 UTC+8, andrew cooke wrote:


 i don't know of any docs that justify this, but i would assume that Union 
 and Number would have very similar performance.


 You would be wrong.
 Union and {T:Number} have similar performance.
 Number performed 1000x worse.

 For several reasons {T:Number} is better, inclusing more sane error 
 messages, less having to type out all subtypes etc.

[julia-users] Re: Never use Containers of abstract types? Aways Union with containers of subtypes, or use containers of Typed Parameterized containers?

[Lyndon White]

On Thursday, 11 June 2015 21:26:25 UTC+8, andrew cooke wrote:


 i don't know of any docs that justify this, but i would assume that Union 
 and Number would have very similar performance.


You would be wrong.
Union and {T:Number} have similar performance.
Number performed 1000x worse.

For several reasons {T:Number} is better, inclusing more sane error 
messages, less having to type out all subtypes etc.

[julia-users] Never use Containers of abstract types? Aways Union with containers of subtypes, or use containers of Typed Parameterized containers?

[Lyndon White]

Containers of abstract types are not performant,
This is documented in the performance notes section of the manual.

However containers of concrete types are not generic.
For example, I am currently redoing a large section of code from using 
Float64s to using Numbers, so that I can use DualNumbers to check 
derivatives.

The reason Number (and other abstract types) are slow, is because are full 
of pointers.
Since each element of a Matix{Number} could potentially be of a different 
type, eg some could be Int8s and some could be Float64s.
But in practice the varse majority of the time, every element is of the 
same type.

The 3 signitures:

Matix{Number}
Matrix{Union(Number, Float64, Float32, Float16, Int...) }
Matrix{T:Number}


All can contain the same information, as they will all contain a 
Matrix{Number}, and none of them will contain a Matrix{NotNumber},
 (Though to fit a concrete typed matrix into a matrix of abstract type you 
need to run a convert),

But they have very different performance.
As I can show with the following Notebook:
http://nbviewer.ipython.org/gist/oxinabox/a00d3b1eb5584467e6b7
Which also compares with Any just for the sake of seeing if contrete types 
help (they don't)

What can be seen t Matrix{Union(Number, Float64) }, Matrix{T:Number} and 
Any perform about the same
and that Matrix{Number} is 3 orders of magnitude slower (not unexpected).

Ergo, because the former can contain anything that can be contained in the 
Matrix{Number}, there is AFAICT, no reason not to use them.
Worse case senario, the contents really is a Matrix of mixed numeric types 
and the performance falls back to the Matrix{Number} case.

Syntactically perhaps it is less nice.
I have a type alias for Matrix{Union(Number, Float64,...)} which solves 
that, to an extent.
Argument could be made that Matrix{T:Number} is even better, but it is 
annoying to refactor code to use that as it is not a simple find and 
replace as a type parameter needs to be added to all functions.




Thoughts, comments, 
Reasons why Matrix{Number} might be users over either of the other cases?

[julia-users] Is anyone, anywhere using Julia on a supercomputer/cluster

[Lyndon White]

Hi,

I have a big numerical problem that julia is nice for.
But I really want to farm it out over a few hundren cores.

I know my local research supercomputing provider (iVec since I am in 
Western Australia),
prefers it if you are running programs in C or Fortran.

But I know they have run things in Python and Matlab.
I know they losely appreciate the trade off between development time and 
CPU time. 
But I think there main hesitation is the knowledge that the CPU cycles 
python wastes could be going to another project,
and that other project could be curing cancer etc.

Julia on the other hand is comparable to C or Fortran, so that objection is 
out.
It is on the other hand imature and not exactly well known.
(I would not be surpised if i was the only user in my universivy.

It would help any argument I might have,
or explination I need to render if I could say: They are using it on the 
super-computers in X.

Have you, or do you know of anyone who used it on supercomputers / 
medium-large clusters?

Did it go well?
What are the pitfalls