Re: [julia-users] ANN node-julia 1.2.0

[Jeff Waller]

Jeff, your continued dedication to this constantly impresses me.  Good work.
>
> -E
>

Heh, said the guy that was fixing my libuv problems @ 3am.  Back at you, 
bro (high five). 

Re: [julia-users] ANN node-julia 1.2.0

[Elliot Saba]

Jeff, your continued dedication to this constantly impresses me.  Good work.
-E

On Thu, Jul 9, 2015 at 1:28 PM, Jeff Waller  wrote:

> A couple of new features with this version
>
> Windows  support (finally)
>
> There were a couple of things preventing this.
>
> First off Julia is compiled with gcc on Windows, but node-gyp needs MSVC,
> that had to be overcome, the
> the good news was the library libjulia.dll can be used by the MIcrosoft
> compiler/linker so long as an implib
> is created first (libjulia.lib).  This can be generated from
> libjulia.dll.  The same thing has to occur with
> libopenlibm.dll, because julia uses libm functionality not available in
> Microsofts libm.  These openlibm symbols
> had to be taken directly from openlibm.dll because though it's linked into
> julia.exe, it's not (can't be?) linked into
> libjulia.dll.  Any embedding program on Windows would suffer from this see
> #11419  for instance.
>
> Second, this really needs to all happen automatically.  The previous
> version assumed these Microsoft libraries
> were already in place, but that's really putting too much of a burden for
> someone that want's to just do `npm install`
> This version takes care of that.
>
>
>
> Shared Buffers
>
> Hey remember the question in this announce
> ,
> if it's possible for Julia and Javascript to share the same underlying
> memory buffer for arrays?  Well it is possible, and this version
> implements that.
>
> Yes, It is more efficient especially if the array is used multiple times,
> as before it would have to be copied back and forth
> and in addition there are some cute tricks.
>
> For instance node has problems with large arrays if it has to manage them
>
> > x = new Int32Array(536870911)
>
> RangeError: Invalid array buffer length
>
>at new ArrayBuffer (native)
>
>at new Int32Array (native)
>
>at repl:1:5
>
>at REPLServer.defaultEval (repl.js:132:27)
>
>at bound (domain.js:254:14)
>
>at REPLServer.runBound [as eval] (domain.js:267:12)
>
>at REPLServer. (repl.js:279:12)
>
>at REPLServer.emit (events.js:107:17)
>
>at REPLServer.Interface._onLine (readline.js:214:10)
>
>at REPLServer.Interface._line (readline.js:553:8)
>
> ...
>
> > x = new Int32Array(268435455)
>
> FATAL ERROR: invalid array length Allocation failed - process out of
> memory
>
>
> But v8 will allow indexing of arrays up to 2^31 -1 if only those could
> somehow be created...
>
> > bizarro% node
>
> > julia = require('node-julia')
>
>
> > var x = julia.eval('Array(Int32,2^31 -1)')
>
> undefined
>
> > x.length
>
> 2147483647
>
> > x[2147483646] = 1;  // does not crash
>
>
> There are some other updates s as well, but that's the highlights.
>
>

[julia-users] ANN node-julia 1.2.0

[Jeff Waller]

A couple of new features with this version

Windows  support (finally)

There were a couple of things preventing this.  

First off Julia is compiled with gcc on Windows, but node-gyp needs MSVC, 
that had to be overcome, the
the good news was the library libjulia.dll can be used by the MIcrosoft 
compiler/linker so long as an implib
is created first (libjulia.lib).  This can be generated from libjulia.dll. 
 The same thing has to occur with
libopenlibm.dll, because julia uses libm functionality not available in 
Microsofts libm.  These openlibm symbols
had to be taken directly from openlibm.dll because though it's linked into 
julia.exe, it's not (can't be?) linked into 
libjulia.dll.  Any embedding program on Windows would suffer from this see 
#11419  for instance.

Second, this really needs to all happen automatically.  The previous 
version assumed these Microsoft libraries
were already in place, but that's really putting too much of a burden for 
someone that want's to just do `npm install`
This version takes care of that.



Shared Buffers

Hey remember the question in this announce 
, 
if it's possible for Julia and Javascript to share the same underlying
memory buffer for arrays?  Well it is possible, and this version implements 
that.

Yes, It is more efficient especially if the array is used multiple times, 
as before it would have to be copied back and forth
and in addition there are some cute tricks.  

For instance node has problems with large arrays if it has to manage them

> x = new Int32Array(536870911)

RangeError: Invalid array buffer length

   at new ArrayBuffer (native)

   at new Int32Array (native)

   at repl:1:5

   at REPLServer.defaultEval (repl.js:132:27)

   at bound (domain.js:254:14)

   at REPLServer.runBound [as eval] (domain.js:267:12)

   at REPLServer. (repl.js:279:12)

   at REPLServer.emit (events.js:107:17)

   at REPLServer.Interface._onLine (readline.js:214:10)

   at REPLServer.Interface._line (readline.js:553:8)

...

> x = new Int32Array(268435455)

FATAL ERROR: invalid array length Allocation failed - process out of memory


But v8 will allow indexing of arrays up to 2^31 -1 if only those could 
somehow be created...

> bizarro% node

> julia = require('node-julia')


> var x = julia.eval('Array(Int32,2^31 -1)')

undefined

> x.length

2147483647

> x[2147483646] = 1;  // does not crash


There are some other updates s as well, but that's the highlights.

[julia-users] ANN node-julia 1.1.0

[Jeff Waller]

This version supports 0.4.x using svec as well as previous 0.4.x and of 
course
0.3.x as well here's the docs  if interested.

It's been a pretty long time + the svec change was breaking, so not all of 
the
feature requests made it in to this, like the cool one 
, but that's next for sure.

Various bug fixes.

A couple new features features

* supporting the result of julia import as an object similar to node.js 
require Issue 6 
* supporting struct types with Javascript accessor functions Issue 7 


Allows for pretty cool syntax, i think (cut-n-pasted from docs)

var julia = require('node-julia');
var JuMP = julia.import('JuMP');
var m = julia.eval('m = JuMP.Model()');
var x = julia.eval('JuMP.@defVar(m,0 <= x <= 2)');
var y = julia.eval('JuMP.@defVar(m,0 <= y <= 30)');

julia.eval('JuMP.@setObjective(m,Max, 5x + 3*y)');
julia.eval('JuMP.@addConstraint(m,1x + 5y <= 3.0)');

var status = JuMP.solve(m);

console.log('Objective value: ',JuMP.getObjectiveValue(m));
console.log('X value: ',JuMP.getValue(x));
console.log('Y value: ',JuMP.getValue(y));

Re: [julia-users] ANN node-julia 1.0.0

[Tony Kelman]

Yeah, considering issue #1 is "Installation is too complex!" I'm not sure 
whether it's exactly production-usable or widely known at this point. But 
of the "linear algebra for node / browsers" projects I've seen out there, 
it appears to be the most sophisticated by a pretty substantial margin. 
Most others look to be simple wrappers of some small fraction of the 
reference netlib blas/lapack, or naive JS implementations that look a lot 
more like the reference versions at a source level than a proper optimized 
blocked, SIMD, multithreaded etc modern BLAS.


On Saturday, January 17, 2015 at 8:59:45 PM UTC-8, Jeff Waller wrote:
>
>
>
> On Saturday, January 17, 2015 at 10:59:13 PM UTC-5, Tony Kelman wrote:
>>
>> Might be interesting to compare the performance here vs 
>> https://github.com/amd/furious.js
>>
>
> Ok I'm up for this, it's a lot of stuff, hope the cut-and-paste install 
> method works.  The unit test page 
>  looks like
> it's not all there,  they haven't updated lately, though (Aug) what's up?
>

Re: [julia-users] ANN node-julia 1.0.0

[Jeff Waller]

On Saturday, January 17, 2015 at 10:59:13 PM UTC-5, Tony Kelman wrote:
>
> Might be interesting to compare the performance here vs 
> https://github.com/amd/furious.js
>

Ok I'm up for this, it's a lot of stuff, hope the cut-and-paste install 
method works.  The unit test page 
 looks like
it's not all there,  they haven't updated lately, though (Aug) what's up?

Re: [julia-users] ANN node-julia 1.0.0

[Jeff Waller]

On Saturday, January 17, 2015 at 10:02:21 PM UTC-5, Test This wrote:
>
> This is awesome. That would provide an easy way to scale a web-application 
> at least to some extent by outsourcing the CPU intensive calculation out of 
> Node. 
>

And you get access to BLAS, Mocha (GPU even), JuMP, Optim, Stats.  Good for 
fast access to website oriented machine learning stuff;  regressions, 
recommender systems, etc.
 

>
> Currently, Julia takes a long time to start up. Does the Julia process 
> start when "include myjuliafile.jl" is called? If so, can one call this 
> line somewhere when the server starts. 
>

Because it's an embed, Julia exists/executes as part of the node process in 
another thread.  The engine starts up the first time a call to eval, exec, 
Script is made.  For first time stuff especially loading big packages, 
there's going to be some lag because of the action of the JIT, but I 
haven't seen lag otherwise.
 

> Otherwise, the start up time will increase the wait for the web clients. 
>

There should not be re-JITing per connection. so I expect this to not be a 
problem.  Feedback? 

Re: [julia-users] ANN node-julia 1.0.0

[Tony Kelman]

Might be interesting to compare the performance here 
vs https://github.com/amd/furious.js


On Saturday, January 17, 2015 at 7:02:21 PM UTC-8, Test This wrote:
>
> This is awesome. That would provide an easy way to scale a web-application 
> at least to some extent by outsourcing the CPU intensive calculation out of 
> Node. 
>
> Currently, Julia takes a long time to start up. Does the Julia process 
> start when "include myjuliafile.jl" is called? If so, can one call this 
> line somewhere when the server starts. 
> Otherwise, the start up time will increase the wait for the web clients. 
>
>
>
>
>
> On Saturday, January 17, 2015 at 12:32:36 PM UTC-5, Jeff Waller wrote:
>>
>>
>>
>> On Saturday, January 17, 2015 at 10:53:36 AM UTC-5, Test This wrote:
>>>
>>> Some basic questions: 
>>>
>>> We know that Node is blocking on cpu intensive tasks. If I use the async 
>>> option, are the calculations run separately. That is does it allow the mode 
>>> process to continue with its event loop. 
>>
>>
>> The answer is yes, and the reason is though node is singly threaded, 
>> native modules can be multi-threaded,
>> the main thread accepts work and then queues it along with the function 
>> to return the result to and returns
>> immediately meanwhile a couple of other threads dequeue the work evaluate 
>> and then enqueue the result and
>> finally the main thread is signaled using uv_async_send.
>>  
>>
>>> If the answer is yes, what happens when julia is busy with a previous 
>>> calculation and a new one is passed to it? 
>>>
>>
>> Currently there's good news and bad news.  The good news is that 
>> node-julia async calls will not block and so
>> node will not block, but the bad news is the the evaluator running in a 
>> separate thread waits for the result before
>> moving on to the next thing.  
>>  
>>
>>>
>>> Thanks so much for creating this.
>>
>>
>> Oh man, you're welcome, most definitely.
>>  
>>
>

Re: [julia-users] ANN node-julia 1.0.0

[Test This]

This is awesome. That would provide an easy way to scale a web-application 
at least to some extent by outsourcing the CPU intensive calculation out of 
Node. 

Currently, Julia takes a long time to start up. Does the Julia process 
start when "include myjuliafile.jl" is called? If so, can one call this 
line somewhere when the server starts. 
Otherwise, the start up time will increase the wait for the web clients. 





On Saturday, January 17, 2015 at 12:32:36 PM UTC-5, Jeff Waller wrote:
>
>
>
> On Saturday, January 17, 2015 at 10:53:36 AM UTC-5, Test This wrote:
>>
>> Some basic questions: 
>>
>> We know that Node is blocking on cpu intensive tasks. If I use the async 
>> option, are the calculations run separately. That is does it allow the mode 
>> process to continue with its event loop. 
>
>
> The answer is yes, and the reason is though node is singly threaded, 
> native modules can be multi-threaded,
> the main thread accepts work and then queues it along with the function to 
> return the result to and returns
> immediately meanwhile a couple of other threads dequeue the work evaluate 
> and then enqueue the result and
> finally the main thread is signaled using uv_async_send.
>  
>
>> If the answer is yes, what happens when julia is busy with a previous 
>> calculation and a new one is passed to it? 
>>
>
> Currently there's good news and bad news.  The good news is that 
> node-julia async calls will not block and so
> node will not block, but the bad news is the the evaluator running in a 
> separate thread waits for the result before
> moving on to the next thing.  
>  
>
>>
>> Thanks so much for creating this.
>
>
> Oh man, you're welcome, most definitely.
>  
>

Re: [julia-users] ANN node-julia 1.0.0

[Jeff Waller]

On Saturday, January 17, 2015 at 10:53:36 AM UTC-5, Test This wrote:
>
> Some basic questions: 
>
> We know that Node is blocking on cpu intensive tasks. If I use the async 
> option, are the calculations run separately. That is does it allow the mode 
> process to continue with its event loop. 


The answer is yes, and the reason is though node is singly threaded, native 
modules can be multi-threaded,
the main thread accepts work and then queues it along with the function to 
return the result to and returns
immediately meanwhile a couple of other threads dequeue the work evaluate 
and then enqueue the result and
finally the main thread is signaled using uv_async_send.
 

> If the answer is yes, what happens when julia is busy with a previous 
> calculation and a new one is passed to it? 
>

Currently there's good news and bad news.  The good news is that node-julia 
async calls will not block and so
node will not block, but the bad news is the the evaluator running in a 
separate thread waits for the result before
moving on to the next thing.  
 

>
> Thanks so much for creating this.


Oh man, you're welcome, most definitely.
 

Re: [julia-users] ANN node-julia 1.0.0

[Test This]

I meant the "node process" not "mode process". Blaming the autocorrect in 
iPad . . .

On Saturday, January 17, 2015 at 10:53:36 AM UTC-5, Test This wrote:
>
> Some basic questions: 
>
> We know that Node is blocking on cpu intensive tasks. If I use the async 
> option, are the calculations run separately. That is does it allow the mode 
> process to continue with its event loop. If the answer is yes, what happens 
> when julia is busy with a previous calculation and a new one is passed to 
> it? 
>
> Thanks so much for creating this. 

Re: [julia-users] ANN node-julia 1.0.0

[Test This]

Some basic questions:

We know that Node is blocking on cpu intensive tasks. If I use the async 
option, are the calculations run separately. That is does it allow the mode 
process to continue with its event loop. If the answer is yes, what happens 
when julia is busy with a previous calculation and a new one is passed to it?

Thanks so much for creating this. 

Re: [julia-users] ANN node-julia 1.0.0

[Jeff Waller]

On Friday, January 16, 2015 at 4:26:30 PM UTC-5, Stefan Karpinski wrote:
>
> This is super cool. I wonder if it wouldn't be possible allow Julia to 
> operate on JavaScript typed arrays in-place?
>

Hmm, maybe!  With some caveats first the good news.  

Here's where the Javascript array buffer is obtained 

Here's the relevant part of the copy from Javascript to C++: it's ptr to 
ptr 
And here it is again from C++ to julia a buffer handoff 


Caveats.
this is all happening in separate threads
v8 has its own memory management, however, ArrayBuffers can be neutered 

node heap size is 2G, having but maybe neutering means it no longer takes 
part in the calculations
Javascript Typed arrays are 1D only.
there's  implicit row-major to column major transformation going on

So multidimensional stuff might be a pain, but vector transfer of 
ownership?   yea probably.

Re: [julia-users] ANN node-julia 1.0.0

[Jeff Waller]

On Friday, January 16, 2015 at 4:20:26 PM UTC-5, Kevin Squire wrote:
>
> Hi Jeff, can you share a link?
>

Hi Kevin, sure here's 2  npm  and 
Documentation 
 

Re: [julia-users] ANN node-julia 1.0.0

[Stefan Karpinski]

This is super cool. I wonder if it wouldn't be possible allow Julia to
operate on JavaScript typed arrays in-place?

On Fri, Jan 16, 2015 at 4:20 PM, Kevin Squire 
wrote:

> Hi Jeff, can you share a link?
>
> Cheers, Kevin
>
> On Fri, Jan 16, 2015 at 1:06 PM, Jeff Waller  wrote:
>
>> So I'm happy to announce version 1.0.0 of node-julia, a Julia engine
>> embedded in node,
>> and io.js now too. It's been a pretty long road and I owe many people
>> (perhaps reading
>> this now) a lot.  I've said many times (maybe not on this forum) that
>> enabling people is
>> the important part and I hope this tool does that.
>>
>> Some of the new features supported since my first update here (in Sept).
>>
>> * both asynchronous and synchronous processing  < uses libuv
>> (inside joke)
>> * use of Javascript typed arrays were possible
>> * Julia composites in JavaScript as (the opaque) JRef
>> * functionalized Scripts
>>
>> All those keywords are probably not that interesting, but I can share
>> this which might be.  I've
>> done some early testing on simple matrix multiplications (will blog, but
>> not done), and it turns
>> out it's actually faster to copy the array from JavaScript into the Julia
>> engine multiply and then
>> copy the result back out than to use JavaScript directly for most
>> matrices (maybe not 3x3).
>> And when compared to the other popular linear algebra packages for node,
>> Julia-within-node
>> can be a lot faster -- sometimes 1000x faster.
>>
>
>

Re: [julia-users] ANN node-julia 1.0.0

[Kevin Squire]

Hi Jeff, can you share a link?

Cheers, Kevin

On Fri, Jan 16, 2015 at 1:06 PM, Jeff Waller  wrote:

> So I'm happy to announce version 1.0.0 of node-julia, a Julia engine
> embedded in node,
> and io.js now too. It's been a pretty long road and I owe many people
> (perhaps reading
> this now) a lot.  I've said many times (maybe not on this forum) that
> enabling people is
> the important part and I hope this tool does that.
>
> Some of the new features supported since my first update here (in Sept).
>
> * both asynchronous and synchronous processing  < uses libuv
> (inside joke)
> * use of Javascript typed arrays were possible
> * Julia composites in JavaScript as (the opaque) JRef
> * functionalized Scripts
>
> All those keywords are probably not that interesting, but I can share this
> which might be.  I've
> done some early testing on simple matrix multiplications (will blog, but
> not done), and it turns
> out it's actually faster to copy the array from JavaScript into the Julia
> engine multiply and then
> copy the result back out than to use JavaScript directly for most matrices
> (maybe not 3x3).
> And when compared to the other popular linear algebra packages for node,
> Julia-within-node
> can be a lot faster -- sometimes 1000x faster.
>

[julia-users] ANN node-julia 1.0.0

[Jeff Waller]

So I'm happy to announce version 1.0.0 of node-julia, a Julia engine 
embedded in node,
and io.js now too. It's been a pretty long road and I owe many people 
(perhaps reading
this now) a lot.  I've said many times (maybe not on this forum) that 
enabling people is
the important part and I hope this tool does that.

Some of the new features supported since my first update here (in Sept).

* both asynchronous and synchronous processing  < uses libuv 
(inside joke)
* use of Javascript typed arrays were possible
* Julia composites in JavaScript as (the opaque) JRef
* functionalized Scripts

All those keywords are probably not that interesting, but I can share this 
which might be.  I've
done some early testing on simple matrix multiplications (will blog, but 
not done), and it turns
out it's actually faster to copy the array from JavaScript into the Julia 
engine multiply and then
copy the result back out than to use JavaScript directly for most matrices 
(maybe not 3x3).  
And when compared to the other popular linear algebra packages for node, 
Julia-within-node
can be a lot faster -- sometimes 1000x faster.

[julia-users] ANN: node-julia

[Jeff Waller]

It's usable enough now, so it's time to announce this.  node-julia 
 is a portal from node.js to Julia. 
 It's purpose is partly to allow the node-HTTP-javascript guys access to 
the excellence that is Julia, and visa versa the Julia guys access to the 
HTTP excellence that is node.js, but more fundamentally to allow the these 
two groups to help and interact with each other.  Well at least thats my 
hope.

It's definitely a work in progress, see the README 
 for the 
limitations, and of course my previous message here 
 about the 
problem I'm currently dealing with on Linux.

If you are familiar with node.js already it can be obtained via npm here