Re: [julia-users] Re: What are the "strengths" of the Julia "ecosystem" (e.g. for bio and quants/finance)? This is not a very specific question (while sub-questions are), you've been warned..

2015-09-25 Thread Andrei Zh 
 

> From browsing issues, it looks like the HttpServer.jl performance issue 
> you referenced below should be now fixed by 
> https://github.com/JuliaWeb/HttpServer.jl/pull/59.


Yes, this issue has been fixed. 

 It seems Julia can have low latency 


Yes, on my later tests latency was pretty good

(are Flask and Splay [numbers, for it and throughput] considered 
> best-in-class, at least for those language?)


Spray is pretty good, Flask is here just for comparison. I don't think 
Flask has been developed with performance in mind. 

Do you only disagree, then only, with the "high throughput" 


As you can see from results in #40, HttpServer.jl already has pretty good 
performance, just I believe Julia has no blockers not to get even more 
(comparable to best-in-class). 

I wander how difficult or stupid it is to support old APIs here in some 
> way..


In most cases it's unreasonable to keep outdated API for many reasons. 

But "probably comparable with Erlang" was not an error, taken 
> conservatively, you are probably talking about what pure benchmark numbers 
> of concurrency (only) might indicate. I believe Erlang has more (than 
> speed) currently not in Julia, while not better for non-concurrency related.


When processing large number of small requests (e.g. text messages in 
social network or requests to API), blocking network operations become a 
bottleneck. E.g. if you have blocking "get()" operation, you either need to 
waste processing time until request is complete or switch processor 
context. Both of these are quite expensive and set a limit to a maximum 
throughput. Erlang has actor-based concurrency, which means that you can 
switch between tasks without switching processor context and get high 
utilization of CPU. Scala's Spray is built on Akka, which is an 
implementation of the same actor model, and gets very good performance. 
Julia has similar concurrency model using Tasks, and that's why I'm so 
optimistic regarding future of the web frameworks in Julia. 

Even if this would work perfectly, I'm not sure (most) Julia (or Java?) 
> users would want to mix (I guess ok for truly good/critical libraries..) at 
> least for GUI/enterprise applications..


Consider, for example, Apache Spark and the whole Hadoop infrastructure. 
There's nothing even close to it in non-JVM world, and that's why other 
languages (including Python and R) build their interfaces to JVM. 


On Friday, September 25, 2015 at 4:39:51 PM UTC+3, Jameson wrote:
>
> From browsing issues, it looks like the HttpServer.jl performance issue 
> you referenced below should be now fixed by 
> https://github.com/JuliaWeb/HttpServer.jl/pull/59.
>
> On Fri, Sep 25, 2015 at 5:47 AM Páll Haraldsson  > wrote:
>
>> On Thursday, September 24, 2015 at 11:07:56 PM UTC, Andrei Zh wrote:
>>>
>>>  
>>>
 I find this statement highly surprising.. wander if you meant to 
 reverse this.. My quant friend who had worked for years in Python had 
 trouble parallelizing Python code (may be resolved now..). I'm not 
 familiar 
 with R, but Python has the GIL and associated problems. I also thought 
 Erlang was best-in-class (for concurrent, not "parallel").

>>>
>>> Oops, you are right, it's exactly the opposite. Consider it a result of 
>>> quick answer between 2 meetings. Julia's capabilities are much better for 
>>> concurrent and parallel programming than these in Python or R. 
>>>
>>
>> But "probably comparable with Erlang" was not an error, taken 
>> conservatively, you are probably talking about what pure benchmark numbers 
>> of concurrency (only) might indicate. I believe Erlang has more (than 
>> speed) currently not in Julia, while not better for non-concurrency related.
>>
>>
>>> For those in the Java/Scala world, I'm less sure about reusing that, I 
 know you can with JavaCall.jl, but understand there are bugs and 
 limitations to it.
>>>
>>>
>>> Unfortunately, that's true. @aviks has made a great work connecting 
>>> Julia to JVMs via Java Native Interface, but as far as I can see, JNI is 
>>> shitty by itself, and it's very hard to to make integration between Julia 
>>> and Java really stable.
>>>
>>
>> Even if this would work perfectly, I'm not sure (most) Julia (or Java?) 
>> users would want to mix (I guess ok for truly good/critical libraries..) at 
>> least for GUI/enterprise applications.. At least for me, it seems involving 
>> a JVM or any VM, is outdated.. Yes, Julia has LLVM, but it's not the same. 
>>
>>>
>>> "HttpServer.jl, has low latency 0.5 ms and high throughput (latency on 
 the same order of Python's Flask and Scala's Spray mature frameworks, and 
 throughput also comparable[82])." 
>>>
>>>
>>> That's funny, because specified reference leads to an issue on 
>>> performance that was opened by me and incorrectly interpreted by the author 
>>> of Wikipedia page. At that test Julia outperformed Flask, but was still 3-6 
>>> times slower than Spray.

Re: [julia-users] Re: What are the "strengths" of the Julia "ecosystem" (e.g. for bio and quants/finance)? This is not a very specific question (while sub-questions are), you've been warned..

2015-09-25 Thread Jameson Nash 
>From browsing issues, it looks like the HttpServer.jl performance issue you
referenced below should be now fixed by
https://github.com/JuliaWeb/HttpServer.jl/pull/59.

On Fri, Sep 25, 2015 at 5:47 AM Páll Haraldsson 
wrote:

> On Thursday, September 24, 2015 at 11:07:56 PM UTC, Andrei Zh wrote:
>>
>>
>>
>>> I find this statement highly surprising.. wander if you meant to reverse
>>> this.. My quant friend who had worked for years in Python had trouble
>>> parallelizing Python code (may be resolved now..). I'm not familiar with R,
>>> but Python has the GIL and associated problems. I also thought Erlang was
>>> best-in-class (for concurrent, not "parallel").
>>>
>>
>> Oops, you are right, it's exactly the opposite. Consider it a result of
>> quick answer between 2 meetings. Julia's capabilities are much better for
>> concurrent and parallel programming than these in Python or R.
>>
>
> But "probably comparable with Erlang" was not an error, taken
> conservatively, you are probably talking about what pure benchmark numbers
> of concurrency (only) might indicate. I believe Erlang has more (than
> speed) currently not in Julia, while not better for non-concurrency related.
>
>
>> For those in the Java/Scala world, I'm less sure about reusing that, I
>>> know you can with JavaCall.jl, but understand there are bugs and
>>> limitations to it.
>>
>>
>> Unfortunately, that's true. @aviks has made a great work connecting Julia
>> to JVMs via Java Native Interface, but as far as I can see, JNI is shitty
>> by itself, and it's very hard to to make integration between Julia and Java
>> really stable.
>>
>
> Even if this would work perfectly, I'm not sure (most) Julia (or Java?)
> users would want to mix (I guess ok for truly good/critical libraries..) at
> least for GUI/enterprise applications.. At least for me, it seems involving
> a JVM or any VM, is outdated.. Yes, Julia has LLVM, but it's not the same.
>
>>
>> "HttpServer.jl, has low latency 0.5 ms and high throughput (latency on
>>> the same order of Python's Flask and Scala's Spray mature frameworks, and
>>> throughput also comparable[82])."
>>
>>
>> That's funny, because specified reference leads to an issue on
>> performance that was opened by me and incorrectly interpreted by the author
>> of Wikipedia page. At that test Julia outperformed Flask, but was still 3-6
>> times slower than Spray.
>>
>
>
> Good throughput, but very high latency
> https://github.com/JuliaWeb/HttpServer.jl/issues/48
> "At the same time, with even very naive test in Python (using requests
> library) I was able to get 1.3ms, and with Julia (Requests.jl) it was even
> lower - about 0.5ms per request."
>
> Then the issue was closed. It seems Julia can have low latency (are Flask
> and Splay [numbers, for it and throughput] considered best-in-class, at
> least for those language?). Splay has "< 1 ms" so unclear how much better,
> it may be. It seems you do not dispute the low latency claim of Julia [on
> the WP page] (any more)?
>
> Do you only disagree, then only, with the "high throughput" of
> Julia/[HttpServer.jl] (and would only change high->good)? [With 1000
> threads] Julia beats Flask, "comparable", while a third of Splay. Maybe
> within an order of magnitude is not comparable.. Not sure about the 6 times
> slower number.
>
> I haven't looked to closely at Requests.jl, so I do not know the
> differences between it and HttpServer.jl are other libraries. It really
> only matters that one library gets good throughput and then you can use
> that one?
>
>
>
>> Seems like I need to repeat my tests to get latest results.
>>
>
> It would be interesting to know if the numbers have improved..
>
>
>> I didn't check if this works the same, or just similarly?
>>
>>
>> They have totally different APIs and approaches in general. The main
>> point is that if you want to keep it working with updated version of Julia
>> and libraries, you have to adapt web application too. And it's really not
>> funny to come back to a code written half a year ago just because libraries
>> it used are now deprecated.
>>
>
> Yes, see your point.. I guess the stuff still works it it ever did.. I see
> for plots that there is a generic package that is supposed to hide the
> differences of six (by now) different plotting packages.. You say the API
> is totally different, I wander how difficult or stupid it is to support old
> APIs here in some way..
>

Re: [julia-users] Re: What are the "strengths" of the Julia "ecosystem" (e.g. for bio and quants/finance)? This is not a very specific question (while sub-questions are), you've been warned..

2015-09-24 Thread Jonathan Malmaud 
I agree with all that - there isn't a web framework for Julia that is at
the level of something Django or RoR. It seems totally reasonable to use
those mature tools for the frontend of your webapp, which could in term
communicate with a Julia backend.

I just meant that some of the lower levels of the stack, like
implementation for the full HTTP spec, proper handling of unicode and
binary data at the HTTP level, solid SSL support, etc is good now.
HttpServer.jl includes examples of setting up HTTPS.

On Thu, Sep 24, 2015 at 9:16 AM, Andrei Zh 
wrote:

> It's great that webstack has got many improvements recently, but as far as
> I can see even more job is still to be done. E.g. for me 2 kinds of web
> apps that I need most often are web UIs and high-performance web services.
> I'm not sure about performance (last time I tested HttpServer it was quite
> moderate, but maybe it has changed already), but for web UIs I miss at
> least following features (taking Mux as the basis):
>
>  - template engine: Mustache.jl can probably be used, but so far Google
> knows about zero common occurrences of Mustache.jl and Mux.jl except for
> very general lists of frameworks
>  - serving static files: possible to do in pure Julia, of course, but it's
> another several hours to implement
>  - sessions: middleware is exactly for this kind of things, but again,
> it's better to have it out of the box, than write everything yourself
>  - authentication and security: how to set up HTTPS? how to restrict
> access to certain pages?
>  - stability: I've just knew that Morsel.jl is now deprecated, if I had
> applications using it, I would need to migrate them now, and I'm really not
> sure Mux.jl won't be deprecated during next year too
>
> This means that if you want to provide users with a nice interface to your
> Julia application, you should either spend a couple of days adding missing
> stuff (and probably not in way suitable for other users) or just use, say,
> Python and do the job in a couple of hours.
>
>
> On Thursday, September 24, 2015 at 2:35:57 AM UTC+3, Jonathan Malmaud
> wrote:
>>
>> The webstack has seen considerable improvement lately. Mux is the most
>> mature and supported webapp framework at this point.
>>
>> On Wednesday, September 23, 2015 at 4:58:01 PM UTC-4, Andrei Zh wrote:
>>>
>>> If you are looking for a best in the class libraries, you probably won't
>>> find many. This is implied by a simple fact that most such libraries had
>>> already been created in other languages by the time Julia was born.
>>> However, if you want something comparable to such best libraries, then I
>>> would stress the following areas (from my experience and highly
>>> subjectively, of course):
>>>
>>>  * image processing (e.g. Images.jl, ImageView.jl), which still changes,
>>> but has quite impressive functionality already
>>>  * deep learning (e.g. Mocha.jl, Strada.jl, Boltzmann.jl) - fast, full
>>> functional, easy to use and modify libraries (compare to frameworks in C++
>>> or Theano, for example)
>>>  * concurrent, parallel and distributed programming (core Julia) - far
>>> behind Python or R, probably comparable with Erlang
>>>  * GPU computing (see JuliaGPU organization) - pretty convenient,
>>> especially combined with Julia's compilation to native code
>>>  * symbolic and metaprogramming (macros, Calculus.jl) -  like Lisp with
>>> infix notation or SymPy, built in the language
>>>
>>> I also expect that Julia will become more popular with development of
>>> new areas for which there are no good libraries at all and Julia may become
>>> perfect solution. At the same time, to keep people involved, we not only
>>> need to add more strengths, but also remove weaknesses. And Julia's web
>>> stack seems to be one of the biggest weaknesses, so if you are interested
>>> and wish to contribute, please, do it.
>>>
>>>
>>> On Wednesday, September 23, 2015 at 5:03:02 PM UTC+3, Páll Haraldsson
>>> wrote:

 On Wednesday, September 23, 2015 at 12:35:47 PM UTC, Randy Zwitch wrote:
>
> Julia is as capable as any of the languages you have mentioned as far
> as I'm concerned. When I read "people want to get work done", I read that
> as "people want SOMEONE ELSE to do the work".
>

 And you would be absolutely right. I tried to phrase the question in a
 positive way with "and help needed?" [For me, that would be mostly non-math
 stuff*, and I've submitted some trivial/beginner.. fixes.]  I'm ok with
 that as I am just tinkering. Imagine Julia had no libraries, as at first
 then I would have been as exited about the language. It is a language that
 makes me think differently and try new paradigms I haven't tried before
 (multiple dispatch).

 I might have tried to build a website (and web server from scratch).
 Some people do not want to be early adopters. I can understand that. I'm
 not so sure you would be by now. I'm asking about the

Re: [julia-users] Re: What are the "strengths" of the Julia "ecosystem" (e.g. for bio and quants/finance)? This is not a very specific question (while sub-questions are), you've been warned..

2015-09-24 Thread Páll Haraldsson 
On fim 24.sep 2015 13:25, Jonathan Malmaud wrote: 

I agree with all that - there isn't a web framework for Julia that is at 
the level of something Django or RoR. It seems totally reasonable to use 
those mature tools for the frontend of your webapp, which could in term 
communicate with a Julia backend. 


Sounds good. I wasn't really sure if using with Python/Django was 
recommended. Python is well supported with PyCall.jl but frameworks call 
you (the "Hollywood principle": "don't call use, we'll call you") unlike 
regular libraries, so I guess you have to mess with callbacks. 

OR you use use Python as your main language and call Julia from it with: 

https://github.com/JuliaLang/pyjulia 

[that uses PyCall indirectly, that you have to install first] 


I wander which is preferred (is pyjulia no longer buggy/inferior to PyCall? 
Much [less] used?) and if you use the second option you end up not using 
Julia much and might never migrate fully to Julia.. [That could be ok 
though.] 


About Ruby on Rails you can use that and call Julia, with RoR_julia_eg 
(haven't heard of calling in the other direction). It may not be as good as 
using Django, as Python will work in the same address space as Julia and 
allows zero-copy. It however may not be too important as: are (individual) 
web page that speed-critical, (and would have to share that much data 
between the languages)? 

I just meant that some of the lower levels of the stack, like 
implementation for the full HTTP spec, proper handling of unicode and 
binary data at the HTTP level, solid SSL support, etc is good now. 
HttpServer.jl includes examples of setting up HTTPS. 

On Thu, Sep 24, 2015 at 9:16 AM, Andrei Zh wrote: 

It's great that webstack has got many improvements recently, but as 
far as I can see even more job is still to be done. E.g. for me 2 
kinds of web apps that I need most often are web UIs and 
high-performance web services. I'm not sure about performance (last 
time I tested HttpServer it was quite moderate, 


https://en.wikipedia.org/wiki/Julia_(programming_language)#For_web_use 

"HttpServer.jl, has low latency 0.5 ms and high throughput (latency on the 
same order of Python's Flask and Scala's Spray mature frameworks, and 
throughput also comparable[82])." 

I'm not sure if Julia's web page is missing something with the most 
important web (or other..) libraries. Possibly session management, if it 
exists: 

but maybe it has 
changed already), but for web UIs I miss at least following features 
(taking Mux as the basis): 

  - template engine: Mustache.jl can probably be used, but so far 
Google knows about zero common occurrences of Mustache.jl and Mux.jl 
except for very general lists of frameworks 
  - serving static files: possible to do in pure Julia, of course, 
but it's another several hours to implement 


Wouldn't that be kind of trivial, since Julia already acts as a web server, 
to load files from disk and forward? 

  - sessions: middleware is exactly for this kind of things, but 
again, it's better to have it out of the box, than write everything 
yourself 


I thought I had seen something related to sessions, but may misremember, 
maybe it was to save REPL sessions.. or related to web/IJulia (only things 
I find now..). 

  - authentication and security: how to set up HTTPS? how to 
restrict access to certain pages? 


mbedTLS.jl? A good replacement for GnuTLS.jl that seem on the way out. 

  - stability: I've just knew that Morsel.jl is now deprecated, if I 
had applications using it, I would need to migrate them now, and I'm 
really not sure Mux.jl won't be deprecated during next year too 


I didn't check if this works the same, or just similarly? Or even not 
that..? Morsel was Sinatra-like and that seems to be a hot thing and often 
the way to go. Python and others reimplemented Ruby's Sinatra micro 
framework with Flask. I'm not sure Julia and Flask would make sense as by 
being "micro", this style of framework is already "complete" in Julia?] 


This means that if you want to provide users with a nice interface 
to your Julia application, you should either spend a couple of days 
adding missing stuff (and probably not in way suitable for other 
users) or just use, say, Python and do the job in a couple of hours. 


-- 
Palli.

P.S.: I replied by e-mail.. And got "Delivery Status Notification 
(Failure)". "You might have spelled or formatted the group name 
incorrectly." Does this work for others from Thunderbird? E-mail works for 
GitHub, that is nice.. and I do not know about Discourse, the proposed new 
forum software..

RE: [julia-users] Re: What are the "strengths" of the Julia "ecosystem" (e.g. for bio and quants/finance)? This is not a very specific question (while sub-questions are), you've been warned..

2015-09-23 Thread David Anthoff 
Optimization should definitely be on this list. The JuMP package is just 
phenomenal, in my mind a much better overall experience for many problems than 
any existing alternative.

 

From: julia-users@googlegroups.com [mailto:julia-users@googlegroups.com] On 
Behalf Of Andrei Zh
Sent: Wednesday, September 23, 2015 1:58 PM
To: julia-users 
Subject: [julia-users] Re: What are the "strengths" of the Julia "ecosystem" 
(e.g. for bio and quants/finance)? This is not a very specific question (while 
sub-questions are), you've been warned..

 

If you are looking for a best in the class libraries, you probably won't find 
many. This is implied by a simple fact that most such libraries had already 
been created in other languages by the time Julia was born. However, if you 
want something comparable to such best libraries, then I would stress the 
following areas (from my experience and highly subjectively, of course):

 

 * image processing (e.g. Images.jl, ImageView.jl), which still changes, but 
has quite impressive functionality already

 * deep learning (e.g. Mocha.jl, Strada.jl, Boltzmann.jl) - fast, full 
functional, easy to use and modify libraries (compare to frameworks in C++ or 
Theano, for example)

 * concurrent, parallel and distributed programming (core Julia) - far behind 
Python or R, probably comparable with Erlang

 * GPU computing (see JuliaGPU organization) - pretty convenient, especially 
combined with Julia's compilation to native code

 * symbolic and metaprogramming (macros, Calculus.jl) -  like Lisp with infix 
notation or SymPy, built in the language

 

I also expect that Julia will become more popular with development of new areas 
for which there are no good libraries at all and Julia may become perfect 
solution. At the same time, to keep people involved, we not only need to add 
more strengths, but also remove weaknesses. And Julia's web stack seems to be 
one of the biggest weaknesses, so if you are interested and wish to contribute, 
please, do it. 

 


On Wednesday, September 23, 2015 at 5:03:02 PM UTC+3, Páll Haraldsson wrote:

On Wednesday, September 23, 2015 at 12:35:47 PM UTC, Randy Zwitch wrote:

Julia is as capable as any of the languages you have mentioned as far as I'm 
concerned. When I read "people want to get work done", I read that as "people 
want SOMEONE ELSE to do the work".

 

And you would be absolutely right. I tried to phrase the question in a positive 
way with "and help needed?" [For me, that would be mostly non-math stuff*, and 
I've submitted some trivial/beginner.. fixes.]  I'm ok with that as I am just 
tinkering. Imagine Julia had no libraries, as at first then I would have been 
as exited about the language. It is a language that makes me think differently 
and try new paradigms I haven't tried before (multiple dispatch).

 

I might have tried to build a website (and web server from scratch). Some 
people do not want to be early adopters. I can understand that. I'm not so sure 
you would be by now. I'm asking about the "ecosystem" not the language per se. 
I know about JuliaQuant, BioJulia, GPU stuff in Julia JuliaWeb etc. I am so 
grateful for what has already been done with the language - and the libraries 
from what I can see. If there where my fields, I think I would jump on Julia 
right now.

 

I'm not sure why people are reluctant, I want to tell them you do not only have 
basic building blocks (linear algebra/matrix multiplication, FFT etc. stuff in 
Base), but also these libraries that (mostly) work, and if not you can help 
fix/contribute. I do not want to oversell Julia, so I keep quiet (mostly) about 
stuff I'm ignorant about..

 

 

* I knew about say, Morsel (Sinatra-like), then Mux is recommended over it. I'm 
not sure, it seems to be a replacement/also Sinatra-style. I've never used 
"full web frameworks". PHP isn't my favorite language and while I'm sure Python 
(or Ruby) is nice for web stuff I'm willing to use Julia even if there is 
(short term) pain/learning experience.. I would want to be able to do what is 
needed in pure Julia. Even knowing about future direction is helpful, there 
might be some duplication of effort and you might end up fixing the wrong 
package.. A list of packages to use/focus on for helping with would be helpful 
in this and the other two areas.

 

> Julia probably isn't the place for them now

 

Are at least some of the packages ready and used in production already?

 

-- 

Palli.