Re: [julia-users] Re: How to efficiently count the number of lines in a file
Thanks! I did not realize that you could use readall() like that. I had been struggling to get the output out of run(). And thanks for the benchmark. Cheers, Daniel. On Wednesday, 19 August 2015 13:38:04 UTC+2, René Donner wrote: You can use parse(Int,split(readall(`wc -l test.txt`))[1]) for that. FYI, a small benchmark of showed countlines to be 3x faster than run(wc..) and 4x faster than the mmap approach. The code for countlines is interesting, nice example for highly efficient Julia code: Am 19.08.2015 um 13:35 schrieb Daniel Carrera dcar...@gmail.com javascript:: I tried running `wc` directly, but the run() command does not return the output of `wc`. On Wednesday, 19 August 2015 12:22:41 UTC+2, Avik Sengupta wrote: You can't get much better than unix command line tools for this sort of thing. Any reason you can't use `wc` directly? I'd do this using run(`wc`...) from within Julia. Regards - Avik On Wednesday, 19 August 2015 11:10:40 UTC+1, Daniel Carrera wrote: Hello, I need to count the number of lines in a large number of UNIX text files, most of which are fairly large. I need help coming up with an efficient implementation of a line-count program. A naive implementation like length(readlines(foo.txt)) is very slow (notice how this loads the whole file into memory just to count newlines). I imagine that it should be possible to count the number of newline characters quickly, like what the wc command does, but I can't figure out how. Does anyone have any ideas? Thanks for the help. Daniel.
Re: [julia-users] Benchmarking Julia HttpServer
Can you include the code you’re using for the HTTP server and for testing (maybe link to a gist if the code is larger than email-appropriate)? Also make sure you’ve checked out the performance tips FAQ: http://docs.julialang.org/en/release-0.3/manual/performance-tips/ http://docs.julialang.org/en/release-0.3/manual/performance-tips/ The profiler is also useful for tracking down performance problems: http://docs.julialang.org/en/release-0.3/manual/profile/ http://docs.julialang.org/en/release-0.3/manual/profile/ -s On Aug 19, 2015, at 6:42 AM, nikolai.mar...@icloud.com wrote: Hi there! I was trying to do some basic benchmark of Julia's HttpServer, however I have some strange behaviour. When I ask ab to do 20k requests in a single thread, at around request #16k Julia starts accepting connections quite slowly (around 20s). If I request two threads, then ab fails with timeout on connecting (60s). I thought that the problem may be with the GC so I disabled it, but that didn't help. I would be grateful for the tips on how to look into this more closely. (Julia 0.3.10 via brew on os x)
Re: [julia-users] Benchmarking Julia HttpServer
My code is really simple: using HttpServer gc_disable() const Z = HELLO WORLD http = HttpHandler() do req::Request, res::Response Response(Z) end server = Server( http ) run( server, 8000 ) And the testing is ab -n2 http://127.0.0.1:8000/asdfasd I also had to patch the HttpServer.jl to close the connection, since otherwise ab doesn't go past the first request. Thank you for the links. I have read the performance tips briefly and have started with the profiling doc. On Wednesday, August 19, 2015 at 4:35:09 PM UTC+3, Spencer Russell wrote: Can you include the code you’re using for the HTTP server and for testing (maybe link to a gist if the code is larger than email-appropriate)? Also make sure you’ve checked out the performance tips FAQ: http://docs.julialang.org/en/release-0.3/manual/performance-tips/ The profiler is also useful for tracking down performance problems: http://docs.julialang.org/en/release-0.3/manual/profile/ -s On Aug 19, 2015, at 6:42 AM, nikolai...@icloud.com javascript: wrote: Hi there! I was trying to do some basic benchmark of Julia's HttpServer, however I have some strange behaviour. When I ask ab to do 20k requests in a single thread, at around request #16k Julia starts accepting connections quite slowly (around 20s). If I request two threads, then ab fails with timeout on connecting (60s). I thought that the problem may be with the GC so I disabled it, but that didn't help. I would be grateful for the tips on how to look into this more closely. (Julia 0.3.10 via brew on os x)
[julia-users] precompile EACCES error
I can't seem to load DataFrames due to a precompilation issue ~~~ julia using DataFrames INFO: Precompiling module DataFrames... ERROR: could not spawn `/Users/joosep/Documents/julia/usr/lib/julia -Cnative -J/Users/joosep/Documents/julia/usr/lib/julia/sys.dylib --output-ji /Users/joosep/.julia/lib/v0.4/DataFrames.ji --output-incremental=yes --startup-file=no --history-file=no --eval 'while !eof(STDIN) eval(Main, deserialize(STDIN)) end '`: permission denied (EACCES) in _jl_spawn at process.jl:226 ~~~ This is a clean build, but maybe I have missed something - has anyone else seen this? I'm using ~~~ - DataFrames0.6.9 julia versioninfo() Julia Version 0.4.0-dev+6723 Commit b59b0dd* (2015-08-14 08:12 UTC) Platform Info: System: Darwin (x86_64-apple-darwin14.3.0) CPU: Intel(R) Core(TM) i5-4278U CPU @ 2.60GHz WORD_SIZE: 64 BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell) LAPACK: libopenblas LIBM: libopenlibm LLVM: libLLVM-3.3 ~~~
Re: [julia-users] Constructor or convert(), Upper case or lower case
On Tuesday, August 18, 2015 at 12:02:36 PM UTC-4, Matt Bauman wrote: `float` is an interesting case as it's regularly used to generically mean: convert to a floating point number *OR* a complex number with floating point components, so that's why it's still here but `int` isn't. float(x) means give me the floating-point version of x. It works on any kind of number, but also on arrays and potentially other containers.
Re: [julia-users] Re: How to efficiently count the number of lines in a file
You can use parse(Int,split(readall(`wc -l test.txt`))[1]) for that. FYI, a small benchmark of showed countlines to be 3x faster than run(wc..) and 4x faster than the mmap approach. The code for countlines is interesting, nice example for highly efficient Julia code: Am 19.08.2015 um 13:35 schrieb Daniel Carrera dcarr...@gmail.com: I tried running `wc` directly, but the run() command does not return the output of `wc`. On Wednesday, 19 August 2015 12:22:41 UTC+2, Avik Sengupta wrote: You can't get much better than unix command line tools for this sort of thing. Any reason you can't use `wc` directly? I'd do this using run(`wc`...) from within Julia. Regards - Avik On Wednesday, 19 August 2015 11:10:40 UTC+1, Daniel Carrera wrote: Hello, I need to count the number of lines in a large number of UNIX text files, most of which are fairly large. I need help coming up with an efficient implementation of a line-count program. A naive implementation like length(readlines(foo.txt)) is very slow (notice how this loads the whole file into memory just to count newlines). I imagine that it should be possible to count the number of newline characters quickly, like what the wc command does, but I can't figure out how. Does anyone have any ideas? Thanks for the help. Daniel.
Re: [julia-users] Re: How to efficiently count the number of lines in a file
On Wed, Aug 19, 2015 at 7:43 AM, Daniel Carrera dcarr...@gmail.com wrote: Thanks! I didn't know about countlines(). Interestingly, countlines() does not seem to include blank lines. That's not a problem for me, but it's good to be aware of: https://github.com/JuliaLang/julia/pull/11947 -- $ cat foo.txt Hello world. $ cat bar.txt Hello World. $ julia ... julia run(`wc -l foo.txt`) 2 foo.txt julia run(`wc -l bar.txt`) 4 bar.txt julia countlines(foo.txt) 2 julia countlines(bar.txt) 2 -- Thanks again. Daniel. On Wednesday, 19 August 2015 13:13:43 UTC+2, Keith Campbell wrote: You could try countlines(). Also, you likely want eachline() rather than readlines(). eachline() will iterate through the file for you. cheers, Keith
[julia-users] Re: How to efficiently count the number of lines in a file
Thanks! I didn't know about countlines(). Interestingly, countlines() does not seem to include blank lines. That's not a problem for me, but it's good to be aware of: -- $ cat foo.txt Hello world. $ cat bar.txt Hello World. $ julia ... julia run(`wc -l foo.txt`) 2 foo.txt julia run(`wc -l bar.txt`) 4 bar.txt julia countlines(foo.txt) 2 julia countlines(bar.txt) 2 -- Thanks again. Daniel. On Wednesday, 19 August 2015 13:13:43 UTC+2, Keith Campbell wrote: You could try countlines(). Also, you likely want eachline() rather than readlines(). eachline() will iterate through the file for you. cheers, Keith
Re: [julia-users] Benchmarking Julia HttpServer
I've got similar results some time ago [1], but stopped investigating because of lack of time. If you could find out the reason why different tools result in different outcomes, it would really push Julia web stack to production level. [1]: https://github.com/JuliaWeb/HttpServer.jl/issues/48 On Wednesday, August 19, 2015 at 4:35:09 PM UTC+3, Spencer Russell wrote: Can you include the code you’re using for the HTTP server and for testing (maybe link to a gist if the code is larger than email-appropriate)? Also make sure you’ve checked out the performance tips FAQ: http://docs.julialang.org/en/release-0.3/manual/performance-tips/ The profiler is also useful for tracking down performance problems: http://docs.julialang.org/en/release-0.3/manual/profile/ -s On Aug 19, 2015, at 6:42 AM, nikolai...@icloud.com javascript: wrote: Hi there! I was trying to do some basic benchmark of Julia's HttpServer, however I have some strange behaviour. When I ask ab to do 20k requests in a single thread, at around request #16k Julia starts accepting connections quite slowly (around 20s). If I request two threads, then ab fails with timeout on connecting (60s). I thought that the problem may be with the GC so I disabled it, but that didn't help. I would be grateful for the tips on how to look into this more closely. (Julia 0.3.10 via brew on os x)
[julia-users] Re: How to efficiently count the number of lines in a file
I tried running `wc` directly, but the run() command does not return the output of `wc`. On Wednesday, 19 August 2015 12:22:41 UTC+2, Avik Sengupta wrote: You can't get much better than unix command line tools for this sort of thing. Any reason you can't use `wc` directly? I'd do this using run(`wc`...) from within Julia. Regards - Avik On Wednesday, 19 August 2015 11:10:40 UTC+1, Daniel Carrera wrote: Hello, I need to count the number of lines in a large number of UNIX text files, most of which are fairly large. I need help coming up with an efficient implementation of a line-count program. A naive implementation like length(readlines(foo.txt)) is very slow (notice how this loads the whole file into memory just to count newlines). I imagine that it should be possible to count the number of newline characters quickly, like what the wc command does, but I can't figure out how. Does anyone have any ideas? Thanks for the help. Daniel.
[julia-users] Re: A cheat page for Julia
The updated version is now available. The 0.4 version is already done (and on my Bitbucket repo http://bitbucket.org/hell316/dbline/src) but I'll wait with replacing the publicly available http://tinyurl.com/JuliaLang document until Julia 0.4 is officially released.
Re: [julia-users] Re: help with try/catch?
https://github.com/JuliaLang/julia/issues/12697 On Tuesday, August 18, 2015 at 3:37:28 PM UTC-7, Yichao Yu wrote: On Aug 18, 2015 6:27 PM, Seth catc...@bromberger.com javascript: wrote: On Tuesday, August 18, 2015 at 3:16:48 PM UTC-7, Yichao Yu wrote: You cannot use import in local scope. Use eval or @eval But why do the other ones (including when the try returns nothing) work? Didn't notice that I guess its an parser bug then
[julia-users] Why does enumerate fail in parallel?
I just rewrote one of my parallel loops and was surprised by this:
list = [a, b, c]
for i in enumerate(list)
println(i)
end
(1,a)
(2,b)
(3,c)
addprocs(2)
@sync @parallel for i in enumerate(list)
println(i)
end
ERROR: `getindex` has no method matching
getindex(::Enumerate{Array{ASCIIString,1}}, ::UnitRange{Int64})
Am I doing something wrong here? Is this expected behaviour?
[julia-users] does JLD handle Int128 (saw at one time it did not)
Or, what is a good approach to saving and reloading Int128s .. -x .. +x?
Re: [julia-users] Embarrassingly parallel workload
Hi Kristoffer, sorry for the delay and thanks for the code. What I want to do is very simple: I have an expensive loop for i=1:N such that each iteration is independent and produces a large array of size M. The result of this loop is a matrix of size MxN. I have many CPU cores at my disposal and want to distribute this work. In the past I accomplished that with MPI in Python: https://github.com/juliohm/HUM/blob/master/pyhum/utils.py Whenever a process in the pool is free it consumes an iteration of the loop. What exactly the @parallel macro in Julia is doing? How can I modify the code I previously posted in this thread to achieve such effect? -Júlio
[julia-users] Re: Why does enumerate fail in parallel?
Enumerate is an iterator, you need to collect the items first:
julia @parallel for i in collect(enumerate(list))
println(i)
end
julia From worker 2: (1,a)
From worker 2: (2,b)
From worker 3: (3,c)
El miércoles, 19 de agosto de 2015, 12:17:35 (UTC-5), Nils Gudat escribió:
I just rewrote one of my parallel loops and was surprised by this:
list = [a, b, c]
for i in enumerate(list)
println(i)
end
(1,a)
(2,b)
(3,c)
addprocs(2)
@sync @parallel for i in enumerate(list)
println(i)
end
ERROR: `getindex` has no method matching
getindex(::Enumerate{Array{ASCIIString,1}}, ::UnitRange{Int64})
Am I doing something wrong here? Is this expected behaviour?
Re: [julia-users] Re: does JLD handle Int128 (saw at one time it did not)
They were split, but google hasn't quite caught up to it. And all the old issues are of course still there in the original HDF5 repository. I don't remember the status of this at all. I hate to say it, but why don't you just try it out and see if it works? If not, please file an issue--- shouldn't be hard to fix. --Tim On Wednesday, August 19, 2015 12:37:32 PM Jeffrey Sarnoff wrote: I saw a note from Tim responding to a general question about the state of JLD. At that time (I thought the date was recent, though I might have misread the year) He said everything works very well except for the 128 bit integers, which were on hold until he could verify that the their sign bit ?something about it? would behave properly under storing and retrieving. After seeing that, I did not try. I don't have the page I saw here .. but just now looking for that I was routed to HDF5.jl .. were they not split recently, --- or am I ready for some coffee? On Wednesday, August 19, 2015 at 2:45:13 PM UTC-4, Stefan Karpinski wrote: JLD is built on top of HDF5. Jeffrey: does JLD not handle Int128 out of the box? On Wed, Aug 19, 2015 at 2:43 PM, Ismael VC ismael...@gmail.com javascript: wrote: Have you tried using it with HDF5.jl? https://github.com/timholy/HDF5.jl El miércoles, 19 de agosto de 2015, 12:50:35 (UTC-5), Jeffrey Sarnoff escribió: Or, what is a good approach to saving and reloading Int128s .. -x .. +x?
Re: [julia-users] Re: does JLD handle Int128 (saw at one time it did not)
JLD is built on top of HDF5. Jeffrey: does JLD not handle Int128 out of the box? On Wed, Aug 19, 2015 at 2:43 PM, Ismael VC ismael.vc1...@gmail.com wrote: Have you tried using it with HDF5.jl? https://github.com/timholy/HDF5.jl El miércoles, 19 de agosto de 2015, 12:50:35 (UTC-5), Jeffrey Sarnoff escribió: Or, what is a good approach to saving and reloading Int128s .. -x .. +x?
[julia-users] Re: Why does enumerate fail in parallel?
Well that works but it's indeed odd, can you open a new issue for this?
El miércoles, 19 de agosto de 2015, 13:48:28 (UTC-5), Ismael VC escribió:
Enumerate is an iterator, you need to collect the items first:
julia @parallel for i in collect(enumerate(list))
println(i)
end
julia From worker 2: (1,a)
From worker 2: (2,b)
From worker 3: (3,c)
El miércoles, 19 de agosto de 2015, 12:17:35 (UTC-5), Nils Gudat escribió:
I just rewrote one of my parallel loops and was surprised by this:
list = [a, b, c]
for i in enumerate(list)
println(i)
end
(1,a)
(2,b)
(3,c)
addprocs(2)
@sync @parallel for i in enumerate(list)
println(i)
end
ERROR: `getindex` has no method matching
getindex(::Enumerate{Array{ASCIIString,1}}, ::UnitRange{Int64})
Am I doing something wrong here? Is this expected behaviour?
Re: [julia-users] Embarrassingly parallel workload
Hi Ismael, MPI is distributed memory, I'm trying to use all the cores in my single workstation with shared memory instead. Thanks for the link anyways. -Júlio
Re: [julia-users] Constructor or convert(), Upper case or lower case
I'm always available to advocate for consonant symmetry and autodydactic expressiveness on Julia's behalf. The consensual upwelling that let the broader decision deserves respect. That motive intent was to finalize one important tailoring of Julia's experience -- the experiences we each have. A few erstwhile appropriate [un]capitalizations are now sharps, and their was never a desire to elevate the ontological necessity of symbol qua symbol annon Symbol over Julia's advancement and the invitingness people find as the use this. On Wednesday, August 19, 2015 at 8:57:44 AM UTC-4, Steven G. Johnson wrote: On Tuesday, August 18, 2015 at 12:02:36 PM UTC-4, Matt Bauman wrote: `float` is an interesting case as it's regularly used to generically mean: convert to a floating point number *OR* a complex number with floating point components, so that's why it's still here but `int` isn't. float(x) means give me the floating-point version of x. It works on any kind of number, but also on arrays and potentially other containers.
Re: [julia-users] Embarrassingly parallel workload
There is an MPI wrapper for Julia, I don't know if it'll suit your needs thoug: https://github.com/JuliaParallel/MPI.jl El miércoles, 19 de agosto de 2015, 13:03:59 (UTC-5), Júlio Hoffimann escribió: Hi Kristoffer, sorry for the delay and thanks for the code. What I want to do is very simple: I have an expensive loop for i=1:N such that each iteration is independent and produces a large array of size M. The result of this loop is a matrix of size MxN. I have many CPU cores at my disposal and want to distribute this work. In the past I accomplished that with MPI in Python: https://github.com/juliohm/HUM/blob/master/pyhum/utils.py Whenever a process in the pool is free it consumes an iteration of the loop. What exactly the @parallel macro in Julia is doing? How can I modify the code I previously posted in this thread to achieve such effect? -Júlio
Re: [julia-users] Embarrassingly parallel workload
Hi Julio,
I believe this is a very common type of workload, especially in scientific
computing.
In C++ one can use OpenMP for this type of computation, in Matlab there is
parfor. From the users perspective both just work.
In Julia, I have not found an easy and convenient way to do such
computation.
The difficulties I have experienced, trying to do this with distributed
arrays and the Julia parallel operations:
1. Having to prepend @parallel before every import/require so that all
parallel processes have all definitions.
2. Working with the distributed arrays API has given me plenty of
headaches; it is more like programming with local/global contexts in OpenCL.
3. (I believe this is fixed now.) There were garbage collection issues and
crashes on Windows when using distributed arrays.
What would be very convenient is a type of OpenMP like parallelism, really
anything that can enable us to write simply
function compute(X::Vector{Float64}, theta)
n = length(X)
A = zeros(Float64, n)
@smp for i=1:n
A[i] = embarassing_parallel(X[i], theta);
end
A
end
Where @smp would correspond to #pragma omp parallel for.
I know this may be difficult to implement for a language as dynamic as
Julia, but it is hard to argue against this simplicity from the users'
point of view.
As Clang/LLVM now support OpenMP (https://clang-omp.github.io/), one
perhaps can recycle the same OpenMP runtime for such lightweight
parallelism?
Thanks,
Sebastian
On Wednesday, 19 August 2015 19:03:59 UTC+1, Júlio Hoffimann wrote:
Hi Kristoffer, sorry for the delay and thanks for the code.
What I want to do is very simple: I have an expensive loop for i=1:N such
that each iteration is independent and produces a large array of size M.
The result of this loop is a matrix of size MxN. I have many CPU cores at
my disposal and want to distribute this work.
In the past I accomplished that with MPI in Python:
https://github.com/juliohm/HUM/blob/master/pyhum/utils.py Whenever a
process in the pool is free it consumes an iteration of the loop. What
exactly the @parallel macro in Julia is doing? How can I modify the code I
previously posted in this thread to achieve such effect?
-Júlio
Re: [julia-users] Re: does JLD handle Int128 (saw at one time it did not)
I can do that. If it goes south, I note the issue. You showed me how to use it the first time, Thanks TIm. On Wednesday, August 19, 2015 at 4:20:22 PM UTC-4, Tim Holy wrote: They were split, but google hasn't quite caught up to it. And all the old issues are of course still there in the original HDF5 repository. I don't remember the status of this at all. I hate to say it, but why don't you just try it out and see if it works? If not, please file an issue--- shouldn't be hard to fix. --Tim On Wednesday, August 19, 2015 12:37:32 PM Jeffrey Sarnoff wrote: I saw a note from Tim responding to a general question about the state of JLD. At that time (I thought the date was recent, though I might have misread the year) He said everything works very well except for the 128 bit integers, which were on hold until he could verify that the their sign bit ?something about it? would behave properly under storing and retrieving. After seeing that, I did not try. I don't have the page I saw here .. but just now looking for that I was routed to HDF5.jl .. were they not split recently, --- or am I ready for some coffee? On Wednesday, August 19, 2015 at 2:45:13 PM UTC-4, Stefan Karpinski wrote: JLD is built on top of HDF5. Jeffrey: does JLD not handle Int128 out of the box? On Wed, Aug 19, 2015 at 2:43 PM, Ismael VC ismael...@gmail.com javascript: wrote: Have you tried using it with HDF5.jl? https://github.com/timholy/HDF5.jl El miércoles, 19 de agosto de 2015, 12:50:35 (UTC-5), Jeffrey Sarnoff escribió: Or, what is a good approach to saving and reloading Int128s .. -x .. +x?
[julia-users] Re: does JLD handle Int128 (saw at one time it did not)
Have you tried using it with HDF5.jl? https://github.com/timholy/HDF5.jl El miércoles, 19 de agosto de 2015, 12:50:35 (UTC-5), Jeffrey Sarnoff escribió: Or, what is a good approach to saving and reloading Int128s .. -x .. +x?
Re: [julia-users] Re: PSA: Changes to Base Docstrings
Hey all, Really good to see the enthusiasm about this change. If you're interested in helping out with the final parts of the transition, head on over to https://github.com/JuliaLang/julia/pull/12703 – Mike On Sat, 8 Aug 2015 at 13:36 Joshua Adelman joshua.adel...@gmail.com wrote: Public service announcement. It's actually on that Wikipedia page under the other section: https://en.m.wikipedia.org/wiki/Public_service_announcement Josh On Aug 8, 2015, at 8:15 AM, Uwe Fechner uwe.fechner@gmail.com wrote: Hello, what does PSA in the title mean? Wikipedia didn't help: https://en.wikipedia.org/wiki/Psa Uwe Am Sonntag, 2. August 2015 21:22:03 UTC+2 schrieb Mike Innes: Hi All, As of #11943 https://github.com/JuliaLang/julia/pull/11943, Julia uses the shiny new doc system to provide help for functions in the REPL. Previously, docstrings were stored in ReStructured text files https://github.com/JuliaLang/julia/tree/master/doc/stdlib as part of the manual, but now all docstrings are kept in Base https://github.com/JuliaLang/julia/blob/master/base/docs/helpdb.jl and the rst docs are automatically generated. What this means immediately is that any updates to Base docstrings must happen in the helpdb.jl file https://github.com/JuliaLang/julia/blob/master/base/docs/helpdb.jl, as opposed to the RST docs. Note that it's still fine to edit the main body of the manual; only function docstrings, which can be identified by the `.. function::` syntax, are affected by this change. Right now everything is in one big file, but we'd like to move the docstrings to appropriate places in Base and convert them to Markdown. We'll need some help with that, as there are a *lot* of docstrings, but when we're ready we'll send out a call for help so that people can get involved. There's still a lot of work to do, but I think this will be a big step forward for documentation in Julia. – Mike
Re: [julia-users] Re: does JLD handle Int128 (saw at one time it did not)
It works On Wednesday, August 19, 2015 at 4:44:51 PM UTC-4, Jeffrey Sarnoff wrote: I can do that. If it goes south, I note the issue. You showed me how to use it the first time, Thanks TIm. On Wednesday, August 19, 2015 at 4:20:22 PM UTC-4, Tim Holy wrote: They were split, but google hasn't quite caught up to it. And all the old issues are of course still there in the original HDF5 repository. I don't remember the status of this at all. I hate to say it, but why don't you just try it out and see if it works? If not, please file an issue--- shouldn't be hard to fix. --Tim On Wednesday, August 19, 2015 12:37:32 PM Jeffrey Sarnoff wrote: I saw a note from Tim responding to a general question about the state of JLD. At that time (I thought the date was recent, though I might have misread the year) He said everything works very well except for the 128 bit integers, which were on hold until he could verify that the their sign bit ?something about it? would behave properly under storing and retrieving. After seeing that, I did not try. I don't have the page I saw here .. but just now looking for that I was routed to HDF5.jl .. were they not split recently, --- or am I ready for some coffee? On Wednesday, August 19, 2015 at 2:45:13 PM UTC-4, Stefan Karpinski wrote: JLD is built on top of HDF5. Jeffrey: does JLD not handle Int128 out of the box? On Wed, Aug 19, 2015 at 2:43 PM, Ismael VC ismael...@gmail.com javascript: wrote: Have you tried using it with HDF5.jl? https://github.com/timholy/HDF5.jl El miércoles, 19 de agosto de 2015, 12:50:35 (UTC-5), Jeffrey Sarnoff escribió: Or, what is a good approach to saving and reloading Int128s .. -x .. +x?
[julia-users] Re: Splitting a multidimensional function
Since f1(x) requires a call to f(x), there's no way for your approach to work in Julia. You probably should define f1(x) as sqrt(x[1]) and f2(x) as 2 * x[2]. -- John On Wednesday, August 19, 2015 at 2:32:38 PM UTC-7, Nikolay Kryukov wrote: I have a problem when I try to separate the components of a multidimensional function. Example: Given the 2D function of a 2D argument: f(x) = [sqrt(x[1]), 2*x[2]] I want to split it into two separate functions which are the components of the original 2D function. I thought that the obvious solution was: f1(x) = f(x)[1] f2(x) = f(x)[2] The second function merely doubles the second component of its argument, as it should: f2([2, 3]) -- 6.0 But the functions don't turn out to be completely decoupled: let's see what happens when we do f2([-2, 3]) -- ERROR: DomainError in f2 at none:1 Even though the second function doesn't do sqrt and doesn't even depend on the first component of the argument, the first component of the original function is still checked and obviously returns an error. How do I decouple a 2D function?
Re: [julia-users] Re: Julia nightlies
Yes, those get updated very frequently, see status.julialang.org for exactly how recent they are at any time. Those only get out of date if there's an issue on the buildbots, which happens occasionally but usually doesn't take long to fix. On Wednesday, August 19, 2015 at 1:50:14 AM UTC-7, Viral Shah wrote: The generic linux binaries - I haven't checked those for a while, but it is worth checking if those are current. On Wed, Aug 19, 2015 at 8:09 AM, Yakir Gagnon 12.y...@gmail.com javascript: wrote: OK, I'm gonna git pull make then. That's the only way, right? On Wednesday, August 19, 2015 at 4:54:05 AM UTC+10, Uwe Fechner wrote: Well, removing the link from the website does not help with the travis failures: https://travis-ci.org/mlubin/NaNMath.jl/jobs/76114115 Regards: Uwe Am Dienstag, 18. August 2015 18:46:31 UTC+2 schrieb Viral Shah: Elliot has been busy lately, and this has not been maintained for a long time. We should probably remove that link from our website. -viral On Tuesday, August 18, 2015 at 9:52:17 PM UTC+5:30, Uwe Fechner wrote: Hello, the nightly Julia builds from: https://launchpad.net/~staticfloat/+archive/ubuntu/julianightlies are currently quiet outdated. As far as I understand they should be auto-generated every night from the head branch of the Julia repository. This seams to be currently broken. Any idea, why? The outdated version is causing problems with failing checks of Travis. Best regards: Uwe -- -viral
[julia-users] Splitting a multidimensional function
I have a problem when I try to separate the components of a multidimensional function. Example: Given the 2D function of a 2D argument: f(x) = [sqrt(x[1]), 2*x[2]] I want to split it into two separate functions which are the components of the original 2D function. I thought that the obvious solution was: f1(x) = f(x)[1] f2(x) = f(x)[2] The second function merely doubles the second component of its argument, as it should: f2([2, 3]) -- 6.0 But the functions don't turn out to be completely decoupled: let's see what happens when we do f2([-2, 3]) -- ERROR: DomainError in f2 at none:1 Even though the second function doesn't do sqrt and doesn't even depend on the first component of the argument, the first component of the original function is still checked and obviously returns an error. How do I decouple a 2D function?
Re: [julia-users] Embarrassingly parallel workload
Hi Sebastian, thanks for sharing your experience in parallelizing Julia code. I used OpenMP in the past too, it was very convenient in my C++ codebase. I remember of an initiative OpenACC that was trying to bring OpenMP and GPU accelerators together, I don't know the current status of it. It may be of interest to Julia devs. -Júlio
Re: [julia-users] Re: Splitting a multidimensional function
You may expect the optimizer to completely omit calculating the first element, but unfortunately for you, the negativity check before the square root will never be optimized away, since it affects the whole program execution and thus *shouldn't* be omitted from the compiled program, even though you're only using the result of the second computation. I would suggest building your functions the other way around, construct your simpler functions, and then build your composite function out of the simpler ones: f1(x) = sqrt(x) f2(x) = 2*x f(x) = [f1(x[1]), f2(x[2])] That should get you what you want. -E
[julia-users] Re: How to efficiently count the number of lines in a file
You can't get much better than unix command line tools for this sort of thing. Any reason you can't use `wc` directly? I'd do this using run(`wc`...) from within Julia. Regards - Avik On Wednesday, 19 August 2015 11:10:40 UTC+1, Daniel Carrera wrote: Hello, I need to count the number of lines in a large number of UNIX text files, most of which are fairly large. I need help coming up with an efficient implementation of a line-count program. A naive implementation like length(readlines(foo.txt)) is very slow (notice how this loads the whole file into memory just to count newlines). I imagine that it should be possible to count the number of newline characters quickly, like what the wc command does, but I can't figure out how. Does anyone have any ideas? Thanks for the help. Daniel.
Re: [julia-users] package compatibility for julia v0.3 and v0.4
On 19/08/15 02:01, Yichao Yu wrote:
On Tue, Aug 18, 2015 at 8:20 PM, Samuele Carcagno
sam.carca...@gmail.com wrote:
On 19/08/15 00:22, Stefan Karpinski wrote:
Did you try `@compat x % UInt8`?
yes, it gives the same error `ERROR: `rem` has no method matching
rem(::Int32, ::Type{Uint8})`
Sorry, didn't realize you are converting to a smaller type. Have you
tried just `trunc(Uint8, x)`? (or `@compat trunc(UInt8, x)` if you
want to use the new type name)
this works on julia v0.3
using Compat
x = @compat Int32(2^8)
@compat trunc(UInt8, x)
but fails on julia v0.4 with `ERROR: InexactError()
in trunc at int.jl:258`
Sam
On Tue, Aug 18, 2015 at 7:18 PM, Samuele Carcagno
sam.carca...@gmail.com mailto:sam.carca...@gmail.com wrote:
On 18/08/15 21:46, Yichao Yu wrote:
On Aug 18, 2015 2:54 PM, Samuele Carcagno
sam.carca...@gmail.com mailto:sam.carca...@gmail.com
mailto:sam.carca...@gmail.com mailto:sam.carca...@gmail.com
wrote:
I understand that if I release a new version of a package
foo and
put `julia 0.4` in `REQUIRE`, the package won't be updated in
julia v0.3
installations. If, after the updated package is released,
somebody tries
to install package foo from julia v0.3 what happens? Will they
still
be able to install the older version of the package?
I'm asking because I'd like to release an updated version of a
package to make it work with julia v0.4. The changes, however, are
incompatible with julia v0.3. I tried using @compat to work
around the
issues but couldn't figure out how to fix this deprecation warning
without breaking v0.3 compatibility:
`uint8(x::Integer) is deprecated, use x % UInt8 instead`
@compat UInt8(x)
thanks for the suggestion but that doesn't do what I need. The
problem is that I need to write 32-bit integers into a 24-bit file
format. This requires some bit fiddling. In julia v0.3 this worked
without deprecation warnings:
write(fid, uint8(thisSample));
write(fid, uint8(thisSample 8));
write(fid, uint8(thisSample 16));
`thisSample` is a 32 bit integer. In julia v0.4 `UInt8(x)` fails
when `x` is greater tha 2^8-1, instead the following works:
write(fid, thisSample % UInt8);
write(fid, (thisSample 8) % UInt8);
write(fid, (thisSample 16) % UInt8);
the problem is it doesn't work in julia v0.3 `x % @compat UInt8`
gives:
`ERROR: `rem` has no method matching rem(::Int32, ::Type{Uint8})`
Sam
Cheers,
Sam
[julia-users] Re: How to efficiently count the number of lines in a file
You could try countlines(). Also, you likely want eachline() rather than readlines(). eachline() will iterate through the file for you. cheers, Keith
Re: [julia-users] How to efficiently count the number of lines in a file
This should work: a = Mmap.mmap(test.txt) n = 1 for i in 1:length(a) if a[i]==10 n+=1 end end @show n Am 19.08.2015 um 12:13 schrieb René Donner li...@donner.at: I guess you could access it using mmap and simply loop through the array: http://docs.julialang.org/en/latest/stdlib/io-network/?highlight=mmap#memory-mapped-i-o I'd be curious, are there even faster alternatives? Am 19.08.2015 um 12:10 schrieb Daniel Carrera dcarr...@gmail.com: Hello, I need to count the number of lines in a large number of UNIX text files, most of which are fairly large. I need help coming up with an efficient implementation of a line-count program. A naive implementation like length(readlines(foo.txt)) is very slow (notice how this loads the whole file into memory just to count newlines). I imagine that it should be possible to count the number of newline characters quickly, like what the wc command does, but I can't figure out how. Does anyone have any ideas? Thanks for the help. Daniel.
[julia-users] Benchmarking Julia HttpServer
Hi there! I was trying to do some basic benchmark of Julia's HttpServer, however I have some strange behaviour. When I ask ab to do 20k requests in a single thread, at around request #16k Julia starts accepting connections quite slowly (around 20s). If I request two threads, then ab fails with timeout on connecting (60s). I thought that the problem may be with the GC so I disabled it, but that didn't help. I would be grateful for the tips on how to look into this more closely. (Julia 0.3.10 via brew on os x)
[julia-users] Re: Julia nightlies
The generic Linux binaries from https://status.julialang.org/download/linux-x86_64 are usually kept a little more up to date, the buildbots repsonsible for those aren't quite as picky as the Launchpad setup on Ubuntu's servers. Travis can be fixed by switching to the more recent language: julia template, which uses the generic Linux binaries rather than the Ubuntu PPA. On Tuesday, August 18, 2015 at 7:39:56 PM UTC-7, Yakir Gagnon wrote: OK, I'm gonna git pull make then. That's the only way, right? On Wednesday, August 19, 2015 at 4:54:05 AM UTC+10, Uwe Fechner wrote: Well, removing the link from the website does not help with the travis failures: https://travis-ci.org/mlubin/NaNMath.jl/jobs/76114115 Regards: Uwe Am Dienstag, 18. August 2015 18:46:31 UTC+2 schrieb Viral Shah: Elliot has been busy lately, and this has not been maintained for a long time. We should probably remove that link from our website. -viral On Tuesday, August 18, 2015 at 9:52:17 PM UTC+5:30, Uwe Fechner wrote: Hello, the nightly Julia builds from: https://launchpad.net/~staticfloat/+archive/ubuntu/julianightlies are currently quiet outdated. As far as I understand they should be auto-generated every night from the head branch of the Julia repository. This seams to be currently broken. Any idea, why? The outdated version is causing problems with failing checks of Travis. Best regards: Uwe
[julia-users] Re: Julia nightlies
Ok, our current .travis.yml file looks like this:
language: cpp
compiler:
- clang
notifications:
email: false
env:
matrix:
- JULIAVERSION=juliareleases
- JULIAVERSION=julianightlies
before_install:
- sudo add-apt-repository ppa:staticfloat/julia-deps -y
- sudo add-apt-repository ppa:staticfloat/${JULIAVERSION} -y
- sudo apt-get update -qq -y
- sudo apt-get install libpcre3-dev julia -y
- if [[ -a .git/shallow ]]; then git fetch --unshallow; fi
script:
- julia -e 'Pkg.init(); Pkg.clone(pwd()); Pkg.test(NaNMath)'
How should I change it?
Uwe
Am Mittwoch, 19. August 2015 08:54:36 UTC+2 schrieb Tony Kelman:
The generic Linux binaries from
https://status.julialang.org/download/linux-x86_64 are usually kept a
little more up to date, the buildbots repsonsible for those aren't quite as
picky as the Launchpad setup on Ubuntu's servers.
Travis can be fixed by switching to the more recent language: julia
template, which uses the generic Linux binaries rather than the Ubuntu PPA.
On Tuesday, August 18, 2015 at 7:39:56 PM UTC-7, Yakir Gagnon wrote:
OK, I'm gonna git pull make then. That's the only way, right?
On Wednesday, August 19, 2015 at 4:54:05 AM UTC+10, Uwe Fechner wrote:
Well, removing the link from the website does not help with the travis
failures:
https://travis-ci.org/mlubin/NaNMath.jl/jobs/76114115
Regards: Uwe
Am Dienstag, 18. August 2015 18:46:31 UTC+2 schrieb Viral Shah:
Elliot has been busy lately, and this has not been maintained for a
long time. We should probably remove that link from our website.
-viral
On Tuesday, August 18, 2015 at 9:52:17 PM UTC+5:30, Uwe Fechner wrote:
Hello,
the nightly Julia builds from:
https://launchpad.net/~staticfloat/+archive/ubuntu/julianightlies
are currently quiet outdated.
As far as I understand they should be auto-generated every night from
the head
branch of the Julia repository.
This seams to be currently broken.
Any idea, why?
The outdated version is causing problems with failing checks of Travis.
Best regards:
Uwe
[julia-users] Re: Julia installer does not put METADATA in user home
You may need to delete User\.julia and try Pkg.init() again. On Tuesday, August 18, 2015 at 10:33:21 PM UTC-7, Kevin Kunzmann wrote: Hi, I was just trying to use Juno + Julia prebundled package on a seecond machine (Win 8). It always complains about being not able to establish a connection. So I tried to install pure Julia first. That works fine and I can start a Julia session in the command line. However, when I want to update the METADATA it complains about not finding it. Indeed User\.julia\v0.3 is empty! How can I set up the correct METADATA structure manually? Best, Kevin Here is my error julia Pkg.update() INFO: Updating METADATA... ERROR: chdir METADATA: no such file or in cd at file.jl:11 (repeats 2 times)
[julia-users] Re: Syntax for slicing with an array of ranges?
julia [foo[r] for r in some_ranges]
2-element Array{Any,1}:
[6,2]
[4,9]
julia vcat([foo[r] for r in some_ranges]...)
4-element Array{Int64,1}:
6
2
4
9
Den onsdag 19 augusti 2015 kl. 04:53:55 UTC+2 skrev John Brock:
Is there an easy way to slice an array using an array of ranges? I'm
looking for something like the following:
foo = [2,6,2,8,4,9]
some_ranges = UnitRange{Int64}[2:3, 5:6]
foo[some_ranges] # gives error; desired output is [6,2,4,9]
[julia-users] Re: Julia installer does not put METADATA in user home
Saved my day, thanks! On Wednesday, 19 August 2015 08:26:57 UTC+2, Tony Kelman wrote: You may need to delete User\.julia and try Pkg.init() again. On Tuesday, August 18, 2015 at 10:33:21 PM UTC-7, Kevin Kunzmann wrote: Hi, I was just trying to use Juno + Julia prebundled package on a seecond machine (Win 8). It always complains about being not able to establish a connection. So I tried to install pure Julia first. That works fine and I can start a Julia session in the command line. However, when I want to update the METADATA it complains about not finding it. Indeed User\.julia\v0.3 is empty! How can I set up the correct METADATA structure manually? Best, Kevin Here is my error julia Pkg.update() INFO: Updating METADATA... ERROR: chdir METADATA: no such file or in cd at file.jl:11 (repeats 2 times)
Re: [julia-users] Julia nightlies
You can use the default .travis file which gets generated on current julia when
you run Pkg.generate(MyPackage,MIT)
# Documentation: http://docs.travis-ci.com/user/languages/julia/
language: julia
os:
- linux
- osx
julia:
- release
- nightly
notifications:
email: false
# uncomment the following lines to override the default test script
#script:
# - if [[ -a .git/shallow ]]; then git fetch --unshallow; fi
# - julia -e 'Pkg.clone(pwd()); Pkg.build(MyPackage); Pkg.test(MyPackage;
coverage=true)'
Am 19.08.2015 um 09:19 schrieb Uwe Fechner uwe.fechner@gmail.com:
Ok, our current .travis.yml file looks like this:
language: cpp
compiler:
- clang
notifications:
email: false
env:
matrix:
- JULIAVERSION=juliareleases
- JULIAVERSION=julianightlies
before_install:
- sudo add-apt-repository ppa:staticfloat/julia-deps -y
- sudo add-apt-repository ppa:staticfloat/${JULIAVERSION} -y
- sudo apt-get update -qq -y
- sudo apt-get install libpcre3-dev julia -y
- if [[ -a .git/shallow ]]; then git fetch --unshallow; fi
script:
- julia -e 'Pkg.init(); Pkg.clone(pwd()); Pkg.test(NaNMath)'
How should I change it?
Uwe
Am Mittwoch, 19. August 2015 08:54:36 UTC+2 schrieb Tony Kelman:
The generic Linux binaries from
https://status.julialang.org/download/linux-x86_64 are usually kept a little
more up to date, the buildbots repsonsible for those aren't quite as picky as
the Launchpad setup on Ubuntu's servers.
Travis can be fixed by switching to the more recent language: julia
template, which uses the generic Linux binaries rather than the Ubuntu PPA.
On Tuesday, August 18, 2015 at 7:39:56 PM UTC-7, Yakir Gagnon wrote:
OK, I'm gonna git pull make then. That's the only way, right?
On Wednesday, August 19, 2015 at 4:54:05 AM UTC+10, Uwe Fechner wrote:
Well, removing the link from the website does not help with the travis
failures:
https://travis-ci.org/mlubin/NaNMath.jl/jobs/76114115
Regards: Uwe
Am Dienstag, 18. August 2015 18:46:31 UTC+2 schrieb Viral Shah:
Elliot has been busy lately, and this has not been maintained for a long
time. We should probably remove that link from our website.
-viral
On Tuesday, August 18, 2015 at 9:52:17 PM UTC+5:30, Uwe Fechner wrote:
Hello,
the nightly Julia builds from:
https://launchpad.net/~staticfloat/+archive/ubuntu/julianightlies
are currently quiet outdated.
As far as I understand they should be auto-generated every night from the head
branch of the Julia repository.
This seams to be currently broken.
Any idea, why?
The outdated version is causing problems with failing checks of Travis.
Best regards:
Uwe
[julia-users] How to efficiently count the number of lines in a file
Hello, I need to count the number of lines in a large number of UNIX text files, most of which are fairly large. I need help coming up with an efficient implementation of a line-count program. A naive implementation like length(readlines(foo.txt)) is very slow (notice how this loads the whole file into memory just to count newlines). I imagine that it should be possible to count the number of newline characters quickly, like what the wc command does, but I can't figure out how. Does anyone have any ideas? Thanks for the help. Daniel.
Re: [julia-users] How to efficiently count the number of lines in a file
I guess you could access it using mmap and simply loop through the array: http://docs.julialang.org/en/latest/stdlib/io-network/?highlight=mmap#memory-mapped-i-o I'd be curious, are there even faster alternatives? Am 19.08.2015 um 12:10 schrieb Daniel Carrera dcarr...@gmail.com: Hello, I need to count the number of lines in a large number of UNIX text files, most of which are fairly large. I need help coming up with an efficient implementation of a line-count program. A naive implementation like length(readlines(foo.txt)) is very slow (notice how this loads the whole file into memory just to count newlines). I imagine that it should be possible to count the number of newline characters quickly, like what the wc command does, but I can't figure out how. Does anyone have any ideas? Thanks for the help. Daniel.
Re: [julia-users] Julia nightlies
Thanks, it is working now.
Am Mittwoch, 19. August 2015 09:30:08 UTC+2 schrieb René Donner:
You can use the default .travis file which gets generated on current julia
when you run Pkg.generate(MyPackage,MIT)
# Documentation: http://docs.travis-ci.com/user/languages/julia/
language: julia
os:
- linux
- osx
julia:
- release
- nightly
notifications:
email: false
# uncomment the following lines to override the default test script
#script:
# - if [[ -a .git/shallow ]]; then git fetch --unshallow; fi
# - julia -e 'Pkg.clone(pwd()); Pkg.build(MyPackage);
Pkg.test(MyPackage; coverage=true)'
Am 19.08.2015 um 09:19 schrieb Uwe Fechner uwe.fec...@gmail.com
javascript::
Ok, our current .travis.yml file looks like this:
language: cpp
compiler:
- clang
notifications:
email: false
env:
matrix:
- JULIAVERSION=juliareleases
- JULIAVERSION=julianightlies
before_install:
- sudo add-apt-repository ppa:staticfloat/julia-deps -y
- sudo add-apt-repository ppa:staticfloat/${JULIAVERSION} -y
- sudo apt-get update -qq -y
- sudo apt-get install libpcre3-dev julia -y
- if [[ -a .git/shallow ]]; then git fetch --unshallow; fi
script:
- julia -e 'Pkg.init(); Pkg.clone(pwd()); Pkg.test(NaNMath)'
How should I change it?
Uwe
Am Mittwoch, 19. August 2015 08:54:36 UTC+2 schrieb Tony Kelman:
The generic Linux binaries from
https://status.julialang.org/download/linux-x86_64 are usually kept a
little more up to date, the buildbots repsonsible for those aren't quite as
picky as the Launchpad setup on Ubuntu's servers.
Travis can be fixed by switching to the more recent language: julia
template, which uses the generic Linux binaries rather than the Ubuntu PPA.
On Tuesday, August 18, 2015 at 7:39:56 PM UTC-7, Yakir Gagnon wrote:
OK, I'm gonna git pull make then. That's the only way, right?
On Wednesday, August 19, 2015 at 4:54:05 AM UTC+10, Uwe Fechner wrote:
Well, removing the link from the website does not help with the travis
failures:
https://travis-ci.org/mlubin/NaNMath.jl/jobs/76114115
Regards: Uwe
Am Dienstag, 18. August 2015 18:46:31 UTC+2 schrieb Viral Shah:
Elliot has been busy lately, and this has not been maintained for a long
time. We should probably remove that link from our website.
-viral
On Tuesday, August 18, 2015 at 9:52:17 PM UTC+5:30, Uwe Fechner wrote:
Hello,
the nightly Julia builds from:
https://launchpad.net/~staticfloat/+archive/ubuntu/julianightlies
are currently quiet outdated.
As far as I understand they should be auto-generated every night from
the head
branch of the Julia repository.
This seams to be currently broken.
Any idea, why?
The outdated version is causing problems with failing checks of Travis.
Best regards:
Uwe
Re: [julia-users] Re: Julia nightlies
The generic linux binaries - I haven't checked those for a while, but it is worth checking if those are current. On Wed, Aug 19, 2015 at 8:09 AM, Yakir Gagnon 12.ya...@gmail.com wrote: OK, I'm gonna git pull make then. That's the only way, right? On Wednesday, August 19, 2015 at 4:54:05 AM UTC+10, Uwe Fechner wrote: Well, removing the link from the website does not help with the travis failures: https://travis-ci.org/mlubin/NaNMath.jl/jobs/76114115 Regards: Uwe Am Dienstag, 18. August 2015 18:46:31 UTC+2 schrieb Viral Shah: Elliot has been busy lately, and this has not been maintained for a long time. We should probably remove that link from our website. -viral On Tuesday, August 18, 2015 at 9:52:17 PM UTC+5:30, Uwe Fechner wrote: Hello, the nightly Julia builds from: https://launchpad.net/~staticfloat/+archive/ubuntu/julianightlies are currently quiet outdated. As far as I understand they should be auto-generated every night from the head branch of the Julia repository. This seams to be currently broken. Any idea, why? The outdated version is causing problems with failing checks of Travis. Best regards: Uwe -- -viral
