So far the best way to overcome it is to install all needed Julia packages
on every machine. This is not very convenient, but at least it's not a
blocker and you need to install some Julia packages manually anyway (though
I'm thinking of creating API for massive installation of packages on all
hmm, I'm already very interested in the project like Julia-on-(Spark/c++?),
and the ser/des issue is a big obstacle.
在 2015年9月21日星期一 UTC+8下午9:30:05，Andrei Zh写道：
>
> Hi,
>
> not yet. I made some initial research regarding serialization of ASTs and
> reconstructing functions from them, but it
Hi,
not yet. I made some initial research regarding serialization of ASTs and
reconstructing functions from them, but it seems quite a tricky procedure
and I have very little time for this project now. I plan to come back to
this issue around the beginning of the next month.
On Mon, Sep 21, 2015
hi， dear，
have you already fixed this problem？
在 2015年8月14日星期五 UTC+8下午11:06:30，Andrei Zh写道：
>
>
> Hi Jake,
>
> your example works because you don't leave Julia session. `foo` is defined
> in this session, so the the pair of module name and function name is enough
> to get function object.
Andrei Zh
I'm confused. Have you actually tried?
julia io = IOBuffer()
IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true,
append=false, size=0, maxsize=Inf, ptr=1, mark=-1)
julia foo(x) = x + 1
foo (generic function with 1 method)
julia serialize(io, foo)
julia
Hi Jake,
your example works because you don't leave Julia session. `foo` is defined
in this session, so the the pair of module name and function name is enough
to get function object. If you save serialized function (or just retype it
byte by byte) , it won't work. Here's an example:
If you define the function with @everywhere, it will be defined on all existing
workers. Likewise, `using MyPackage` loads the package on all workers.
--Tim
On Thursday, August 13, 2015 03:10:54 PM Andrei Zh wrote:
Ok, after going through serialization code, it's clear that default
Yes, but once again, I'm not using Julia workers, but instead completely
independent Julia processes, running on different machines and ruled by
Spark, not by Julia's ClusterManager. I.e. workflow looks like this:
1. Julia process 1 starts JVM and connects to Spark master node.
2. Julia
Ok, after going through serialization code, it's clear that default
implementation doesn't support serializing function code, but only its
name. For example, here's relevant section from
`deserialize(::SerializationState, ::Function)`:
mod = deserialize(s)::Module
name = deserialize(s)::Symbol
JLD doesn't support serializing functions but Julia itself does.
On Mon, Aug 10, 2015 at 3:43 PM, Andrei Zh faithlessfri...@gmail.com
wrote:
I'm afraid it's not quite true, and I found simple way to show it. In the
next code snippet I define function `f` and serialize it to a file:
julia
On Monday, August 10, 2015 01:13:15 PM Tony Kelman wrote:
Should
probably use some different extension for that, .jls or something, to avoid
confusion.
Yes. That has been sufficiently confusing in the past, we even cover this here:
I'm afraid it's not quite true, and I found simple way to show it. In the
next code snippet I define function `f` and serialize it to a file:
julia f(x) = x + 1
f (generic function with 1 method)
julia f(5)
6
julia open(example.jld, w) do io serialize(io, f) end
Then I close Julia REPL and
The above code wasn't using the HDF5-based JLD package/format, it was just
using .jld as a file extension to store the results of serialize(). Should
probably use some different extension for that, .jls or something, to avoid
confusion.
On Monday, August 10, 2015 at 12:45:35 PM UTC-7, Stefan
Yes, I incorrectly assumed `serialize` / `deserialize` use JLD format. But
anyway, even when I saved the function into example.jls or even plain
byte array (using IOBuffer and `takebuf_array`), nothing changed. Am I
missing something obvious?
On Monday, August 10, 2015 at 11:40:03 PM UTC+3,
My question is: does Julia's serialization produce completely
self-containing code that can be run on workers? In other words, is it
possible to send serialized function over network to another host / Julia
process and applied there without any additional information from the first
15 matches
Mail list logo
