Re: [rust-dev] [ANN] Rust ported to DragonFlyBSD
Am 29.07.2014 um 20:37 schrieb Rob Latham: After trying to cross-compile Rust by specifying --target x86_64-pc-dragonfly-elf to Rust’s own configure script and spending numerous hours just to note that the build fails... Howdy, fellow crazy cross-compiling person. Did you by any chance come across my message from a week or so ago? One response seemed likely to help you if you were encountering a specific kind of build failure: https://mail.mozilla.org/pipermail/rust-dev/2014-July/010827.html what other errors were you encountering with the --target approach? When building the related C libraries you also need to specify the --sysroot to point to the DragonFly tree, otherwise it's building the libraries targeted at DragonFly but with Linux header files, which clearly is *wrong*. Actually I wasn't able to cross-compile LLVM. Also when using the --target approach, I always had to wait for hours until it finished the Linux (host) build before it started to build the target. I think I was just waiting too much time for compiles to finish. If someone else figures out how to cross-compile to DragonFly with the --target approach I am happy to know :). Regards, Michael ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] [ANN] Rust ported to DragonFlyBSD
Am 30.07.2014 um 16:04 schrieb Rob Latham: On Wed, Jul 30, 2014 at 8:44 AM, Michael Neumann mneum...@ntecs.de wrote: Am 29.07.2014 um 20:37 schrieb Rob Latham: what other errors were you encountering with the --target approach? When building the related C libraries you also need to specify the --sysroot to point to the DragonFly tree, otherwise it's building the libraries targeted at DragonFly but with Linux header files, which clearly is *wrong*. I'd set this up in mk/platform.mk. Check out how arm-apple-ios does it: CFG_IOS_SDK = $(shell xcrun --show-sdk-path -sdk iphoneos 2/dev/null) CFG_IOS_FLAGS = -target armv7-apple-darwin -isysroot $(CFG_IOS_SDK) -mios-version-min=7.0 CFG_CFLAGS_arm-apple-ios := -arch armv7 -mfpu=vfp3 $(CFG_IOS_FLAGS) https://github.com/rust-lang/rust/blob/master/mk/platform.mk#L158 I tried this as you can see here [1]. But you have to consider that Mac OS X comes with an iphone SDK while Linux does not come with DragonFly SDK :D. I think no-one ever tried to cross-compile a program for DragonFly :). [1]: https://github.com/mneumann/rust/commit/ab1124980f97558af87b789a1c0772bfa7e23704 Regards, Michael ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
[rust-dev] [ANN] Rust ported to DragonFlyBSD
Hi all, I am happy to announce that I succeeded in porting Rust to the DragonFly operating system. [This article][1] describes `how` and might also be of interest to others porting Rust to NetBSD or OpenBSD. Within the next week I submit patches to the LLVM project (segmented stack support for DragonFly) and also to the rust repo itself. My `dragonfly` branch of the Rust repository can be found [here][2]. If someone is interested to try out early, I can provide some binaries until I set up some automatic builds. Regards, Michael [1]: http://www.ntecs.de/blog/2014/07/29/rust-ported-to-dragonflybsd/ [2]: https://github.com/mneumann/rust/tree/dragonfly ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] PipeStream.by_ref() - Multiple applicable methods problem
Am 29.04.2014 22:51, schrieb Alex Crichton:
The by_ref() method exists on both the Reader and the Writer trait,
and you're working with a stream which implements both Reader and
Writer (hence the confusion by the compiler).
You could work around it with something like:
fn rdr'a, T: Reader(t: 'a mut T) - RefReader'a, T {
t.by_ref()
}
Eventually, with UFCS, you'll be able to do something like:
let rdr = Reader::by_ref(mut inp);
(hopefully soon!)
Thanks so much! This works!
Actually I was trying to explicitly specify the type as in:
let rdr: RefReaderPipeStream = t.by_ref();
and was wondering why it failed.
Regards,
Michael
___
Rust-dev mailing list
Rust-dev@mozilla.org
https://mail.mozilla.org/listinfo/rust-dev
[rust-dev] PipeStream.by_ref() - Multiple applicable methods problem
Hi,
I don't know how to figure out calling by_ref() for a PipeStream:
use std::io::Process;
use std::io::BufferedReader;
fn main() {
let mut child = match Process::new(/usr/bin/xzcat,
[test.log.xz.to_owned()]) {
Ok(child) = child,
Err(e) = fail!(failed to execute child: {}, e),
};
let inp = child.stdout.get_mut_ref();
// ERROR occurs here
let mut rdr = BufferedReader::new(inp.by_ref());
for line in rdr.lines() {
print!({}, line.unwrap());
}
assert!(child.wait().success());
}
This is the error output:
test.rs:13:37: 13:49 error: multiple applicable methods in scope
test.rs:13 let mut rdr = BufferedReader::new(inp.by_ref());
^~~~
test.rs:13:37: 13:49 note: candidate #1 is `std::io::Writer::by_ref`
test.rs:13 let mut rdr = BufferedReader::new(inp.by_ref());
^~~~
test.rs:13:37: 13:49 note: candidate #2 is `std::io::Reader::by_ref`
test.rs:13 let mut rdr = BufferedReader::new(inp.by_ref());
^~~~
test.rs:13:17: 13:36 error: failed to find an implementation of trait
std::io::Reader for std::io::RefWriter,std::io::pipe::PipeStream
test.rs:13 let mut rdr = BufferedReader::new(inp.by_ref());
^~~
Any hints?
Actually what I want to accomplish is to iterate line by line over a
PipeStream.
Regards,
Michael
___
Rust-dev mailing list
Rust-dev@mozilla.org
https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] About RFC: A 30 minute introduction to Rust
Am 05.03.2014 03:54, schrieb Fernando Pelliccioni:
So, now, what is the difference between...
// rust
let i = ~1234;
and
// C++
auto i = make_uniqueint(1234);
?
The Rust code is shorter, but perhaps, more illegible. I think it is
matter of taste.
But now, the real advantage is not as great as before, or not?
Except that Rust will catch use-after-move at compile time, while C++
will segfault at runtime.
Try something like this:
let i = ~1234;
let j = i; // i moves into j
println!({}, *j); // this is OK
println!({}, *i); // tries to print `i` . Compile ERROR
This will not compile in Rust!
Try the same in C++:
auto i = make_uniqueint(1234);
auto j = std::move(i);
cout *j endl; // 1234
cout *i endl; // Segmentation Fault
Maybe there are warnings generated by C++ which detect this
use-after-move... the point is that unique pointers in C++ are more a
library add-on rather than built into the language.
I think there is a bad intension of the author of the article to enlarge
the advantage.
Question, How to write the following in Rust using the operator ~ ?
auto e = make_uniqueEmployee(Peter);
In Rust this will be:
let e = ~Employee {name: ~Peter};
...and, what about shared-ownership?
I am not using shared ownership very often, but this should work:
let e = @Employee {name: ~Peter};
But I think this will change towards using either GcEmployee or
RcEmployee so you have the choice between reference counting and
(thread local) garbage collection.
If the author informs all this, I would think that there is no bad
intention, but he doesn't, and he uses the ugliest C++ code possible.
You interpret this as bad intention but I am sure there was no bad
intention involved by the author of the article. It's always difficult
to show off the advantages of new languages without hurting the feelings
of users of the old language.
I love both Rust and C++, but for C++ this is more a love-hate
relationship :)
Regards,
Michael
___
Rust-dev mailing list
Rust-dev@mozilla.org
https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] reader.lines() swallows io errors
Am 19.02.2014 08:52, schrieb Phil Dawes: Is that not a big problem for production code? I think I'd prefer the default case to be to crash the task than deal with a logic bug. The existence of library functions that swallow errors makes reviewing code and reasoning about failure cases a lot more difficult. This is why I proposed a FailureReader: https://github.com/mozilla/rust/issues/12368 Regards, Michael ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] redis-rs Pipelining and Connections
Am 19.01.2014 21:58, schrieb Armin Ronacher:
Hi,
I'm currently wrapping all of redis in a fairly high-level library
similar to the Python binding. It's currently living here:
https://github.com/mitsuhiko/redis-rs
Cool, another redis library :). This is mine: [1]
In general I did not encounter many problems with that but there are
some open questions in regards to how pipelining and connection
pooling should work.
In general, this is roughly how the library works:
extern mod redis;
fn main() {
let client = redis::Client::open(redis://127.0.0.1/).unwrap();
let mut con = client.get_connection().unwrap();
println!(Got value: {}, con.get(my_key).unwrap_or(no value));
}
Pipelining:
I currently have no idea how to implement this. The API I had in mind
was this:
let mut con = client.get_connection().unwrap();
let mut counter, data;
con.pipeline()
.incr(counter).tap(|value| { counter = value; })
.get(data_key).tap(|value| { data = value; })
.execute();
The general idea is pretty simple: whereas a regular redis connection
immediately returns the results the pipeline buffers them up and will
execute the tap'ed callbacks to return the data. Unfortunately I have
no idea how this can be implemented currently. There are two issues
with that: first of all I don't fancy implementing all methods twice
(once for the connection and once for the pipeline), secondly the
.tap() method needs to change signature depending on the return value
of the most recent operation.
I think, if you add something like Postpone(mut Connection) to the
Value type it could work.
Method tap would only be defined for type Value and will fails it it's
value is not Postpone.
Something like that:
enum Value {
Nil,
Int(int64),
Data(~[u8]),
Error(~str),
Status(~str),
Postpone(mut Connection)
}
impl Value {
fn tap(self, fn callback) - Connection {
match *self {
Postpone(conn) = { conn.add_callback(callback); conn }
_ = fail!()
}
}
}
Of course incr() etc. will only return Postpone if it is in pipeline
mode, otherwise it will execute
normally and return the redis value.
Lastly because the pipeline borrows the connection as mutable the code
currently would need to be placed in a separate scope, otherwise the
con object becomes unusable after the pipeline call.
Connections:
I don't know what the best way to deal with connections is. Right now
I have a client object which tries to connect to redis and does the
address resolution. The actual connection however is provided by a
get_connection() function on it which will connect and return a
connection object. This way two tasks can have a connection each. I
was thinking of extending this with a connection pool but I'm not sure
how to do this properly since I don't want that the client needs to be
mutable to get a connection. That would make it much harder to use
with multiple tasks.
Hm, in my rust-redis library, I just connect in redis::Client::new().
That's pretty simple. What's the problem if each task just calls
Client::new() instead of get_connection()? If address resolution is your
problem, I'd solve it differently.
Regards,
Michael
[1]: https://github.com/mneumann/rust-redis
___
Rust-dev mailing list
Rust-dev@mozilla.org
https://mail.mozilla.org/listinfo/rust-dev
[rust-dev] ASCII character literals
Hi,
There are lots of protocols based on ASCII character representation. In
Rust, the natural way to represent them is
by an u8 literal (optionally wrapped within std::ascii::Ascii).
What I am missing is a simple way to represent those literals in code.
What I am doing most of the time is:
fn read_char() - Optionchar {
match io.read_byte() {
Some(b) = Some(b as char),
None = None
}
}
And then use character literals in pattern matching. What I'd highly
prefer is a way to directly repesent ASCII characters
in the code, like:
match io.read_byte().unwrap {
'c'_ascii =
}
If macros would work in patterns, something like:
match ... {
ascii!('a') = ...
}
would work for me too. Ideally that would work with range patterns as
well, but of course an ascii_range!() macro would
do the same.
Is this useful to anyone?
Regards,
Michael
___
Rust-dev mailing list
Rust-dev@mozilla.org
https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] RFC: Future of the Build System
rustc is just another regular Rust application. So use the tools that any other Rust application (will) use ;-) I think at some point in time there will be a capable build tool written in Rust (like there is for all the other languages). Then it would make sense to switch using it for the compiler as well. Michael Am 11.01.2014 08:56, schrieb George Makrydakis: There is little reason to believe that having a build system in Rust would make It harder for people to package. I do understand the predependecy argument, but the Rust compiler itself in order to compile has predependencies anyway, as does any similar project. Therefore the decisional weight of choosing a non - rust based solution over a rust one because Debian packagers have problems packaging a compiler is not adequately justified. Using a well known build system as a means to appeal to programmers is seemingly an advantage, but it does not exonerate them from having to be competent in Rust before they write useful programs. And that has a learning curve superior to that of a build system. As for boost's jam I have nothing to say other than boost having its own build system makes it easy for boost first; this does not mean that their needs are those of everybody else and boost is a library, not a programming language itself. So, again, a decision based on picking a popular solution on the basis of such a comparison, has flawed background. Lastly, imagine the irony of Rust proposing to use python, c, c++ based build tools for simple packages. That would make packagers more frustrated because of a wider set of dependecies. While end users would have to also deal with a known system, its eventual inadequacies could not be met directly by Rust devs unless they start amending that system in order to deal with them. Therefore, maintenance overhead is inescapable either way, with the pessimization of relying in another nom - Rust project in order to make it worth your while to enjoy programming in Rust. The only valid argument against having a build system proposed as the official, defacto, cross - platform way of building rust packages written in rust is its development and maintenance overhead for the rust core team itself. That problem is easily circumvented by not proposing one right now and letting it to the end developer decide. If however an official build system is to be proposed, Rust developers merit having it done on their own platform, thus proving rust's worth. It is 2014 after all. G. Lee Braiden leebr...@gmail.com wrote: On 10/01/14 08:16, Gaetan wrote: I am not in favor of a customized build system. For instance boost library use their jam build system, and i never figured how to use it in my projects. I push to use standard and well proved build system like cmake or scons, at least for major components. This would give a nice example of how to use it in any projects. I'd agree with that on both counts: the principle of using something standard, and the two recommendations. CMake would probably get my vote, because it's not so much a build tool, as a meta tool for whichever system you prefer, so it would fit in well with various platform-specific IDEs, unusual platforms (android, embedded, ...), etc. That said, scons is also a strong contender, and which of the two is more open to integrating patches and working with new languages is very much worth considering. I think Rust will be contributing to the wider community by lending its support (and patches) to a common, modern build system, AND it will get something back in terms of users who already know the build system. On Friday, January 10, 2014, George Makrydakis wrote: Hello, Having a build system entirely dependent of Rust alone, would make the entire experience in deploying the language extremely cohere. The only counter - argument is indeed that it would require some work to get this to fruition. I would like to know if this has any chance of getting priority soon enough. Bear in mind that Debian are having a lot of issues packaging Rust already, because it self-compiles. If the build tool also had a Rust pre-dependency, that would be a big step backwards. ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
[rust-dev] Porting rust to DragonFlyBSD
Hi there, At the moment rust only supports Linux/FreeBSD/Windows/MacOSX. I'd like to be able to compile it on DragonFlyBSD [1]. I am trying to get the FreeBSD stage0/bin/rustc to run on DragonFly, yet with no success. Is it possible to generate a static rustc binary somehow? Or what in general is the procedure to port rustc to a different platform? Regards, Michael [1]: http://www.dragonflybsd.org/ ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
[rust-dev] [ANN] rust-toml - TOML configuration file parser
Hi all, rust-toml [1] is TOML [2] configuration file parser :) Regards, Michael [1]: https://github.com/mneumann/rust-toml [2]: https://github.com/mojombo/toml ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] Configuration files library for Rust
Am 04.01.2014 15:01, schrieb Flaper87: Hi Guys, I was looking around and I couldn't find a config file parser library for Rust. Getopt support seems to be pretty complete and stable, which would make the development of such library easier. There is now a TOML parser for Rust [1] :) [1]: https://github.com/mneumann/rust-toml Regards, Michael ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
[rust-dev] [ANN] rust-redis and rust-msgpack
Hi all, rust-redis: A Redis client library written in pure Rust. Thanks to the new rust runtime it is pretty fast, despite being only 200 lines of code. rust-msgpack: Fully featured and high performance msgpack implementation for Rust. Both work with rust 0.9-pre. Regards, Michael ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] [ANN] rust-redis and rust-msgpack
Am 04.01.2014 17:14, schrieb Michael Neumann: Hi all, rust-redis: A Redis client library written in pure Rust. Thanks to the new rust runtime it is pretty fast, despite being only 200 lines of code. rust-msgpack: Fully featured and high performance msgpack implementation for Rust. Both work with rust 0.9-pre. Too stupid/sleepy that I forgot the links: http://github.com/mneumann/rust-redis http://github.com/mneumann/rust-msgpack Regards, Michael ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] Configuration files library for Rust
Am 04.01.2014 18:38, schrieb Corey Richardson: oslo.config looks decent. On some projects I've worked on, we started out using INI files but found them severely lacking once we wanted to extend the options. We ended up using libconfig[0], which I think is an excellent library. In multibuilder[1], we use extra::serialize to load a config directly into the struct we'll be using. It's super convenient, but a bit unfortunate in that it's impossible to make a field truly optional (OptionT requires the field to be null, iirc). [0] http://www.hyperrealm.com/libconfig/ [1] https://github.com/huonw/multibuilder/blob/master/main.rs#L68 There is also TOML [1], an extended version of the INI config file format, which is used by a variety of languages. [1]: https://github.com/mojombo/toml Regards, Michael ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] Configuration files library for Rust
Am 04.01.2014 21:30, schrieb Tony Arcieri: On Sat, Jan 4, 2014 at 12:26 PM, Gaetan gae...@xeberon.net mailto:gae...@xeberon.net wrote: I m interessed on having your feedback on json and yaml vs toml for instance. JSON is ugly and token-ridden for configuration files. Having worked with tools that use it for this purpose, I find the configuration hard to read. Then there's the issue of comments, which are particularly important for configuration files. Some things have adopted JS-style comments for this purpose, but that's technically not standard JSON. YAML's problem is indentation errors can turn into configuration errors, and they're incredibly tricky to spot. I've run into cases where we didn't spot problems until we deployed to production because the production section of a configuration file was misindented. And, I think it's pretty hard to write a YAML parser. The spec is pretty extensive. Whereas a TOML or INI parser, you can hack within a few hours. Regards, Michael ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] Static initialisation of LinearMap
Am 02.07.2013 17:29, schrieb Alex Crichton:
I was looking for something like: static h:HashMapK,C =
{(K,V),(K,V).}.
Is this possible at all?
What would be much easier is to use a sorted array and binary search for
lookup.
But sorting at compile time seams to be tricky, for simple values ok,
but for more
complex keys I think it's impossible.
Regards,
Michael
___
Rust-dev mailing list
Rust-dev@mozilla.org
https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] RFC: Explicit stack switching
Am 02.02.2013 10:16, schrieb Patrick Walton:
On 2/1/13 11:02 PM, Brian Anderson wrote:
In the library we add this sort of function that simply guarantee that
the closure has some amount of stack available.
do reserve_stack(Standard) { rust_task_fail(); }
do reserve_stack(Tiny) {... }
do reserve_stack(Large) { }
do reserve_stack(Size(4096)) { }
My main worry about this is that it's always guesswork. Determining
how much stack a C function needs is really hard and involves doing a
lot of non-local reasoning. Getting it wrong can result in exploitable
security vulnerabilities. From a safety POV, it seems that you always
really want as big a stack as possible, unless the function is
something trivial like floor().
Ultimately we never often want a stack to be increased in size as this
seems to be a quite expensive operation. Calling a function that crosses
stack space
boundaries (i.e. when it has to alloc new stack space) inside a loop
might severly affect performance. do reserve_stack could be used to
prevent that, but
the programmer need to be aware of that. At least theoretical it should
be possible to calculate how much stack space a given function needs
including
all the space that all called functions need. Recursive functions would
result in unlimited stack space, just because we cannot analyse the
depths of
the calls. But for most of the code, we would know the maximum stack
space used. We could use this when we start new tasks to allocate a
contiguous
stack large enough to hold the whole computation of that task without
the need to resize (in which case we could optimize away the checks in front
of each function).
Most functions called by FFI should have a pretty flat call hierarchy.
At least functions in lib_c do have. Ideally there is information at
link time how much
space a given function consumes (I read something about this topic on
LLVM segmented stacks). If not, then it is always a guess and it might
be better
to perform those calls only inside a special task and make sure this
task has enough space and that it is protected against stack overflowing
(don't have
native threads a guard page after the stack segment?).
Michael
___
Rust-dev mailing list
Rust-dev@mozilla.org
https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] Misc questions
Am 30.01.2013 00:43, schrieb Brian Anderson:
On 01/29/2013 04:47 AM, Michael Neumann wrote:
Am 29.01.2013 03:01, schrieb Brian Anderson:
On 01/28/2013 05:29 PM, Graydon Hoare wrote:
On 13-01-28 04:56 PM, Brian Anderson wrote:
I think libuv is doing too much here. For example, if I don't
want to
remove the socket from the event
queue, just disable the callback, then this is not possible. I'd
prefer when I could just tell libuv that
I am interested in event X (on Windows: I/O completion, on UNIX: I/O
availability).
Yet the optimization you suggest has to do with recycling the buffer,
not listening for one kind of event vs. another.
In general I'm not interested in trying to get underneath the
abstraction uv is providing. It's providing an IOCP-oriented
interface,
I would like to code to that and make the rust IO library not have to
worry when it's on windows vs. unix. That's the point of the
abstraction
uv provides, and it's valuable. If it means bouncing off epoll a
few too
many times (or reallocating a buffer a few too many times), I'm not
too
concerned. Those should both be O(1) operations.
Is it possible to do this optimization later or do we need to plan
for
this ahead of time? I would prefer to use the uv API as it's
presented
to start with.
The optimization to use a caller-provided buffer should (a) not be
necessary to get us started and (b) be equally possible on either
platform, unix or windows, _so long as_ we're actually sleeping a task
during its period of interest in IO (either the pre-readiness sleep
or a
post-issue, pre-completion sleep). In other words, if we're simulating
sync IO, then we can use a task-local buffer. If we're _not_
simulating
sync IO (I sure hope we do!) then we should let uv allocate and free
dynamic buffers as it needs them.
But I really hope we wind up structuring it so it simulates sync IO.
We're providing a task abstraction. Users _want_ the sync IO
abstraction
the same way they want the sequential control flow abstraction.
Presenting the scheduler-originating I/O as synchronous is what I
intend. I am not sure that we can guarantee that a task is actually
waiting for I/O when an I/O event occurs that that task is waiting
for. A task may block on some other unrelated event while the event
loop is doing I/O. Pseudocode:
let port = IOPort::connect(); // Assume we're doing I/O reads using
something portlike
while port.recv() {
// Block on a different port, while uv continues doing I/O on
our behalf
let intermediate_value = some_other_port.recv();
}
This is why I'm imagining that the scheduler will sometimes need to
buffer.
I don't think so. Let me explain.
This anyway is only a problem (which can be solved) iff we want to be
able to treat I/O like a
port and want to wait for either one to resume our thread. And I
assume we want this, so
that we can listen on an I/O socket AND for example for incoming
messages at the same time.
The kernel provides a way to do (task-local) blocking I/O operations.
There is no way for the
task to return from a read() call unless data comes in or in case of
EOF (or any other error
condition).This behaves basically like a blocking POSIX read() call,
just that it is converted
into asynchronous read by libuv under the hood. To expose I/O as
port, we have to start
a new task:
let fd = open(...);
let (po, ch) = streams::pipe();
do task::spawn {
loop {
let buf: ~[u8] = vec::from_fn(1000, || 0);
let nread = fd.read(buf, 1000);
if nread 0 {
ch.send(Data(buf))
}
else if nread == 0 {
ch.send(EOF)
}
else {
ch.send(Error)
}
}
}
Yes, a single call to 'read' will not return until some I/O arrives,
but after 'read' returns I/O continues to arrive and that I/O needs to
be stored somewhere if the task doesn't immediately block in another
call to 'read' on that same fd. Taking the above example:
loop {
// This will block until data arrives at which point the task will
be context-switched in and the data returned.
let nread = fd.read(buf, 1000);
// This will put the task to sleep waiting on a message on cmd_port
let command = cmd_port.recv();
}
Until data arrives on cmd_port the task cannot be scheduled. While the
task is asleep the I/O loop can't be blocked since other tasks are
using it too. So in the meantime uv continues to receive data from the
open fd and it needs to live somewhere until the task calls 'read'
again on the same fd. Perhaps there's something I don't understand
about the uv API here, but I think that once we start reading uv is
going to continually provide us with data whether we are ready for it
or not.
// now we can treat `po` as a Port and call select() on it
But I don't think channel I/O will be used that often.
Note that one big advantage is that we can specify the buffer size
ourself!
When we would let libuv create a buffer for us, how would
Re: [rust-dev] Misc questions
Am 29.01.2013 03:01, schrieb Brian Anderson:
On 01/28/2013 05:29 PM, Graydon Hoare wrote:
On 13-01-28 04:56 PM, Brian Anderson wrote:
I think libuv is doing too much here. For example, if I don't want to
remove the socket from the event
queue, just disable the callback, then this is not possible. I'd
prefer when I could just tell libuv that
I am interested in event X (on Windows: I/O completion, on UNIX: I/O
availability).
Yet the optimization you suggest has to do with recycling the buffer,
not listening for one kind of event vs. another.
In general I'm not interested in trying to get underneath the
abstraction uv is providing. It's providing an IOCP-oriented interface,
I would like to code to that and make the rust IO library not have to
worry when it's on windows vs. unix. That's the point of the abstraction
uv provides, and it's valuable. If it means bouncing off epoll a few too
many times (or reallocating a buffer a few too many times), I'm not too
concerned. Those should both be O(1) operations.
Is it possible to do this optimization later or do we need to plan for
this ahead of time? I would prefer to use the uv API as it's presented
to start with.
The optimization to use a caller-provided buffer should (a) not be
necessary to get us started and (b) be equally possible on either
platform, unix or windows, _so long as_ we're actually sleeping a task
during its period of interest in IO (either the pre-readiness sleep or a
post-issue, pre-completion sleep). In other words, if we're simulating
sync IO, then we can use a task-local buffer. If we're _not_ simulating
sync IO (I sure hope we do!) then we should let uv allocate and free
dynamic buffers as it needs them.
But I really hope we wind up structuring it so it simulates sync IO.
We're providing a task abstraction. Users _want_ the sync IO abstraction
the same way they want the sequential control flow abstraction.
Presenting the scheduler-originating I/O as synchronous is what I
intend. I am not sure that we can guarantee that a task is actually
waiting for I/O when an I/O event occurs that that task is waiting
for. A task may block on some other unrelated event while the event
loop is doing I/O. Pseudocode:
let port = IOPort::connect(); // Assume we're doing I/O reads using
something portlike
while port.recv() {
// Block on a different port, while uv continues doing I/O on our
behalf
let intermediate_value = some_other_port.recv();
}
This is why I'm imagining that the scheduler will sometimes need to
buffer.
I don't think so. Let me explain.
This anyway is only a problem (which can be solved) iff we want to be
able to treat I/O like a
port and want to wait for either one to resume our thread. And I assume
we want this, so
that we can listen on an I/O socket AND for example for incoming
messages at the same time.
The kernel provides a way to do (task-local) blocking I/O operations.
There is no way for the
task to return from a read() call unless data comes in or in case of EOF
(or any other error
condition). This behaves basically like a blocking POSIX read() call,
just that it is converted
into asynchronous read by libuv under the hood. To expose I/O as port,
we have to start
a new task:
let fd = open(...);
let (po, ch) = streams::pipe();
do task::spawn {
loop {
let buf: ~[u8] = vec::from_fn(1000, || 0);
let nread = fd.read(buf, 1000);
if nread 0 {
ch.send(Data(buf))
}
else if nread == 0 {
ch.send(EOF)
}
else {
ch.send(Error)
}
}
}
// now we can treat `po` as a Port and call select() on it
But I don't think channel I/O will be used that often.
Note that one big advantage is that we can specify the buffer size ourself!
When we would let libuv create a buffer for us, how would it know the
buffer size? The alloc_cb you provide to libuv upon uv_start_read() will get
a suggested_size parameter passed, but this is 64k by default, and libuv
cannot know what kind of I/O protocol you are handling. When I do
line oriented I/O, I would not need a full 64k buffer allocated for every
read, which in the worst case would only return one byte in it in case
of a very slow sender (send one byte each second). Or is 64k enough
for receiving a very large packet. We clearly want a way to tell the I/O
system how large we expect the packet to be that will arrive over I/O
otherwise this is completely useless IMHO.
We would still have one separate iotask per scheduler. This is a native
thread and runs the I/O loop. There is no way to do that inside the
scheduler as we would block any task while waiting for I/O.
The callbacks like on_read_cb would simply notify the scheduler
that the task that was responsible for doing this read operation
can now resume. As the scheduler lives in another thread
(the thread in which all tasks of that scheduler live in)
and might be active, we need to do some locking here.
When the scheduler gets
Re: [rust-dev] Misc questions
Am 29.01.2013 02:29, schrieb Graydon Hoare: On 13-01-28 04:56 PM, Brian Anderson wrote: I think libuv is doing too much here. For example, if I don't want to remove the socket from the event queue, just disable the callback, then this is not possible. I'd prefer when I could just tell libuv that I am interested in event X (on Windows: I/O completion, on UNIX: I/O availability). Yet the optimization you suggest has to do with recycling the buffer, not listening for one kind of event vs. another. In general I'm not interested in trying to get underneath the abstraction uv is providing. It's providing an IOCP-oriented interface, I would like to code to that and make the rust IO library not have to worry when it's on windows vs. unix. That's the point of the abstraction uv provides, and it's valuable. If it means bouncing off epoll a few too many times (or reallocating a buffer a few too many times), I'm not too concerned. Those should both be O(1) operations. I think allocating a buffer performs much better than waking up the event loop for every read, because waking up the event loop involves kernel activity on both scheduler and iotask, while malloc should in most cases be pure user-level. And allocating buffers allows us to asynchronously continue reading while the task is still doing some computations. If we would allow multiple readers on a single port (do we? I think channels in Go allow that), then we would even have a very simple way to load balance I/O to multiple tasks. This could actually make sense in many scenarios. Kind of work-stealing. And we could build arbitrary pipelines. Of course we can simulate the same by using a dispatcher task, but this would incur some overhead. Is it possible to do this optimization later or do we need to plan for this ahead of time? I would prefer to use the uv API as it's presented to start with. The optimization to use a caller-provided buffer should (a) not be necessary to get us started and (b) be equally possible on either platform, unix or windows, _so long as_ we're actually sleeping a task during its period of interest in IO (either the pre-readiness sleep or a post-issue, pre-completion sleep). In other words, if we're simulating sync IO, then we can use a task-local buffer. If we're _not_ simulating sync IO (I sure hope we do!) then we should let uv allocate and free dynamic buffers as it needs them. We are kind of simulating sync IO by using a channel. But IO would be async in the background (if we do not want to wakup the event loop for every read), so we would need buffers. But I really hope we wind up structuring it so it simulates sync IO. We're providing a task abstraction. Users _want_ the sync IO abstraction the same way they want the sequential control flow abstraction. Yes. I (now) fully agree. (Indeed, on an appropriately-behaving system I fully expect task=thread and sync IO calls=system calls) If using a SizedChannel(1) each io.recv would correspond to one read() syscall, except that the syscall could have happend long ago. Channels with longer queues would mean that up to n (size of queue) read() calls could have happend. Regards, Michael ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] Misc questions
Am 29.01.2013 01:56, schrieb Brian Anderson:
On 01/28/2013 03:37 PM, Michael Neumann wrote:
Am 28.01.2013 22:46, schrieb Brian Anderson:
On 01/26/2013 04:07 AM, Michael Neumann wrote:
Am 26.01.2013 13:01, schrieb Michael Neumann:
Am 26.01.2013 12:28, schrieb Michael Neumann:
Another question: When a task sends a message to another task,
and this task is waiting exactly for this event, will it directly
switch to that task, or will it buffer the message?
Sometimes this could be quite handy and efficient. I rember this
was done in the L4 microkernel (www.l4ka.org), which only allowed
synchronous IPC. It could make sense to provide a
send_and_receive directive, which sends to the channel and lets
the scheduler know that it is now waiting for a message to
receive from another port. So send_and_receive could
directly switch to the other task, and when this does a send back
to the calling task, it will switch back to it. If you don't have
send_and_receive as atomic operation, there
is no way to switch back to the other task, as it might still be
running.
as it might still be running is here of course wrong (as we
switched to another thread). What I wanted to say is, that it is
not waiting for any event, so it is not in a blocking state, so that
we cannot directly switch back (matching the recv() and the send()).
Ideally the task that wants to read would do the non-blocking I/O
itself, and the scheduler would just notify when it can read.
But I think this is not possible with libuv as you
have no control over when to read (except using uv_read_start() /
_stop). I think this would be much more efficient and even more
powerful (one can read directly into a buffer...
there is no need to allocate a new buffer for each read as done by
libuv). So what I would suggest is the following:
// task
blocking_read(socket, buffer, ...)
// this will register socket with the schedulers event queue
(if not yet done) and block.
// once the scheduler will receive an data is available
event from the kernel
// it will unblock the task.
// then the task will do an non-blocking read() on it's own.
I'm not that familiar with the uv API. Is there a distinct 'data
available' event that happens before we start reading? I've been
assuming that, as you say, we have to control over when the read
events happen, so we would need to check whether the task initiating
this read was currently waiting for data, and either buffer it or
context switch to the task depending on its state.
No there isn't! The reason why, as far as I understand it, lies in
the way Windows handles reads. In UNIX you get notified, when you can
read, while in Windows,
you get notified when a read completed, so you are basically doing
the read asynchronously in the background (saving you another context
switch to the kernel).
I think this is called Proactor (the UNIX-way is called Reactor).
libuv wants to do this in a platform-independent way, where the
programmer who uses libuv does
not have to care about which platform he is working with.
So when we think about this sequence in libuv
uv_read_start(fd)
- on_read_cb gets triggered
uv_read_stop(fd)
what it does internally is the following:
UNIX:
register event for `fd` in event queue
epoll()
- allocate buffer
- read(fd, nonblocking)
- call on_read_cb
unregister event
Windows:
allocate buffer
start asynchronous read request
wait for completion (of any outstanding I/O)
- call on_read_cb
I think libuv is doing too much here. For example, if I don't want to
remove the socket from the event
queue, just disable the callback, then this is not possible. I'd
prefer when I could just tell libuv that
I am interested in event X (on Windows: I/O completion, on UNIX: I/O
availability).
I think a simple hack would be to store the buffer address and size
of buffer in the uv_handle_t structure:
struct our_handle {
uv_handle_t handle;
void *buffer;
size_t buffer_size;
}
and then have the alloc_cb return that:
static uv_buf_t alloc_cb(uv_handle_t *handle, size_t suggested_size)
{
struct our_handle *h = (struct our_handle*)handle;
return uv_buf_init(h-buffer, h-buffer_size);
}
You specify the alloc_cb in uv_read_start(). The only thing that you
need to consider
is that when on_read_cb gets called, you better call uv_read_stop(),
otherwise
the buffer could be overwritten the next time.
Well, yes, this should work for both UNIX and Windows. If you need
specific help, let me know.
I've been hacking a lot with libuv lately and I can't wait using
async I/O in rust (which actually
performs well).
Is it possible to do this optimization later or do we need to plan for
this ahead of time? I would prefer to use the uv API as it's presented
to start with.
I welcome any help here. One important and big step we need to get
through before trying to integrate uv into the scheduler is to create
safe Rust bindings
Re: [rust-dev] Problem with conflicting implementations for traits
Am 26.01.2013 19:20, schrieb Steven Blenkinsop:
You could define an `enum DefaultT= T`
Hm, can you make an example? How will that work? How do I use that?
Michael
On Friday, 25 January 2013, Michael Neumann wrote:
Hi,
I am getting the following error:
msgpack.rs:545:0: 555:1 error: conflicting implementations for a trait
msgpack.rs:545 http://msgpack.rs:545 pub implD: DecoderWithMap,
msgpack.rs:546 http://msgpack.rs:546 K: serialize::DecodableD,
msgpack.rs:547 http://msgpack.rs:547 V: serialize::DecodableD
~[(K,V)]: serialize::DecodableD {
msgpack.rs:548 http://msgpack.rs:548 static fn decode(self, d:
D) - ~[(K,V)] {
msgpack.rs:549 http://msgpack.rs:549 do d.read_map |len| {
msgpack.rs:550 http://msgpack.rs:550 do vec::from_fn(len) |i| {
...
msgpack.rs:539:0: 543:1 note: note conflicting implementation here
msgpack.rs:539 http://msgpack.rs:539 pub implD: DecoderWithMap,
T T: serialize::DecodableD {
msgpack.rs:540 http://msgpack.rs:540 static fn decode(self, d:
D) - T {
msgpack.rs:541 http://msgpack.rs:541
serialize::Decodable::decode(d as serialize::Decoder)
msgpack.rs:542 http://msgpack.rs:542 }
msgpack.rs:543 http://msgpack.rs:543 }
It's obvious that the two trait implementations conflict, as the
one (for T) is
more general as the other (for ~[(K,V)]). Is there anything I can
do to fix it?
I have found this discussion [1] but I see no solution to the problem.
For msgpack, I want to support maps. They are specially encoded,
so I need a
special Decoder (serialize::Decoder does not support maps in any
way). Above I
tried to extend the serialize::Decoder trait for read_map() and
read_map_elt(),
leading to DecoderWithMap:
pub trait DecoderWithMap : serialize::Decoder {
fn read_mapT(self, f: fn(uint) ⟶ T) ⟶ T;
fn read_map_eltT(self, _idx: uint, f: fn() ⟶ T) ⟶ T;
}
Then I tried to implement Decodable for ~[(K,V)] (which I want to
use as a rust
representation as a map; here I'd probably run into problems again as
serializer defines a generic implementation for ~[] and for tuples...
this at least I could solve by using a different type).
Now I would need to reimplement Decodable for any type I use, so I
tried
to use the second generic trait implementation. But this is where
it failed
with a conflict.
Would it be possible to override a standard (more generic) trait
implementation,
either implicitly (like C++ is doing) or explictly?
Best,
Michael
[1]: https://github.com/mozilla/rust/issues/3429
___
Rust-dev mailing list
Rust-dev@mozilla.org
https://mail.mozilla.org/listinfo/rust-dev
___
Rust-dev mailing list
Rust-dev@mozilla.org
https://mail.mozilla.org/listinfo/rust-dev
[rust-dev] Illegal borrow unless pure
Hi,
I am trying to get the following example to work. What I find strange
is, that
I can call vec::push in fn main(), but when I do the same from fn
pushit(), it fails
with:
t.rs:6:17: 6:24 error: illegal borrow unless pure: unique value in
aliasable, mutable location
t.rs:6 vec::push(mut a.arr[0], 3);
^~~
t.rs:6:2: 6:11 note: impure due to access to impure function
t.rs:6 vec::push(mut a.arr[0], 3);
^
Example:
struct A {
arr: ~[ ~[int] ]
}
fn pushit(a: mut A) /*unsafe*/ {
vec::push(mut a.arr[0], 3);
}
fn main() {
let mut a: A = A {arr: ~[ ~[1,2,3], ~[3,4,5] ] };
vec::push(mut a.arr[0], 3); // WORKS
pushit(mut a); // DOES NOT work!!!
error!(%?, a);
}
I am a bit lost here. When I use unsafe it works, but is it safe???
Best,
Michael
___
Rust-dev mailing list
Rust-dev@mozilla.org
https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] Misc questions
Am 25.01.2013 01:37, schrieb Patrick Walton:
On 1/24/13 3:55 PM, Michael Neumann wrote:
Hi,
Again a couple of random question...
* Would it be possible to optimize this kind of enum (two cases, where
one case contains a borrowed pointer)
into a simple pointer, where None would be represented as the null
pointer?
enum OptionA {
None,
Some(~A)
}
As the pointer to A can never be null it should be possible. This
probably wouldn't affect performance much,
but when storing it into an Array that would save a lot of space
(basically cut down space usage half).
Yes. This has been on the agenda for years. The reason why we don't
make any guarantees as to the memory layout for enums is precisely so
that we can implement optimizations like this.
Great to know that this will eventually be implemented.
* match() statements. I think the order in which the matches are
performed are important. But when I have
a very simple statement like this:
match io.read_char() as u8 {
0x0c = ...,
0x0d = ...,
0x0f .. 0x1a =
...
}
will the compiler construct an efficient goto jump table or will it
construct sequential if statements instead?
´ My question is if it makes sense to reorder more frequent cases to the
top or not.
LLVM will construct a jump table. I've verified this in my NES emulator.
Great. Hopefully it merges common statements as well. That is for
0x00 .. 0x0f = io::println(...)
it don't generate 15 separate io::println instructions. I assume it
generates a table which contains IP offsets.
Also I wonder why I get a non-exhaustive patterns error message for
this one:
match c as u8 {
0 .. 255 = 1
}
The exhaustiveness checker currently doesn't know about integer
ranges. This is probably a bug.
It's unituitive, so I think it's a bug :)
* Using str::as_bytes()
I cannot get str::as_bytes working. The example in the documention is
not working for several reasons (wrong syntax...)
I tried this:
fn write(buf: ~[u8]) {
io::println(fmt!(%?, buf));
}
fn main() {
let mystr = ~Hello World;
do str::as_bytes(mystr) |bytes| {
write(*bytes);
}
}
But get the compiler error:
t.rs:8:10: 8:16 error: moving out of dereference of immutable pointer
t.rs:8 write(*bytes);
^~
I think you want `[u8]`.
Of course! I feel little stupid now *g*.
* What exaclty is the semantic of as? Is it like a C-cast?
Imagine if I have
let b: u8 = 255;
let s: i8 = b as i8;
This gives -1 for s. But when I do b as i32, it gives 255.
If I want to keep the sign I have to do (b as i8) as i32.
It's supposed to be like a C cast. This seems like a bug to me.
Hm, for me it makes sense somehow.
I think that every cast from unsigned to signed (or vice versa)
will first extend the size to the requested size so that:
u8 as i32 equivalent to (u8 as u32) as i32
and this is clearly different to (u8 as i8) as i32, because
i8 as i32 will sign-extend, i.e. keep the upper bit.
Thanks for your answers,
Michael
___
Rust-dev mailing list
Rust-dev@mozilla.org
https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] Misc questions
Am 26.01.2013 12:28, schrieb Michael Neumann: Another question: When a task sends a message to another task, and this task is waiting exactly for this event, will it directly switch to that task, or will it buffer the message? Sometimes this could be quite handy and efficient. I rember this was done in the L4 microkernel (www.l4ka.org), which only allowed synchronous IPC. It could make sense to provide a send_and_receive directive, which sends to the channel and lets the scheduler know that it is now waiting for a message to receive from another port. So send_and_receive could directly switch to the other task, and when this does a send back to the calling task, it will switch back to it. If you don't have send_and_receive as atomic operation, there is no way to switch back to the other task, as it might still be running. as it might still be running is here of course wrong (as we switched to another thread). What I wanted to say is, that it is not waiting for any event, so it is not in a blocking state, so that we cannot directly switch back (matching the recv() and the send()). Ideally the task that wants to read would do the non-blocking I/O itself, and the scheduler would just notify when it can read. But I think this is not possible with libuv as you have no control over when to read (except using uv_read_start() / _stop). I think this would be much more efficient and even more powerful (one can read directly into a buffer... there is no need to allocate a new buffer for each read as done by libuv). So what I would suggest is the following: // task blocking_read(socket, buffer, ...) // this will register socket with the schedulers event queue (if not yet done) and block. // once the scheduler will receive an data is available event from the kernel // it will unblock the task. // then the task will do an non-blocking read() on it's own. Basically it's the same what libuv does internally on it's own, just that the responsibility for doing the read's for example is moved into the task itself, so there is no longer a need for an I/O task and we gain full control of the asynchronous reads. The advantage is: * we no longer need messages for I/O. * more flexibility * much better memory usage (no need to copy anymore) * the design is much easier and better to understand, libraries become so much easier Maybe that's just what you want to implement with the scheduler rewrite? Best, Michael ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] Misc questions
Am 26.01.2013 13:01, schrieb Michael Neumann: Am 26.01.2013 12:28, schrieb Michael Neumann: Another question: When a task sends a message to another task, and this task is waiting exactly for this event, will it directly switch to that task, or will it buffer the message? Sometimes this could be quite handy and efficient. I rember this was done in the L4 microkernel (www.l4ka.org), which only allowed synchronous IPC. It could make sense to provide a send_and_receive directive, which sends to the channel and lets the scheduler know that it is now waiting for a message to receive from another port. So send_and_receive could directly switch to the other task, and when this does a send back to the calling task, it will switch back to it. If you don't have send_and_receive as atomic operation, there is no way to switch back to the other task, as it might still be running. as it might still be running is here of course wrong (as we switched to another thread). What I wanted to say is, that it is not waiting for any event, so it is not in a blocking state, so that we cannot directly switch back (matching the recv() and the send()). Ideally the task that wants to read would do the non-blocking I/O itself, and the scheduler would just notify when it can read. But I think this is not possible with libuv as you have no control over when to read (except using uv_read_start() / _stop). I think this would be much more efficient and even more powerful (one can read directly into a buffer... there is no need to allocate a new buffer for each read as done by libuv). So what I would suggest is the following: // task blocking_read(socket, buffer, ...) // this will register socket with the schedulers event queue (if not yet done) and block. // once the scheduler will receive an data is available event from the kernel // it will unblock the task. // then the task will do an non-blocking read() on it's own. Basically it's the same what libuv does internally on it's own, just that the responsibility for doing the read's for example is moved into the task itself, so there is no longer a need for an I/O task and we gain full control of the asynchronous reads. The advantage is: * we no longer need messages for I/O. * more flexibility * much better memory usage (no need to copy anymore) * the design is much easier and better to understand, libraries become so much easier * and message passing could be done synchronous, i.e. very fast :) ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
[rust-dev] Misc questions
Hi,
Again a couple of random question...
* Would it be possible to optimize this kind of enum (two cases, where
one case contains a borrowed pointer)
into a simple pointer, where None would be represented as the null pointer?
enum OptionA {
None,
Some(~A)
}
As the pointer to A can never be null it should be possible. This
probably wouldn't affect performance much,
but when storing it into an Array that would save a lot of space
(basically cut down space usage half).
* match() statements. I think the order in which the matches are
performed are important. But when I have
a very simple statement like this:
match io.read_char() as u8 {
0x0c = ...,
0x0d = ...,
0x0f .. 0x1a =
...
}
will the compiler construct an efficient goto jump table or will it
construct sequential if statements instead?
´ My question is if it makes sense to reorder more frequent cases to the
top or not.
Also I wonder why I get a non-exhaustive patterns error message for
this one:
match c as u8 {
0 .. 255 = 1
}
* Using str::as_bytes()
I cannot get str::as_bytes working. The example in the documention is
not working for several reasons (wrong syntax...)
I tried this:
fn write(buf: ~[u8]) {
io::println(fmt!(%?, buf));
}
fn main() {
let mystr = ~Hello World;
do str::as_bytes(mystr) |bytes| {
write(*bytes);
}
}
But get the compiler error:
t.rs:8:10: 8:16 error: moving out of dereference of immutable pointer
t.rs:8 write(*bytes);
^~
* A ~str is internally represented by an ~[u8] vector.
It is basically a heap-allocated
struct rust_vec {
size_t fill;
size_t alloc;
uint8_t data[];
}
When I correctly read the code the string is allocated in-place, i.e.
for a string of size 5,
you will allocate sizeof(struct rust_vec) + 5 + 1 bytes. So the data
pointer points past the
data. As a reallocation can change the pointer to the rust_vec, I know
understand why
I have to pass sometimes a mut ~[T]into a function. In case the
reallocation returns a new
pointer, the passed pointer has to be updated.
I guess slices use the same rust_vec struct, but allocated on the stack,
where data points to
another ~vectors data.
What I don't understand is the following comment for struct rust_vec:
size_t fill; // in bytes; if zero, heapified
Please correct me if I am wrong.
* There is a severe performance bug in TcpSocketBuf.read(). I am trying
to fix it right now myself,
once I am done, I will do another post. This explains why I get very bad
network I/O performance.
Basically the function copies the internal buffer over and over again,
once for each call.
This is especially bad when using read_line(), as it calls read() for
every byte.
* What exaclty is the semantic of as? Is it like a C-cast?
Imagine if I have
let b: u8 = 255;
let s: i8 = b as i8;
This gives -1 for s. But when I do b as i32, it gives 255.
If I want to keep the sign I have to do (b as i8) as i32.
* I don't like the way libuv is currently integrated into the system. It
works, but performance is
quite low and IMHO the blocking interface is not very usable. For
example I want to write a process
that accepts messages from other processes, and then writes something to
the socket or reads from
the socket. This will currently not work, as reading from the socket
will block the process, and
then no more requests can be sent to the process.
So instead of using the read() / write() API of an io::Reader, I'd
prefer to expose the read/write
events of libuv via messages (this is already done between the iotask
and the read()/write() methods,
but it is not accessible to the end-user).
So instead of:
io.read(...)
one would simply write:
readport.recv()
The same for writes. EOF results in closing the readport. The question
is how these messages
should look like to be usable for the programmer (how to handle errors?).
What do you think?
Actually there would be connecting ports, which receive events whenever
a new connection is established.
A successfully established connection would then be represented by a
readport and writechannel.
* I'd like to know more how the task scheduler and the pipes work
together. Is there any info available somewhere?
Also, if I would create a native pthread in C, could I simply call an
external rust function?
Best,
Michael
___
Rust-dev mailing list
Rust-dev@mozilla.org
https://mail.mozilla.org/listinfo/rust-dev
[rust-dev] Trying to get simple TCP server working
Hi,
I am trying to write a very simple TCP server (code given below), which
just reads from a
socket, but somehow the read() blocks forever. Any idea what I am doing
wrong?
I also tried to create a task for each new incoming connection, but the
same happens.
I actually found this example on the internet...
When I do wget http://127.0.0.1:4000/; it prints
- Server is listening
- New client
- Accepted!
- Unwrapped
and then it blocks forever. I also cannot create new connections at this
point.
Best,
Michael
/*
* Simple TCP server
*/
extern mod std;
use std::net::tcp;
use std::net::ip;
use std::uv;
fn main() {
tcp::listen(ip::v4::parse_addr(127.0.0.1), 4000, 100,
uv::global_loop::get(),
|_comm_chan|{
error!(Server is listening);
},
|new_client, _comm_chan|{
error!(New client);
let result = tcp::accept(new_client);
if result.is_ok(){
error!(Accepted!);
let socket = result::unwrap(move result);
error!(Unwrapped);
// Now do stuff with this socket
let data = socket.read(100); // XXX: This blocks
io::println(fmt!(%?, data));
}else{
error!(Not accepted!);
}
} */
});
}
___
Rust-dev mailing list
Rust-dev@mozilla.org
https://mail.mozilla.org/listinfo/rust-dev
[rust-dev] Misc questions and ideas
Hi,
I've spent the last days hacking in Rust and a few questions and ideas
have accumulated over that time.
* If I use unique ~pointers, there is absolutely no runtime overhead,
so neither ref-counting nor GC is involved, right?
* Heap-allocated pointers incur ref-counting. So when I pass a
@pointer, I will basically pass a
struct heap_ptr {ptr: *byte, cnt: uint}
around. Right?
* vec::build_sized() somehow seems to be pretty slow. When I use it,
instead of a for() loop, my rust-msgpack library slows down by
factor 2 for loading msgpack data.
Also, I would have expected that vec::build_sized() will call my
supplied function n times. IMHO the name is little bit
misleading here.
* I do not fully understand the warning of the following script:
fn main() {
let bytes =
io::read_whole_file(path::Path(/tmp/matching.msgpack)).get();
}
t2.rs:2:14: 2:78 warning: instantiating copy type parameter with a not
implicitly copyable type t2.rs:2 let bytes =
io::read_whole_file(path::Path(/tmp/matching.msgpack)).get();
^~~~
Does it mean that it will copy the ~str again? When I use pattern
matching instead of get(), I don't get this warning, but it seems to
be slower. Will it just silence the warning???
* This is also strange to me:
fn nowarn(bytes: [u8]) {}
fn main() {
let bytes = ~[1,2,3];
nowarn(bytes);
let br = io::BytesReader { bytes: bytes, pos: 0 }; // FAILS
}
t.rs:6:36: 6:41 error: mismatched types: expected `/[u8]` but found
`~[u8]` ([] storage differs: expected but found ~) t.rs:6 let br =
io::BytesReader { bytes: bytes, pos: 0 }; ^
It implicitly converts the ~pointer into a borrowed pointer when
calling the function, but the same does not work when using the
BytesReader struct. I think, I should use a make_bytes_reader
function, but I didn't found one.
* String literals seem to be not immutable. Is that right. That means
they are always heap allocated. I wished they were immutable, so
that writing ~my string is stored in read-only memory.
I never thought this would be possible:
let s = ~my string;
let mut s2 = s;
s2[0] = 'c' as u8;
Is there a way how a function which takes a ~str can state that it
will not modify the content?
In this regard I very much like the way the D language handles this.
It uses const to state that it won't modify the value, while the
value itself may be mutable. Then there is immutable, and a value
declared as such will not change during the whole lifetime.
Of course in Rust, thanks to unique pointers, there is less need for
immutability, as you cannot share a unique pointer between threads.
* Appending to strings. It's easy to push an element to an array by
doing:
let mut v: ~[int] = ~[1,2];
v.push(3);
v.push(4);
But when I want to append to a string, I have to write:
let mut s: ~str = ~;
let mut s = str::append(s, abc);
let mut s = str::append(s, def);
I found this a bit counter-intuitive. I know there exists +=, but
this will always create a new string. A operator would be really
nice to append to strings (or to arrays).
* Default initializers for structs. Would be nice to specify them like:
struct S {a: int = 4, b: int = 3};
I know I can use the .. notation, and this is very cool and more
flexible, but I will have to type in a lot of code if the struct get
pretty large.
const DefaultS = S{a: 4, b: 3}; // imagine this has 100 fields :)
let s = S{a: 4, ..DefaultS};
* Metaprogramming
Given an arbitrary struct S {...} with some fields, it would be nice
to somehow derive S.serialize and S.deserialize functions
automatically. Are there any ideas how to do that? In C++ I use the
preprocessor and templates for that. In D, thanks to
compile-time-code-evaluation, I can write code that will introspect
the struct during compile-time and then generate code.
I guess I could write a macro like:
define_ser_struct!(S, field1, int, field2, uint, ...)
which would generate the struct S and two functions for
serialization. Would that be possible with macros?
Thanks in advance,
Michael
___
Rust-dev mailing list
Rust-dev@mozilla.org
https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] Misc questions and ideas
Am Sun, 23 Dec 2012 12:20:07 -0500
schrieb Patrick Walton pwal...@mozilla.com:
On 12/23/12 10:43 AM, Michael Neumann wrote:
Hi,
[...]
* I do not fully understand the warning of the following script:
fn main() {
let bytes =
io::read_whole_file(path::Path(/tmp/matching.msgpack)).get();
}
t2.rs:2:14: 2:78 warning: instantiating copy type parameter with
a not implicitly copyable type t2.rs:2 let bytes =
io::read_whole_file(path::Path(/tmp/matching.msgpack)).get();
^~~~
Does it mean that it will copy the ~str again? When I use pattern
matching instead of get(), I don't get this warning, but it
seems to be slower. Will it just silence the warning???
Yes, it means it will copy the string again. To avoid this, you want
result::unwrap() or option::unwrap() instead. I've been thinking for
some time that .unwrap() should change to .get() and .get() should
change to .copy_value() or something.
That's strange. If I use result::unwrap() it is consistently becoming
much slower! But the warning goes away. While get() is faster, but
there is this warning. Btw, there is also no .unwrap(), just
result::unwrap(). I believe that unwrap() is copying, while get() is
passing a reference somehow.
Michael
___
Rust-dev mailing list
Rust-dev@mozilla.org
https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] net::tcp::TcpSocket slow?
Am 21.12.2012 05:17, schrieb Patrick Walton: I just profiled this. Some thoughts: On 12/20/12 9:12 PM, Brian Anderson wrote: First, stack switching. Switching between Rust and C code has bad performance due to bad branch prediction. Some workloads can spend 10% of their time stalling in the stack switch. This didn't seem too high, actually. It should only be ~20,000 stack switches (read and write) if we solve the following issue: Second, working with uv involves sending a bunch of little work units to a dedicated uv task. This is because callbacks from uv into Rust *must not fail* or the runtime will crash. Where typical uv code runs directly in the event callbacks, Rust dispatches most or all of that work to other tasks. This imposes significant context switching and locking overhead. This is actually the problem. If you're using a nonblocking I/O library (libuv) for a fundamentally blocking workload (sending lots of requests to redis and blocking on the response for each one), *and* you're multiplexing userland green threads on top of it, then you're going to get significantly worse performance than you would if you had used a blocking I/O setup. We can make some of the performance differential up by switching uv over to pipes, and maybe we can play dirty tricks like having the main thread spin on the read lock so that we don't have to fall into the scheduler to punt it awake, but I still don't see any way we will make up the 10x performance difference for this particular use case without a fundamental change to the architecture. Work stealing doesn't seem to be a viable solution here since the uv task really needs to be one-task-per-thread. Maybe the best thing is just to make the choice of nonblocking versus blocking I/O a choice that tasks can make on an individual basis. It's a footgun to be sure; if you use blocking I/O you run the risk of starving other tasks on the same scheduler to death, so perhaps we should restrict this mode to schedulers with 1:1 scheduling. But this would be in line with the general principle that we've been following that the choice of 1:1 and M:N scheduling should be left to the user, because there are performance advantages and disadvantages to each mode. Once this sort of switch is implemented, I would suspect the performance differential between Ruby and Rust to be much less. So I think I should benchmark it against Erlang for example or any other evented language which also do message passing instead of direct callbacks. I can imagine that if I would use libuv directly (lets say in C), and as such avoid message sending and scheduling, it would have similar performance to the blocking solution. Would you agree? The best thing I can do is to use blocking I/O here anyway as it's better to have just one connection to Redis and multiplex that, so I can easily use one native thread for that. I am just very new to Rust, and the only thing I found was tcp_net. So I think I should define my own FFI socket calls, right? Thanks! Best, Michael ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
[rust-dev] Warn about implicit integer conversion
Hi, let a: u8 = 10_000; // this does not fit inside u8! io::println(fmt!(%?, a)); // print 10 I would have expected that rust would warn me if I try to assign an integer constant that doesn't fit into the types range. Maybe there exists a warning that I can enable? Best, Michael ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
[rust-dev] Reading shared immutable data
Hi, We have a very huge immutable data structure that we want to share (read-only) between many light-weight threads. From what I have seen, I should use Arc. Is there any other way to share the data between threads? And when using Arc, can I access the data in parallel by all threads? Best regards, Michael ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
Re: [rust-dev] Reading shared immutable data
Am 18.12.2012 03:11, schrieb Patrick Walton: On 12/17/12 6:03 PM, Michael Neumann wrote: Hi, We have a very huge immutable data structure that we want to share (read-only) between many light-weight threads. From what I have seen, I should use Arc. Is there any other way to share the data between threads? You can also: * Use a reader-writer lock (RWLock). This is still safe. I will look into this. * Use unsafe code by casting to an unsafe pointer and sending the unsafe pointer over a channel; do this only as a last resort. But don't I get into problems with GC when I do that? So I create the data structure heap allocated in one thread, then get an unsafe pointer to it and sent this to all other threads. Of course in the originating thread I need to keep the reference to the data, otherwise the unsafe pointer will point to garbage. * Turn your shared state into a task and have other tasks access the data by sending it messages. This is the classical actor model solution, but usually ARC will be more efficient. But this doesn't allow for parallelism, and that's what we want. And when using Arc, can I access the data in parallel by all threads? Yes. The only synchronization cost you'll pay is the cost of an atomic CPU instruction whenever you send the data. I would just send the data once at thread creation. Hm, I have to play with that. Thanks a lot! Michael ___ Rust-dev mailing list Rust-dev@mozilla.org https://mail.mozilla.org/listinfo/rust-dev
