Re: [MacRuby-devel] book idea - making it look like Ruby
On Fri, Feb 11, 2011 at 4:08 PM, Rich Morin r...@cfcl.com wrote: At 6:09 PM -0200 2/11/11, Caio Chassot wrote: On 2011-02-11, at 15:25 , Matt Aimonetti wrote: Magically converting a snake_case method call to a CamelCase method dispatch is bad for peformance and documentation. It's not clear to me that there would be a significant performance impact. It might be possible to do (most of) the translations at start-up time, avoiding much impact during execution. In JRuby it currently has no performance impact. The camel_cased versions are defined as aliases to the original. I forget whether we define them as traditional aliases (wrapped by another structure) or if we just bind the same method object under all names. We have also debated making the aliases *redispatch* so that if you wanted to replace them all you'd just replace the original camelCased version. That turned out to be a perf issue in practice, so we backed it off until we can come up with a nice way to have multiple names resolve to the same method object. FWIW, JRuby users would go ape if we took away snake_cased versions of Java names. They love having everything look like Ruby, and I personally think it helps the code flow a lot more naturally to have a single style...not to mention you don't have to remember whether you're calling a snake_cased or camelCased method...you just always use snake_case. The question of documentation is also somewhat iffy. There is already a difference in appearance between the ObjC and MacRuby calling syntaxes. Would merely allowing (ie, not requiring) the use of snake_case make this much problem much worse? As far as I know, there's never been any confusion about snake_cased aliases in JRuby. People read the JavaDocs, mentally switch to snake_case, and proceed. - Charlie ___ MacRuby-devel mailing list MacRuby-devel@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel
Re: [MacRuby-devel] WeakRef advice
Another important use of weakrefs came to mind: avoiding finalization. In general, finalizers are a bad idea. They often block GC, or at least block each other. They add overhead to GC cycles, since either finalizers must be run or objects must be enqueued for another thread to run their finalizers. By using WeakReferences (and especially PhantomReferences) you can drive finalization from userland code, avoiding much of the GC overhead that might otherwise result. - Charlie On Fri, Feb 18, 2011 at 5:28 PM, Charles Oliver Nutter head...@headius.com wrote: On Tue, Feb 15, 2011 at 5:28 PM, Laurent Sansonetti lsansone...@apple.com wrote: Hi Alan, Do you control the data structure that holds a reference to 'B'? If yes, you may want to use NSHashTable which supports weak references. To me, this sounds like a design problem. Maybe your project can be re-architectured to avoid this pattern. Weak references and weak lists/hashes are often very useful for caches you need to age out or when you need to associate data with a given instance of an object without holding hard references. Using weak references is definitely not a design problem in itself. I'm also disappointed that you'd recommend people use NSHashTable rather than just fixing WeakRef :) In JRuby, we nuked weakref.rb some time ago because it has a lot of problems. Instead, we wrap the JVM's builtin weakrefs. See also the weakling gem, which (for JRuby) provides additional features mentioned in that Ruby redmine ticket like reference queues and a simple weak ID hash to use in place of _id2ref. - Charlie ___ MacRuby-devel mailing list MacRuby-devel@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel
Re: [MacRuby-devel] WeakRef advice
On Tue, Feb 15, 2011 at 5:28 PM, Laurent Sansonetti lsansone...@apple.com wrote: Hi Alan, Do you control the data structure that holds a reference to 'B'? If yes, you may want to use NSHashTable which supports weak references. To me, this sounds like a design problem. Maybe your project can be re-architectured to avoid this pattern. Weak references and weak lists/hashes are often very useful for caches you need to age out or when you need to associate data with a given instance of an object without holding hard references. Using weak references is definitely not a design problem in itself. I'm also disappointed that you'd recommend people use NSHashTable rather than just fixing WeakRef :) In JRuby, we nuked weakref.rb some time ago because it has a lot of problems. Instead, we wrap the JVM's builtin weakrefs. See also the weakling gem, which (for JRuby) provides additional features mentioned in that Ruby redmine ticket like reference queues and a simple weak ID hash to use in place of _id2ref. - Charlie ___ MacRuby-devel mailing list MacRuby-devel@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel
Re: [MacRuby-devel] Ruby sort algorithm
On Sun, Jan 30, 2011 at 6:19 PM, Morgan Schweers cyber...@gmail.com wrote: Greetings, Ruby's sort algorithm is quicksort, last I checked, and quicksort is not stable (which is the property you're looking for in a sort). There are a bunch of ways around this, including writing your own, but one cute, quick, but possibly performance-impairing, approach I've seen (Matz's suggestion) is: FWIW, JRuby originally had a stable Array#sort, but because it was slower than MRI's unstable sort (and we got bug reports to that effect) we replaced it with an unstable hybrid sort based on quicksort. Shortly after we did that, someone raised an issue against MRI to get it to move to a stable sort :) As far as I know, MRI hasn't changed yet. - Charlie ___ MacRuby-devel mailing list MacRuby-devel@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel
Re: [MacRuby-devel] Concurrency bug in ControlTower
On Wed, Jan 26, 2011 at 4:50 PM, Laurent Sansonetti
lsansone...@apple.com wrote:
No, only locals and dynamic (block) variables.
To be honest I always disliked this semantic change too. I think it was a
mistake to add it. It will probably be reverted for 1.0.
I started to try to implement it, and it's not easy. I would need to
cascade all the way up the containing scopes, cloning each one. It's
very messy.
I would have no objections to this behavior going away.
What if instead of e.g. async, you had the following form:
queue.async(a, b) {|a, b| ... }
Because 1.9 forces variables in the block's argument list to be local
to the block, this would avoid the sharing of variables completely.
This would work with anything proc-like as well:
job = -(a, b) { ... } # block-local a and b
queue.async(a, b, job) # outer scope's a and b
Of course you might want to rename them in the block, to avoid
confusion, but this neatly does what tap did in my suggested fix
(and I don't know why I didn't think of it sooner).
- Charlie
___
MacRuby-devel mailing list
MacRuby-devel@lists.macosforge.org
http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel
Re: [MacRuby-devel] Thread safety in apply example?
On Wed, Jan 26, 2011 at 4:45 PM, Laurent Sansonetti lsansone...@apple.com wrote: Hi Charles! Sorry for the late response. As others have noted, in this snippet, #apply is called on a sequential queue (queues created by Queue.new are always sequential), therefore there shouldn't be any problem here. If the queue was concurrent, however, there would be a thread safety issue. I think this rdoc snippet should be rewritten to avoid confusion. Yup, I understand now. Thank you! I see that you're wrapping a GCD-like interface in JRuby, it's very cool! I assume you want your interface to be cross platform, but in the case of JRuby running on Mac OS X, maybe we can extract our code as a C extension, this way JRuby would use the system GCD. Maybe we can also work together on creating a good test/spec suite for the GCD interface, because it's currently lacking. To be honest I'm more and more downbeat about C extensions. On JRuby, we have opted to disallow C extensions from running concurrently, since there's just too many extensions that mutate global state. They also make it impossible to spin up multiple JRuby instances in the same JVM, since ALL C extensions hold references to a specific JRuby runtime or classes it contains in global state (i.e. no extensions out there are MVM-safe). So there are certainly other possiblities: 1. FFI. The libdispatch APIs are all C-based and seem to be pretty consistent. A single FFI binding might work for everyone including MRI. 2. Attempt to build off Java/Cocoa integration, as seen in this attempt: http://pixel.recoil.org/code/rococoa/index.html. I don't know how reliable that integration is though, or if Apple is going to continue maintaining it. A C extension would probably be a last resort since it has many limitations. re: testing: Yes! Are there many tests for this yet? I have not looked for them, so I have no idea. I'd love to see a complete suite (or at least complete for the highest-level surface area of functionality) that we could all try to meet. - Charlie ___ MacRuby-devel mailing list MacRuby-devel@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel
Re: [MacRuby-devel] Concurrency bug in ControlTower
On Mon, Jan 24, 2011 at 11:13 AM, Joshua Ballanco jball...@gmail.com wrote:
Regarding the potential bug...well...where to begin? So, yes, MacRuby blocks
have the same semantics as regular Ruby blocks, *except* when they are
dispatched through GCD. In that case, ivars are shared, but local variable
get copied. I'm sure Laurent can explain why better than I, but it has to do
with the semantics of libdispatch and the uncertainty inherent in when a
dispatched block or function will execute (i.e. if local variables were not
copied during dispatch, they might go out of scope and be collected before
GCD ever gets around to running the code).
Wow, that's very surprising. I'm not sure I agree with bending Ruby
semantics so drastically, even to help concurrency. Or at least, I'd
expect other threaded scenarios to be consistent:
~ ➔ ruby -ve a = 0; Thread.new { a += 1 }.join; p a
MacRuby 0.8 (ruby 1.9.2) [universal-darwin10.0, x86_64]
1
~ ➔ ruby -ve a = 0; q = Dispatch::Queue.concurrent; q.sync {a += 1}; p a
MacRuby 0.8 (ruby 1.9.2) [universal-darwin10.0, x86_64]
0
The implicitness in being able to mutate the surrounding scope is
certainly problematic. This is one reason Java's anonymous inner
classes require that referenced variables be declared final...to
indicate they can't be mutated by the body of the anonymous inner
class's methods.
The result is that people end up using one-element arrays and the
like, but people find ways around anything.
So I suppose this applies to anything in the surrounding scope,
including visibility, $~, and so on?
To illustrate how this impacts async semantics wrt ruby, here's a sample
done with MacRuby/GCD and canonical Ruby:
I'll see how this might be implemented in JRuby.
In other words, not a bug in MacRuby...also, this code is scheduled for a
rewrite that will turn it into a bit more of a continuation-passing style of
handling the connections. I don't think tap is really the solution we are
looking for here, since after dispatching, the main #accept loop doesn't
really care about that connection (other than that, at some point, it is
responsibly closed).
tap would be a clean way to explicitly isolate each loop's
environment, if you weren't already doing that.
- Charlie
___
MacRuby-devel mailing list
MacRuby-devel@lists.macosforge.org
http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel
Re: [MacRuby-devel] Thread safety in apply example?
On Tue, Jan 25, 2011 at 1:00 AM, Joshua Ballanco jball...@gmail.com wrote: On Mon, Jan 24, 2011 at 8:20 PM, Charles Oliver Nutter head...@headius.com wrote: I'm curious what you mean by this. Can you point out the code? Is it actually attempting to rewrite local variable or instance variable or array modifications in code? Sorry, I mis-remembered. Not a mix-in, but Dispatch::Proxy from the dispatch gem is what I was thinking of. It uses the delegator pattern to funnel all access through a serial queue. Admittedly, this technique is a bit heavy handed (and, I'm just now realizing, somewhat inefficiently implemented...). Still, it gets the job done... Ok, got it. Unsurprisingly HawtDispatch also provides a built-in queue-backed delegation proxy using Java reflection. Of course, we can do a lot better than that in Ruby. I'm building a JRuby-based mimic API of the MacRuby GCD API: https://github.com/headius/jcd That sounds like an excellent idea! One issue that we've come across repeatedly in MacRuby is how to teach Ruby programmers to write grown-up thread-safe code (and, from my half-hearted attempts to follow what's going on in JRuby, it seems you've been dealing with this as well). That is, something like @my_array[i] += 1 is safe enough when you have a GIL, but is not when you have true low-level reentrancy. (In fact, even @my_array[i] = foo is not thread-safe!) Yes, I think JRuby has helped force the issue for years, and now MacRuby gets to help :) Things used to be a lot worse. I remember spending long hours writing emails and IM responses to people that were almost angry that I decided JRuby's Array, Hash, and String would not be thread-safe. They really didn't get why we couldn't just do what MRI does. It took a lot of effort and a few favors from key greybeards in the Ruby community for people to accept that JRuby was making the pragmatic decision. And hopefully MacRuby has had an easier time selling such decisions as a result. The Ruby community has come a long way, but there's obviously more work to do. Instead of attempting to bring the world of spin-locks and mutexes (and when to use one instead of the other) into Ruby, I think it's probably more useful to introduce a transactional programming model. This LtU post comes to mind: http://lambda-the-ultimate.org/node/4070 Yes, you're right for most scenarios. There are of course many places where queues or transactions are far too heavy. What we need to do is ensure that as better concurrency models become available for Rubyists, we do what we can to ensure they work across Ruby implementations and share the load of educating users. That's what I hope to do with JCD, and I hope someone will be able to do the same for MRI in a way that works on all its supported platforms. I have other concurrency-related projects you may be interested in: The ruby-atomic gem, providing explicitly atomic references (and CAS-like operations) on JRuby and MRI: https://github.com/headius/ruby-atomic. I collaborated with MenTaLguY on this one. The cloby gem, which wraps Clojure's STM and Ref to allow creating (J)Ruby objects with all transactional instance variables: https://github.com/headius/cloby I have also worked with MRI and Rubinius folks on various forms of Multi-VM (MVM) API, providing a standard interface for launching isolated ruby environments in the same process. Of course JRuby has been able to do this for years (since org.jruby.Ruby is just another object), but MVM and similar APIs may be the only way MRI ever gets access to true in-process concurrency. Anyway, I'd be glad to lend a hand (and whatever slice of my limited time I can spare) to the effort. In particular, test suites/specs are difficult in this space, as the MacRuby team has learned first-hand. I don't have a lot to offer in this area. We benefited years ago from the big threadsafety push in Rails 2.2, by providing rails core a 32-way box and letting them beat the hell out of Rails and JRuby under heavy concurrent load. I know of no similar efforts since, for any library. What I have now is a mostly-complete Queue and Object, but I'm still figuring out how to map the other libdispatch primitives to what HawtDispatch provides. The library currently is complete enough to run a fork of ControlTower I made called ChaosBazaar: http://github.com/headius/ChaosBazaar So, now I'm curious: have you gotten to the dispatch_source_t part of libdispatch? I wonder, because this is where I would predict the most difficulty. In OS X, the dispatch sources are made possible through the use of a kqueue, which is kinda like a select but at the kernel level. Also, there might be some difficulty in mapping the semantics directly due to the high degree of asynchrony achieved by libdispatch. For example: the following is perfectly legal: myq = Dispatch::Queue.new myq.async do myq.cancel! *keep doing stuff on 'myq'* end ...because cancelation
Re: [MacRuby-devel] Thread safety in apply example?
On Tue, Jan 25, 2011 at 2:11 AM, Charles Oliver Nutter head...@headius.com wrote: I did have to hack around the parser logic, since native extensions largely mean death for concurrency on JRuby (and by native I mean C extensions using MRI's API). Instead, I lifted code from Mongrel and Rack to use Mongrel's parser to populate a Rack environment, and managed to make the result function well enough to benchmark. I should clarify this... Mongrel has shipped a Java version of its Ragel-based parser for many years. JCD and ChaosBazaar have no native/C code whatsoever, and will run on any platform with a reasonably current JVM. I'd be happy to roll my CT/CB parser hackage back into mainline if we can find a clean way to isolate impl-specific backends. It would be easier to maintain a single CT atop multiple GCD lookalike APIs than to collaborate on largely-independent forks. - Charlie ___ MacRuby-devel mailing list MacRuby-devel@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel
Re: [MacRuby-devel] Thread safety in apply example?
On Mon, Jan 24, 2011 at 10:24 AM, Matt Massicotte massico...@apple.com wrote: My guess is that most problems need a thread-safe mechanism, where manipulating an array is just one part of a more complex system. Pushing the details of the thread-safety into the array just hides the problem, and will likely trade more locking for possibly simpler code. Which could be a very good trade, don't get me wrong. I'm not trying to knock your implementation at all. In some cases, it's a great trade. It's just not a trade-off you want to make for *all* access, which is why concurrent-threaded Ruby impls like JRuby and MacRuby don't make that guarantee for Array, Hash, etc. Of course, regular C Ruby isn't concurrent-threaded, so they have had little motivation to do *anything* to improve the state of concurrent programming on Ruby, like adding thread-safe collections or simpler synchronization mechanisms. I think the actual underlying problem here is Queue#apply's rdoc entry. Manipulating a non-thread-safe data structure (which is all of Foundation's collection classes) in an apply is a bug and even worse a conceptual error. Where do you open bugs on the dispatch gem? Queue#apply is actually defined in gcd.c in MacRuby's code, so this would be a MacRuby bug. I'd have filed it, but I wasn't sure if I was missing something about MacRuby's Array and thread-safety. - Charlie ___ MacRuby-devel mailing list MacRuby-devel@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel
Re: [MacRuby-devel] Thread safety in apply example?
On Mon, Jan 24, 2011 at 11:26 AM, Joshua Ballanco jball...@gmail.com wrote: On Mon, Jan 24, 2011 at 12:15 PM, Charles Oliver Nutter head...@headius.com wrote: Queue#apply is actually defined in gcd.c in MacRuby's code, so this would be a MacRuby bug. I'd have filed it, but I wasn't sure if I was missing something about MacRuby's Array and thread-safety. Yeah, probably a documentation bug...but only sorta. Parallel/Not-Parallel will depend on the queue. Apply just queues the block asynchronously the number of times indicated by the argument. How the block executes depends on the queue. Ok, that makes sense. The documentation implied that apply always runs the jobs in parallel, so I assumed that meant they could potentially run concurrently. If instead they're serial, the example as written should be safe. It seems like a poor example in any case, however, since someone might assume using a concurrent queue is safe as well. - Charlie ___ MacRuby-devel mailing list MacRuby-devel@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel
Re: [MacRuby-devel] Thread safety in apply example?
On Mon, Jan 24, 2011 at 11:39 AM, Jordan K. Hubbard j...@apple.com wrote: FWIW, we've tried this same approach experimentally (use a per-object serial queue behind the scenes to implement an async setter / sync getter model) over in CoreFoundation land, and it seemed to work reasonably well for the most part. We ran into some compatibility issues with a few of the methods which just didn't lend themselves to a thread-safe object model, enough that we couldn't simply do this as a completely transparent solution, but it mostly Just Worked and I think MacRuby (or, for that matter, JRuby) could do something very similar and probably get away with it since Ruby arrays/hashes/strings aren't quite so inclined to hand pointers to their internal state around to clients of the API. Laurent doesn't exactly agree with me since he's still focused on performance, but I think it should definitely be possible to declare a Ruby object as contended in some way such that everything then goes through a serial queue implicitly to access it - I'm not saying this sh ould be true for every object, just the ones for which you need thread safety. I'd be extremely surprised if the overhead frrom making an object thread safe with a queue wasn't much higher than simply synchronizing all access with normal low-level locking primitives. It might be useful to transparently proxy all calls targeting a specific object through a queue-based delegator (as I believe you or someone has rigged up in the Dispatch gem), but I'd be very surprised if it were a good generally-applicable thread-safety mechanism for mutable state. Sometimes you just need to use locks. - Charlie ___ MacRuby-devel mailing list MacRuby-devel@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel
Re: [MacRuby-devel] Thread safety in apply example?
On Mon, Jan 24, 2011 at 11:23 AM, Joshua Ballanco jball...@gmail.com wrote:
Arrays in MacRuby are not inherently thread safe. However,
Dispatch::Queue.new creates a serial queue. So, there is an error in that
each invocation of the block will execute serially. If, instead, the example
had used Dispatch::Queue.concurrent, then there would be a potential bug,
but the easiest way to work around that is:
gcdq = Dispatch::Queue.concurrent
@result = []
@result_q = Dispatch::Queue.new('@result_assignment_queue')
gcdq.apply(5) { |i| res = i * i; @result_q.async { @result[i] = res } }
p @result
Yes, forcing the updates to run serially rather than in parallel. The
poor-man's fork/join.
There's also a mixin available in the Dispatch gem which abstracts away the
need for a serial queue for assignments by monkey-patching assignments to
all happen on a serial queue.
I'm curious what you mean by this. Can you point out the code? Is it
actually attempting to rewrite local variable or instance variable or
array modifications in code?
For those wondering about the advantages to this technique, it's worth
noting that GCD has a much lower overhead than more traditional methods of
parallelism. That said, if the calculation for each iteration is small, this
is not a useful technique. However, if the calculation for each iteration is
long or (more likely) non-deterministic -- as in waiting for an external
call or a remote connection -- then this can be a useful technique to
continue doing useful work while letting iteration complete in the
background.
I suppose it's time to explain why I'm looking into this stuff at all.
I'm building a JRuby-based mimic API of the MacRuby GCD API:
https://github.com/headius/jcd
I started out using java.util.concurrent Executors directly, and later
moved to using HawtDispatch, a libdispatch-like API. The latter has
done a reasonably good job of matching the necessary dispatch_*
functions in an OO API, and for the pieces that exist the wrapping job
has been trivial.
What I have now is a mostly-complete Queue and Object, but I'm still
figuring out how to map the other libdispatch primitives to what
HawtDispatch provides. The library currently is complete enough to run
a fork of ControlTower I made called ChaosBazaar:
http://github.com/headius/ChaosBazaar
There's no real reason I shouldn't be able to support most of the
MacRuby GCD APIs using just what's available on the JVM (i.e. no
native or OS dependencies), which would allow those APIs to be used on
any platform (and GCD client code that's not otherwise bound to
MacRuby to work on JRuby without modification). I think that would be
a good thing for concurrency in the Ruby world.
- Charlie
___
MacRuby-devel mailing list
MacRuby-devel@lists.macosforge.org
http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel
Re: [MacRuby-devel] Thread safety in apply example?
On Sat, Jan 22, 2011 at 2:00 PM, macr...@djc.net wrote: Taking http://developer.apple.com/library/mac/#documentation/Cocoa/Conceptual/Multithreading/ThreadSafetySummary/ThreadSafetySummary.html as a sign, the answer appears to be: neither of these can be safe, the latter even more so since the result is unordered. Yes, that's not too surprising I guess If one is paranoid, not even sizing the array beforehand (a very good idea) is safe, unless someone can definitively state that no one (inc. MacRuby) will be modifying/moving/otherwise manipulating this mutable object while you are That's certainly not specified behavior, although I *think* it would always be safe in JRuby and MRI (don't know about MacRuby). a bit of a disappointment, since a) we know resizing occurs far less frequently then access (in general) b) making this all thread safe can easily be done at the Framework level c) most of the time, the behavior is exactly what we see here w.r.t. sizing (e.g. it grows UP, never down, and not up and down ... until GC kicks in and removes it all) In JRuby we ship a nonstandard extension JRuby::Synchronized you can include into any class (or object) to force all methods to be synchronized. The example in that case would look like: @result = [] @result.extend JRuby::Synchronized Or even better, create a SyncArray subclass that includes it. I'd like for there to be a standard way to do this across Ruby impls (in particular, I'd like a standard set of thread-safe collections in Ruby proper), but perhaps the JRuby solution will provide a starting point for everyone else. - Charlie ___ MacRuby-devel mailing list MacRuby-devel@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel
Re: [MacRuby-devel] Thread safety in apply example?
On Sat, Jan 22, 2011 at 8:52 AM, Ernest N. Prabhakar, Ph.D. prabh...@apple.com wrote: My assumption was that doing a parallel assign: result[i] = i*i would be safe, since it always accessed a unique portion of memory, but doing a serial append: result i*i would not. But that may have been a mistake on my part, since the size (at least) needs to be updated. Anyone know better? I believe you're correct about the size update being the problem. Unless the result array has all the space it needs initially, the example could easily trigger it to resize, and with concurrent threads all writing at the same time it's very likely some results will get lost. It could be improved slightly if you initialize @result with a guaranteed-sized backing store, like this: @result = Array.new(5) This would ensure the array already has a 5 element backing store prepared, and so concurrent writes would be writing different memory locations. But it still seems like a bad precedent, since the implementation details of doing concurrent writes to Array are hidden. It could be maintaining an immutable linked list structure instead of an array, in which case every random-access update would need to modify all nodes in one direction or the other, and you're back to concurrent mutation. - Charlie ___ MacRuby-devel mailing list MacRuby-devel@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel
[MacRuby-devel] Thread safety in apply example?
I'm curious about this example in Queue#apply's rdoc:
* gcdq = Dispatch::Queue.new('doc')
* @result = []
* gcdq.apply(5) {|i| @result[i] = i*i }
* p @result #= [0, 1, 4, 9, 16, 25]
apply is said to issue the jobs in parallel, so this would be making
concurrent updates to the @result array. Are simple arrays in MacRuby
thread-safe?
- Charlie
___
MacRuby-devel mailing list
MacRuby-devel@lists.macosforge.org
http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel
[MacRuby-devel] Concurrency bug in ControlTower
Apologies reporting this here; I'm not sure where I should report bugs in ControlTower. I believe there's a bug in ControlTower around line 34: def open @status = :open while (@status == :open) connection = @socket.accept # here @request_queue.async(@request_group) do The connection local variable is used in the async block below to parse the request and send the response. Unless MacRuby's blocks behave differently than regular Ruby, this variable is shared across all activations of that block. As a result, it's possible that two concurrent requests will end up using each others' connection, and usually just blowing up as a result. The fix I've come up with is to wrap the @request_queue.async call in a tap call: def open @status = :open while (@status == :open) conn = @socket.accept conn.tap do |connection| @request_queue.async(@request_group) do Since async can't accept any explicitly-passed state, this seems like the safest way to ensure the connection reference is not shared by separate invocations. I'm not sure if it's possible in GCD, but having async take an optional argument with explicitly-passed state might also be a good way to fix this. - Charlie ___ MacRuby-devel mailing list MacRuby-devel@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel
Re: [MacRuby-devel] Static Typing
On Thu, Jan 6, 2011 at 1:26 AM, Henry Maddocks henry.maddo...@gmail.com wrote: Neither of those projects have anything to do with MacRuby and as far as I can they don't have much to do with MRI either so their impact will be academic at best. The Mirah project is trying to make ruby more like Java. Well there already is a language a lot like Java, it's called Java. If you prefer the Obj-C way of doing things then feel free to use Obj-C. Me, I like [Mac]Ruby just the way it is. Mirah is not trying to make Ruby more like Java. It uses Ruby syntax and its apparent features as a starting point, and maps than as directly to the JVM's type system and libraries as possible. It is a Ruby syntax for writing JVM code. It is not Ruby. It does not intend to be Ruby. It does not intend to change Ruby. People are writing and running production applications with Mirah already, so the impact is more than academic even now. I am certainly interested in the potential of static or gradual typing in Ruby, but it's as much a research topic as anything. My use case would be to eliminate dynamic dispatches when explicit target can be determined, and to improve performance of code that does heavy numeric algorithms by making it possible to optimize them down directly to primitive CPU maths. I will repeat, though: Mirah is not a statically-typed Ruby. It is a new language that borrows Ruby's (beautiful) syntax and maps it directly to statically-typed JVM code. - Charlie ___ MacRuby-devel mailing list MacRuby-devel@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel
Re: [MacRuby-devel] Return from thread context
On Fri, Jul 10, 2009 at 3:02 PM, Laurent
Sansonettilsansone...@apple.com wrote:
Now, consider:
def foo
begin
yield
ensure
p :ok
end
end
def bar
foo { return 42 }
end
p bar
Here you don't have a choice, since you need to honor the ensure block.
In Ruby, return is semantically similar to an exception in some cases. This
is unfortunate :-)
Return isn't really an exception, since you can't catch/rescue it. The
issue here is that ensure blocks are a hard requirement, and they must
fire even if a non-local return unrolls through them. The return here
doesn't necessarily need to be implemented as an exception, but it
does need to fire ensure blocks.
In JRuby, we do use an exception, but we don't have any other way of
unrolling the stack.
- Charlie
___
MacRuby-devel mailing list
MacRuby-devel@lists.macosforge.org
http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel
Re: [MacRuby-devel] Contributions (Was: Experimental branch status)
Eloy Duran wrote: What would be great is if you could first complete the openssl part of the rubyspec, which desperately needs some love. Wow, I concur a thousand times over. I'd love to have a complete set of OpenSSL specs out there, but I'm totally underqualified to write them. - Charlie ___ MacRuby-devel mailing list MacRuby-devel@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel
Re: [MacRuby-devel] A MacRuby-users group?
Laurent Sansonetti wrote: Hi Łukasz, This mailing-list was usually created for all people developing with MacRuby, not necessarily people developing MacRuby itself. But I can understand that the group name is ambiguous. Just to let everyone know, there is absolutely no problem posting anything related to MacRuby on this list. It's also true that there are more users discussions than core discussions here, but this might change once we get more people involved in the project. I think that the list traffic is still very light so I wonder if we need 2 separate lists. Any though? If most people agree to create a -users group I can ask for an official (macosforge.org) one. Bleh...most projects I'm on have separate dev and user lists, and I hate it (including JRuby). Half the time people aren't sure which list to post to, and none of those projects (nor MacRuby) have enough traffic to really need separate lists. I'd say keep it all on one list for as long as possible. - Charlie ___ MacRuby-devel mailing list MacRuby-devel@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel
Re: [MacRuby-devel] Strings, Encodings and IO
Vincent Isambart wrote: I think everyone agrees that having a Ruby 1.9 String specs will be necessity. And we'll also need to decide what parts of it to follow and what parts we do not need to. For example handling access to characters in a string with a partly invalid encoding exactly the same way as 1.9 seems hard to do: GOD yes...any sort of complete string specs would be most welcome. Marcin Mielzynski, our porting machine, believes that oure 1.9 String stuff is basically done--and true to form the test_string tests in Ruby 1.9 repository do seem to mostly function--but I have no confidence in the coverage and completeness of the existing tests (having spent almost no time actually looking at them, though). We have also been kicking around how to transparently handlg UTF-16 by just using Java strings...but having moved to our own Oniguruma port, it would mean some regexp behaviors would have to change back to Java regex behavior. We simply could not match Ruby regex exactly until we ported the same engines Ruby uses :( - Charlie - Charlie ___ MacRuby-devel mailing list MacRuby-devel@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel
Re: [MacRuby-devel] Strings, Encodings and IO
Vincent Isambart wrote: the test_string tests in Ruby 1.9 repository do seem to mostly function You mean test/ruby/test_m17n.rb, test/ruby/test_m17n_comb.rb, test/ruby/test_io_m17n.rb and test/ruby/enc/test_*.rb? test/ruby/test_string.rb does not contain anything m17n related. We simply could not match Ruby regex exactly until we ported the same engines Ruby uses :( No two regexp engines have the same behavior, there's nothing anyone can do about that... Well, the problems we ran into is that those behavioral differences hindered our ability to run stuff like Rails. We didn't really have a choice. It looks like Oniguruma has a support for UTF-16 so I was thinking about using that in MacRuby. But as Oniguruma sees everything as a list of bytes, I do not know if you could use the Oniguruma UTF-16 support without modifying your Oniguruma port. Yes, I have talked with Marcin about us doing a separate fork of JOni that works with Java's UTF-16 characters directly. I think it could become the best Java regexp engine. - Charlie ___ MacRuby-devel mailing list MacRuby-devel@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel
Re: [MacRuby-devel] Strings, Encodings and IO
Vincent Isambart wrote: Hi again, So plan B: We emulate Ruby 1.9 strings behavior on top of of NSString/NSData. I'm really interested in this discussion too. A little background for JRuby: We started out (or really, the original authors started out) with JRuby using all Java strings and stringbuffers for Ruby's String. This made interop easy and of course simplified implementation, but it ran into the same problems you enumerated in your original post: * Java's strings are all UTF-16. In order to represent binary data, we ended up using a raw encoder/decoder and only using the bottom byte of each character. Wasteful, since every string was 2x as large, and slow, since IO had to up/downcast byte[] contents to char[] and back. * So we made a move about two years ago to using all byte[]-based strings in JRuby. This allowed us maximum compatibility, maximum performance, and a future-proof path, but it damages interop. Currently whenever you pass a string across the Ruby/Java boundary, we have to do the decode/encode. That affects performance pretty severely. We also had to implement our own regular expression engine, since no Java regex works with byte[]. * We want to move to an intermediate version, where we sometimes have a byte[]-backed string and sometimes a char[]/String-backed string. IronRuby does this already. This is, however, predicated on the idea that byte[]-based strings rarely become char[]-based strings and vice versa. I don't have any evidence for or against that yet. So it's a nearly identical problem for MacRuby, as I understand it. I'm interested in discussion around this topic, since we are still moving forward with JRuby and would like to improve interop with Java libraries. I will offer the following food for thought: * Going with 100% objc strings at first is probably a good pragmatic start. You'll have the perf/memory impact of encoding/decoding and wasteful string contents, but you should be able to get it functioning well. And since interop is a primary goal for MacRuby (where it's been somewhat secondary in JRuby) this is probably a better place to start. * We have considered having a modified 1.9 mode that normalizes all strings into UTF-16 internally. That might work for you as well. I presume there are byte-decoding APIs in objc that could produce your standard strings. You'd be able to at least pretend you support 1.9 encoding, but be transcoding to the user's selected encoding. It wouldn't be fast, but it would work. * Alternatively, you could only support a minimum set of encodings and make it explicit that internally everything would be UTF-16 or MacRoman. In MacRuby's case, I think most people would happily accept that, just as a lot of JRuby users would probably accept that everything's UTF-16 since that's what they get from Java normally. Ultimately this is the exact reason I argued over a year ago that Ruby 1.9 should introduce a separate Bytes class used for IO. I was denied. It's definitely a sticky issue, and Ruby has made it even stickier in 1.9 with arbitrary encoding support. None of the proposed solutions across all implementations (including JRuby) have really seemed ideal to me. - Charlie ___ MacRuby-devel mailing list MacRuby-devel@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel
Re: [MacRuby-devel] Running the experimental branch
Laurent Sansonetti wrote: For instance, the optimizations that are (and will be) implemented in the experimental branch are not random but were selected after having profiled a big MacRuby/Cocoa application and found many areas where we performed badly. I assume that other implementations are using similar techniques. The interesting thing about many of these microbenchmarks is that a substantial part of MacRuby's performance seems to come from one key optimization: recursive calls. MacRuby (experimental) fib(30) with fcalls (fib(...): 0.06 0.00 0.06 ( 0.053868) 0.05 0.00 0.05 ( 0.053773) 0.05 0.00 0.05 ( 0.055541) 0.05 0.00 0.05 ( 0.049992) 0.06 0.00 0.06 ( 0.054645) MacRuby (experimental) fib(30) with calls (self.fib(...)): 0.18 0.00 0.18 ( 0.182371) 0.18 0.00 0.18 ( 0.178691) 0.17 0.00 0.17 ( 0.177174) 0.17 0.00 0.17 ( 0.179554) 0.18 0.00 0.18 ( 0.181848) The latter numbers are roughly the same as JRuby performance without recursion optimization (with or without fcall). The recursion opt is actually pretty clever...I experimented with it a few weeks back (achieving similar results, http://gist.github.com/67329) and backed off temporarily to work on other things. I also wasn't sure how common it would be in real code, nor was I sure it could be done safely while maintaining all Ruby semantics. Maybe we'll be able to figure that out together. :) - Charlie ___ MacRuby-devel mailing list MacRuby-devel@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel
Re: [MacRuby-devel] branches/experimental
Matt Aimonetti wrote: Hi Charlie, I don't think/hope you do it on purpose, but it seems that you're asking questions just to prove that Laurent is wrong and that whatever he will do will end up slowing down the current experimental branch. I think you're misinterpreting me. I'd love for Laurent to be right, and I'd love to know how to get around the cases that end up slowing down JRuby. I sincerely hope it's possible, since it would mean JRuby can probably do whatever MacRuby does in those cases. And I may be able to help if some of the optimization ideas are discussed more openly; I had similar discussions with IronRuby folks at RubyConf 2007 and saved them going down a path that I knew would eventually break code. - Charlie ___ MacRuby-devel mailing list MacRuby-devel@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel
Re: [MacRuby-devel] branches/experimental
Charles Oliver Nutter wrote: Matt Aimonetti wrote: Hi Charlie, I don't think/hope you do it on purpose, but it seems that you're asking questions just to prove that Laurent is wrong and that whatever he will do will end up slowing down the current experimental branch. I think you're misinterpreting me. I'd love for Laurent to be right, and I'd love to know how to get around the cases that end up slowing down JRuby. I sincerely hope it's possible, since it would mean JRuby can probably do whatever MacRuby does in those cases. And I may be able to help if some of the optimization ideas are discussed more openly; I had similar discussions with IronRuby folks at RubyConf 2007 and saved them going down a path that I knew would eventually break code. Hmm, I just realized more openly sounded bad. I just mean I'm here if anyone wants to discuss ideas, and I might be able to help. Carry on! :) - Charlie ___ MacRuby-devel mailing list MacRuby-devel@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel
Re: [MacRuby-devel] branches/experimental
Laurent Sansonetti wrote: I don't think it's a good idea to provide a way to turn off optimizations and I do not see the point in benchmarking dead code in general (I would never do this). I think it's actually very useful to provide a way to turn off specific optimizations, if only because they may eventually run into cases where they break something. But they're also useful when writing benchmarks that have dead code on purpose... For some benchmarks it's very difficult to get a reasonable measurement without forcing some dead code to run. For example, benchmarking a single local variable access gets completely lost in the method or block invocation that surrounds it. By forcing several successive local variable accesses to execute, you get a better picture of what the actual cost is. At any rate, if you have good benchmarks for things like local variables, we can certainly use those for now. Good to know, I just hope they are not doing this 30 million times in a loop or something :-) Well, it gets called numerous times per request. In the end, though, Rails performance has not actually been very execution-bound. We've had Ruby code running faster than Ruby 1.8 for almost two years, but we only recently started to post 10-20% performance gains for Rails itself. Rails performance, and probably most large applications' performance, all seem heavily dependent on core classes being as blazing fast as possible. It's a balancing act, and we often completely ignore execution performance for a whole release to work on core classes instead. Yes, Binding is not implemented yet. Do not worry I have read the MRI source code and know how Binding works and how to provide a compliant implementation. Please stay tuned. Well, I'd certainly like to hear what you're planning for this particular case. Just let me know when you're ready to talk about it. I've gone over several options when optimizing JRuby, and the block-as-binding issue makes most of them infeasible. - Charlie ___ MacRuby-devel mailing list MacRuby-devel@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel
Re: [MacRuby-devel] branches/experimental
Laurent Sansonetti wrote:
It's possible by modifying the source code and comment the call to
createInstructionCombiningPass() and createCFGSimplificationPass(), but
I do not recommend to remove these because I think it would break the
way we compile Dwarf exception handlers in blocks.
# In my personal benchmark suite I try to make sure these optimizations
do not provide false positive numbers when comparing against YARV.
Ok, it would be nice if there were a simpler way to turn those off; I
don't want to break 'experimental' completely, but it would be nice to
get real benchmark results in these cases.
Yes, currently calling eval with a literal string will call the JIT, so
doing this in a loop will most likely eat all your memory :-)
I plan to address that later by fallbacking to the LLVM interpreter in
some cases, but I doubt this will affect real-world applications. Also,
there are ways to improve the JIT (nothing has been done yet).
In Rails, as recently as 2.2 (I haven't checked 2.3) there's a small bit
of eval'ed code used to look up constants. Before we fixed our parser
performance, we found it was a bottleneck. So that's at least one
real-world case.
Yes as you noticed Binding has not been implemented yet :-( This is on
the very top of my TODO list (needed for IRB) and I already know how to
implement it without disabling our current local variables into CPU
registers optimization.
How will you do that? Given that a block can be used as a binding, you
can't statically inspect contained blocks to determine which variables
are used and which aren't. For example, this code:
def foo
a = 1
bar { }
puts a
end
def bar(b)
eval a = 2, b.binding
end
foo
This should print out 2 but prints out 1 in 'experimental' right
now. This one feature is the primary reason JRuby can only put local
variables in registers when there's no blocks present. When there's a
block present, any variable can be accessed via its binding at any time.
I've argued for this feature to be removed, but I have been unsuccessful.
Current JRuby is also putting local variables in registers (via HotSpot
doing so for Java locals) when there's no blocks present.
- Charlie
___
MacRuby-devel mailing list
MacRuby-devel@lists.macosforge.org
http://lists.macosforge.org/mailman/listinfo.cgi/macruby-devel
