Hi John, Thanks again for you reply!
sorry, I will not use your "poc", I don't want to have Bunny/AMQP putting > sticks in my wheels. I will try to build an example that really states > "hey > it's ruote's fault". > Ok, fair enough... > In your "poc" readme, you don't mention ruote grinding to a halt. > I don't know what exactly causes our workflow application grinding to a halt. In my minimal ruote poc it is not happening. Thus it needs further investigation from my part. It seems to relate to failing participants / participants that raise an error in the concurrence block. But only when running a lot of flows simultaneously. It does not happen when I run 5 flows. Those errors are raised when the expected information is not found in other systems (or these systems are simply not available). I will investigate that further and let you know. > The perfomance section doesn't tell how many ruote workers you are running > and which storage is used for the result. > The benchmarks mentioned in the README.md file are all done with a single engine, single worker, and Redis storage. I tried FsStorage, but that was even slower. > > For me, there is not much difference whether I run the application with > > multiple engines, a single engine, or handled by multiple processes. > > Multiple workers? > I made an EngineDistributor (it's in lib/yaap/engine_distributor.rb) that starts 1 or more workflow engines, each with their own database. As there is nothing to share between different workflows, that seemed like a good idea to speed thing up to me. But that does not make a difference. That is wat I mean with multiple engines (and == multiple workers). I could not see the benefits for using EngineParticipants to use multiple workers in our case. Because we use AMQP, we can easily start more then one process (bin/app.rb) to handle the workflow requests that come in. The load will then be divided by the different processes that are running. That's what I mean with multiple processes. > > Mind that there is a 'progress' key in the workitem hash that is dumped > by > > the test script. It is really weird, but some steps seem to be started > and > > finished more then once... While that should not be the case (workflow > is > > in participants/yaap_mobile_auditor/workflows/main.conf.rb). > > That's an interesting piece of feedback. > However, I have the feeling that it is caused by the :merge_type => 'concat' > Your whole example is superbly packaged, but I find it confusing that a > workflow definition is places under participants/. > I agree... :-) That's because of historical reasons. But rest assured, it's like that only while we are working on it. The idea is to have different workflows, each of which resembles in it's own gem. That makes it very easy to update a single workflow. > In your tests, the concurrent_iterator after "step5", how many branches > does > it "spawn"? > In the example app, it does not spawn, so step 6 is never executed. In the application we are building, it is running once in 85% of the cases, and 2 or maybe max 3 times in the rest of the cases. In our case, we would like to start a lot of workflows at the same time (100 / 200) and have them run all at the same time (as much as possible) and return asap. We do a lot of IO stuff, databases / http calls etc, so that should speed things up. But I have a feeling that when I start 200 workflows with the example poc application, the first step is started 200 times. When that first step is finished for all of them, it looks like it is waiting for 10 seconds, and only then continues with the second step. It looks like it is doing reply and apply actions for 10 seconds, before it starts with the second step. The problem is still there when I remove the concurrence. Hope this provides some more information. Thanks a lot for the good work and all the help!! :-) Jarra -- -- you received this message because you are subscribed to the "ruote users" group. to post : send email to [email protected] to unsubscribe : send email to [email protected] more options : http://groups.google.com/group/openwferu-users?hl=en --- You received this message because you are subscribed to the Google Groups "ruote" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
