Darren Cook wrote:
seems to me that making algorithms heavier and then demonstrating that
they are stronger with the same number of playouts misses the point -
why would one not run an experiment under the same time conditions instead?
* The heavier algorithm might be unoptimized;
That is probably a good pragmatic point, but that does not make the
experiment any more valid. You are already designing the algorithm so
that it performs better at the same number of playouts, by trading off
speed. I don't think it is then valid to also use that as proof that it
performs better - the experiment is slanted against the lighter, faster
algorithm.
Unfortunately, "it is unoptimized", while usually a good argument,
interferes with one of the main dimensions of the experiment, which is
speed.
* The heavier algorithm might be easier to parallelize (or put into
hardware);
That seems unintuitive, but I confess I have no experience whatsoever in
that respect.
* The scaling behaviour might be different. E.g. if fuego and Valkyria
are both run with 10 times more playouts the win rate might change. Just
to dismiss an algorithm that loses at time limits that happen to suit
rapid testing on today's hardware could mean we miss out on the ideal
algorithm for tomorrow's hardware. (*)
I am not sure I follow this argument. How do you intend to prove that,
unless you run the algorithms with 10 times more playouts? In that case,
I would still argue that you should run them with X times longer time
limits, not with 10 times more playouts, unless you can assume (with
proof or good evidence, so you can set differential numbers of playouts
between the two) that tomorrow's hardware will favour one algorithm more
than the other.
* By analyzing why the win rate of the super-heavy algorithm is better
might give other people ideas for lighter but still effective playouts.
This I can accept. In general, I do think it is interesting to see the
win rate at the same number of playouts as background analysis, but I
don't see it as a convincing evidence of advancement over other algorithms.
Christian
_______________________________________________
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/
