Darn, Sourceforge still doesnt use the "Reply-To:" field for the mailing
lists.
Sorry for the double post, Pope.
On Wed, Jan 16, 2013 at 10:18 PM, Michael T. Pope <mp...@computer.org>wrote:
> On Wed, 16 Jan 2013 07:28:13 PM Pedro Rodrigues wrote:
> > What is the current best way to evaluate changes in the AI?
> >
> > For whoever is working with AI-related code, how do you evaluate your
> > changes?
>
> I have a hairy collection of scripts that automatically runs games with 7
> AIs
> + 1 observer from a known starting configuration, mines the log files, and
> produces output like this:
>
Would it be possible to share those scripts? It would be a good starting
point for my own changes.
Regarding the starting configuration, which parameters do you consider
important?
> > On a related subject (as proper AI-testing many times rely on removing
> the
> > randomness of the system) , what was the reasoning behind the removal of
> > the PseudoRandom interface in rev.6988, and make all implementations
> > inherit from Java.util.random?
>
> [Terminological aside: I tend to use the term inherit / inheritance if
> there
> is a class hierarchy relationship, so as none of our classes actually
> extend
> Random I do not wish to use that term here. I would say we "use" Random.
> Or if this was C I would say we "include" java.util.Random.]
>
MockPseudoRandom is a subclass from random, which was my warning sign
(ironically it uses an internal random object instead of using its own
inherited classes, probably due to the PseudoRandom past.
> IIRC PseudoRandom did nothing but wrap nextInt. There were a few places
> where
> other functionality of java.util.Random would have been helpful (e.g.
> nextDouble).
It would have been easy to change the interface to add the required
methods, as well as the few classes that implemented it and having the
implementation class actually just use delegation to call the corresponding
method on the java.util.random object it would be composed with, so it was
simply a question of creating the wrapper method that delegated to the
random object method (this after the work needed, see below); a sync of the
existing client and server classes could also have been easier with this
architecture, i believe.
> At the time two other processes were either underway or
> planned--- giving each of the AIs its own RNG (so as to improve
> determinism)
> and routing all game-changing random generation through a logging function
> of
> some sort (like Utils.randomInt) so as to be able to track problems with
> random numbers (remember all the complaints we have about the combat system
> being rigged?). Its a couple of years ago now, so memory is fading, but in
> the end I expect I decided that PseudoRandom was not doing anything useful.
>
That is the thing, you looked only to the game and missed the Unit testing
angle.
> > The use of the interface is much preferable
> > as it makes it clear that several implementations can be used in a given
> > method call (thus making the code more readable), and avoids unforeseen
> > consequences when using those other implementations, as they do not have
> > inherited methods that may give different results from those intended (or
> > even desirable).
>
> This would be a good argument if we had any intention of using anything
> other
> than java.util.Random to generate random numbers. I think that would be a
> really bad idea.
Not what im getting at.
In the long run, unit testing will most likely provide a more effective way
to test for large number of use cases (obvious not all).
When unit testing code that uses random, you often need to remove such
randomness either to get some meaningful result, or force a starting
condition, with deterministic developments.
The easiest way to do so is to inject a mock object (technically a "Test
Stub"), that usually returns a predetermined value (or values), and the
most clean way to do it is to use interfaces (one of the reasons they are
preferred over abstract/concrete arguments for methods).
Another way to use it is by substituting in run time the RNG (via a debug
option) so that we can test specific results, given a starting condition.
The current approach (of having Mock objects be subclasses of Random, and
using Random as the type for components and method arguments) kinda works
but requires upcasting and downcasting, and may have unexpected results due
to interaction with the overrriden and inherited methods of Random; using
an interface instead of a concrete class removes all those problems.
I have put a lot of work into making sure that every[1]
> random number generated by FreeCol derives ultimately from the initial
> seeding
> of a single java.util.Random at game initialization time.
Which was very good work, but you ended up throwing the baby with the bath
water, as they say, i think ;)
Looking at the 0.9.2 release (the last stable one with PseudoRandom), i see
that the implementing classes (one for the server, one for the client) used
several different RNG methods, which could be one cause for such problems.
However, i must admit i didnt touch that code ever, so i may be
over-simplifying the issue
> Indeed, AFAICT, if
> the thread scheduling could be made deterministic so that the AIs always
> interacted with the server in the same order, an all-AI FreeCol game would
> be
> exactly repeatable.
That was my expectation with passing a starting seed to freecol (not
tested), sorry to hear it wont work.
> That aside, when a user says `The RNG is broken', we now
> can reply `Then so are all users of java.util.Random, everywhere'.
>
I remember the discussions; the use of the interface and the revamped
implementations would achieve the same results i believe (since it would
use java.util.Random under the hood anyway).
------------------------------------------------------------------------------
Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS,
MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current
with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft
MVPs and experts. ON SALE this month only -- learn more at:
http://p.sf.net/sfu/learnmore_122712
_______________________________________________
Freecol-developers mailing list
Freecol-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/freecol-developers
