On Mon, 2009-11-23 at 22:20 -0500, M. Ranganathan wrote:
> Does this all sound feasible?
Very ambitious.
> As you (developers and testers) are going to be the primary audience
> for this tool, I would like to get some early feedback on these ideas.
>
> Than you for your thoughts in advance.
I think we want an environment in which we can create simulations that
provide all the network interactions for a test of one or more services.
It will have to include DNS requests/responses as well as SIP and
perhaps HTTP. The environment must also be able to load the
configuration data for any service and start and stop the service
instance.
I think a tool to initially create scenarios from tcpdump traces is an
interesting idea, but I suspect that generating one that's useful would
be much more work than just doing it by hand. At most, it should be a
tool that's run once to create the script which is then adjusted by the
developer to create the test scenario that runs automatically.
The notion of having the tool do "causal analysis" sounds way too hard.
It's often hard to figure out why things work the way they do when
analyzing traces by hand - building a program to do it in a meaningful
way is a multiple-Phd level undertaking, I think.
There are lots of times when what we want is to say:
- With this configuration
- And these requests
- does component X generate these responses?
We don't have that today, and so I think we should build that first
because the payoff for just that much would be huge. It should be
runnable in a batch mode with little or no training on the part of the
developer ("make regressiontest").
I also think that we should focus first on a framework capable of
testing exactly one component at a time - the complexity of configuring
and controlling multiple components is not worth the incremental
benefit. If we can regression test every network interaction of every
component separately, we'll have a far more robust system than we have
now. It's not that I don't think that doing a test of the entire system
is a bad idea - it's just that the interactions make those tests far
more complex, and put much greater demands on the configuration and
control components of the system (also, most developers don't know the
system as a whole well enough to write those tests, but do know the
component they're working on well enough to know whether or not it is
doing the right thing).
_______________________________________________
sipx-dev mailing list [email protected]
List Archive: http://list.sipfoundry.org/archive/sipx-dev
Unsubscribe: http://list.sipfoundry.org/mailman/listinfo/sipx-dev
sipXecs IP PBX -- http://www.sipfoundry.org/
