On Mon, Jul 14, 2008 at 1:05 PM, Subrata Modak <[EMAIL PROTECTED]>
wrote:
> Dear Hisashi San,
>
> Since we have already embarked on our path to mutual co-operation in
> Linux testing and test cases development, i would like to discuss some
> of my ideas, which i think is capable of adding shot-in-the arm for
> linux testing.
>
> I want to talk about Test Driven Development for Linux Kernel Testing,
> which can bring us very early effective results and can catch Bug(s), if
> any, in the kernel code much much early, and even before the code makes
> it´s way through the stable kernel release.
>
> The idea is to submit the test cases to LTP, for certain kernel
> features, which makes its way through the kernel-mm tree. When a
> developer submits his/her feature to mm-kernel, he gives a specification
> of the features. Depending on the feature specification available either
> from the developer himself, or, from manpages, if it is available that
> soon, tests cases will be written.
>From now on, I'll be agitating more to get man pages provided more with new
syscalls and ther kernel-userland interfaces. That will mean either I twist
developers arms to write pages ;-), or I write them myself, with help from
them. I do think that man-pages, if well written, are often sufficient as
(or at least a very good base for) a test specification. Here's an example
that I did with the timerfd API, finding two bugs in the process:
http://thread.gmane.org/gmane.linux.kernel/613442 . I did something similar
while writing the utimensat(2) man page, finding 5 or 6 different bugs in
the end, see
http://linux-man-pages.blogspot.com/2008/06/whats-wrong-with-kernel-userland_30.html
.
> Now, we have an inherent question,
> who writes and submits those test cases ?? Either it can be the
> developer himself, or, somebody from the test community.
Or, IMO, preferably both, working together.
> Since, the
> developer is not bound to submit any test case,
I can't help but feel that part of the solution here is to agitate more so
that the developer is (strongly) encouraged to submit test cases along with
patches that change the kernel-userspace interface. As things stand, there
are far too many bugs in released interrfaces.
> we can assume we need to
> initiate doing something in this regard.
>
> Having said that, i want to propose that, since in your project you have
> already taken up the Job of making available the system call test cases
> up-to-date, we can score a point here. Whenever a new system call code
> enters the mm-tree in the kernel, somebody in your project can write the
> test case(s) immediately and make it available to:
>
> 1) Your Project,
> 2) LTP,
> 3) LKML,
I would also be very happy to see any test cases you produce for new system
calls. What list do I need to subscribe to?
> We will use the test case(s) to test that particular syscall feature and
> report Bug(s) to the developer(s) and LKML. Meanwhile, updates to the
> test cases will also go on depending on feedback from community. With
> that we will address the following issues:
> 1) Catch bug at the earliest for the new syscalls in the kernel, so that
> the cost of Bug fixing, when something is discovered later, can be
> hugely reduced,
Bug fixing is perhaps the lesser problem. The greater problem is that if
there are bugs, we must change the ABI to fix them, and if that is done
after a stable kernel release, then userland programmer's can end up having
to do things like special case their code according to the kernel version.
> 2) We have a regular and systematic flow of test cases for both LTP and
> Crackerjack. This process becomes institutional and gurantees us maximum
> and continued code coverage for syscalls in future,
> 3) LTP-mm tree is born,
> 4) Others/developers get motivated to submit test cases(to LTP and
> Crackerjack) at the time of submission of kernel features to
> kernel-mm-tree, if we show them the benefit of early testing and it´s
> effectiveness of controlling more bugs to seep in before it is too late
> to fix.
>
> For this to happen, we need to make our test cases ready when the
> corresponding kernel features are in -mm tree. Let me know your ideas on
> this.
Part of the problem here is knowing when interface changes have occurred.
Seeing new system calls in a release is easy. But, for the project to be
truly effective, all kernel-userland interfacechanges need to be checked
for, including
system calls
ioctl()s
netlink
/proc
/sys (okay -- that's a big ask at the moment)
virtual /dev interfaces used for kernel-uerland communication
And probably other things I forgot right now.
Cheers,
Michael
On Mon, 2008-07-14 at 13:39 +0900, [EMAIL PROTECTED]
wrote:
> Subrata,
>
> >Masatake is already working to port the missing syscall test cases in
> >LTP from the Crackerjack Project to LTP. And it is already a huge
> I am glad to hear that.
> >effort. I think the idea here is great as we are not re-inventing the
> Yes, you can do this and we can use our time to develope more test cases.
> >to port from you, unless somebody in LTP community has already submitted
> In that case, we will refer your cases and make them better.
> >https://lists.sourceforge.net/mailman/listinfo/ltp-list
> I joined now. I am very surprised that your community are
> working so active. I will check and I will post anything
> I can collaborate.
> >We will be collaborating as far as we can, as all of us striving to
> >Linux better.
> Yes, agree. We will continue this effort for our purpose, Linux better.
> Please post anything to both our mailing list and cc-d your mailing list
> when you have to contact me.
>
> Thanks,
>
> Hisashi
>
> >送信者 ： Subrata Modak <[EMAIL PROTECTED]>
> >主題 ： Re: Re[2]: Re[2]: Re[2]: Re[2]: [Crackerjack-devel] [LTP]Crackerjack
andLinuxTes
> >受信日 ：08/07/12 04:10
> >属性 ： なし
> >
> >Dear Hisashi Hashimoto San,
> >
> >Thank you for writing back to us. We are delighted to colloborate on all
> >fronts to make Linux testing better. My comments embedded in the
> >following lines.
> >
> >On Fri, 2008-07-11 at 13:53 +0900, [EMAIL PROTECTED]
> >wrote:
> >> Subrata,
> >>
> >> Sorry for long time no reply.
> >> We had internal meeting of our project
> >> regarding how to proceed.
> >> Current our plan is:
> >> (1) Unfortunattelly, our test cases do not cover all system call yet.
> >> Total number of system call in Kernel 2.6.26 is almost 326.
> >> But, we developed around 299.
> >
> >Masatake is already working to port the missing syscall test cases in
> >LTP from the Crackerjack Project to LTP. And it is already a huge
> >effort. I think the idea here is great as we are not re-inventing the
> >same wheel. We can use your test cases released under GPLv2 and port to
> >LTP format.
> >
> >We gain by getting more kernel test cases in LTP without the need to
> >massively write things. And you retain your goal of providing a
> >regression comparisn framework. In future, whenever the no. of syscalls
> >goes increasing in kernels, and as soon as you write them ,we will try
> >to port from you, unless somebody in LTP community has already submitted
> >a test case in that regard. Or, even we can concentrate to write test
> >cases in some other are as well.
> >
> >Presently, the first job is to port the existing ones from you to us.
> >
> >> Therefore, we would like to start developing in soon.
> >> The starting date would be some time in August.
> >> (2) Some of our developed test cases do not meet our quality criteria.
> >> As you know, we set our quality goal as covarage ratio more than
> >> yours.
> >> We are now checking all tests again. And if necessary, we need
to
> >> enhance test cases.
> >
> >Good to know that. As and when you do this, we might also relook into
> >ours and change accordingly.
> >
> >> (3) Through our acitivity (1) and (2), we would like to exchange
> >> information and technical/engineering discussion on this mailing
list.
> >
> >Definitely yes. All of you can subscribe to the ltp mailing list:
> >https://lists.sourceforge.net/mailman/listinfo/ltp-list
> >
> >And please add us as CC to all the technical discussion you have. Guys
> >in ltp mailing list will need to subscribe yours if they want to reply
> >back to you.
> >
> >>
> >> Yesterday, 4 Maintainers(Andrew Morton, Paul Moor, James Morris, Thomas
Gleixner)
> >> visit Japan to make presentation regarding the latest kernel/security
activity.
> >> I had opportunity to make presentation to them about our current
status.
> >> I present that
> >> (1) We released test cases 300/326.
> >> (2) We plan to make complete set and enhance quality.
> >> (3) We continue to talk with other test project(LTP as you,
AutoTest:Martin Bleigh),
> >> and exchange information and status, and others.
> >> Maintainers said/recommend to expand the target, to continue the effort
> >> and collaborate with LTP and so on.
> >
> >We will be collaborating as far as we can, as all of us striving to
> >Linux better.
> >
> >
> >Regards--
> >Subrata
> >
> >>
> >> I hope you and we can do the collaboration work on test project.
> >>
> >> Best,
> >>
> >> Hashimoto Hisashi
> >
> >
> >
> Hisashi Hashimoto
> Section Manager
> Open Source Software Promotion Center
> Platform Software
> Hitachi, Ltd., Software Division
> Tel : +81-45-862-8424, Fax : +81-45-862-9047
> [EMAIL PROTECTED]
--
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
man-pages online: http://www.kernel.org/doc/man-pages/online_pages.html
Found a bug? http://www.kernel.org/doc/man-pages/reporting_bugs.html
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list
