Linux-Development-Sys Digest #700, Volume #8 Mon, 7 May 01 23:13:10 EDT
Contents:
A "loopback" device for audio? (Tim Shoppa)
Re: How to get a number of processors (Ronald Cole)
Re: A "loopback" device for audio? (Erik de Castro Lopo)
Re: file_operations changed radically in 2.4!!?? (Luc Van Oostenryck)
Re: How to get a number of processors (John Beardmore)
Re: How to get a number of processors (John Beardmore)
Re: How to get a number of processors (Luc Van Oostenryck)
SCSI emulation for CD-R ("Philip Lee")
Re: Linux, streams and the standard library (David Konerding)
Re: How to get a number of processors (Kaz Kylheku)
----------------------------------------------------------------------------
Date: Mon, 07 May 2001 18:24:58 -0400
From: Tim Shoppa <[EMAIL PROTECTED]>
Subject: A "loopback" device for audio?
I (and I'm sure others here too) have proprietary (e.g. RealAudio)
players that will stream data in over the net, decode it, and play
it out to /dev/audio or /dev/dsp. But they *will* not allow
me to save the audio to disk for later playing - a significant
deficiency when I want to time-shift my listening from a live-only
audio service.
What would solve my problems is a "loopback" device for audio, such
that the RealAudio player would *think* it was going to /dev/dsp, but
really it was going to a file on disk. Does such a driver already
exist? If so, where can I find it?
Obviously, writing one myself would be straightforward, but I'd
rather not reinvent such an obviously useful thing if one already
existed!
Tim.
------------------------------
From: Ronald Cole <[EMAIL PROTECTED]>
Crossposted-To: comp.os.linux.development.apps
Subject: Re: How to get a number of processors
Date: 07 May 2001 15:27:06 -0700
Chris <[EMAIL PROTECTED]> writes:
> Nope... it's present in 2.0.7, certainly.
As was pointed out to me when I fashioned a fix for the broken SMP ps
on RHL6.0, way back when...
--
Forte International, P.O. Box 1412, Ridgecrest, CA 93556-1412
Ronald Cole <[EMAIL PROTECTED]> Phone: (760) 499-9142
President, CEO Fax: (760) 499-9152
My GPG fingerprint: C3AF 4BE9 BEA6 F1C2 B084 4A88 8851 E6C8 69E3 B00B
------------------------------
Date: Mon, 07 May 2001 10:52:42 +1000
From: Erik de Castro Lopo <[EMAIL PROTECTED]>
Subject: Re: A "loopback" device for audio?
Tim Shoppa wrote:
>
> I (and I'm sure others here too) have proprietary (e.g. RealAudio)
> players that will stream data in over the net, decode it, and play
> it out to /dev/audio or /dev/dsp. But they *will* not allow
> me to save the audio to disk for later playing - a significant
> deficiency when I want to time-shift my listening from a live-only
> audio service.
>
> What would solve my problems is a "loopback" device for audio, such
> that the RealAudio player would *think* it was going to /dev/dsp, but
> really it was going to a file on disk. Does such a driver already
> exist? If so, where can I find it?
>
> Obviously, writing one myself would be straightforward, but I'd
> rather not reinvent such an obviously useful thing if one already
> existed!
http://www.zip.com.au/~erikd/vsound/
Erik
--
=================================================================
Erik de Castro Lopo [EMAIL PROTECTED] (Yes, its valid)
=================================================================
------------------------------
Date: Tue, 08 May 2001 00:51:14 +0200
From: Luc Van Oostenryck <[EMAIL PROTECTED]>
Reply-To: [EMAIL PROTECTED]
Subject: Re: file_operations changed radically in 2.4!!??
yin wang wrote:
>
> Hey,
> I found that file_operations struct in <linux/fs.h> changed too much in
> 2.4 kernel source. It seems they added a first element named "module owner"
> to it. This will make every module change their code to be compatable to
> this!
You can, if you want, ignore this field; if the struct is decalred as
static like it is normally
it will simply be initialzed to a nul pointer; but then you can't have
the benefit of it
(you must do the MOD_INC & MOD_DEC "manually")
--
Luc Van Oostenryck
Do not use the address in the 'reply-to' field, but this instead:
<luc dot vanoostenryck at belgacom dot net>
------------------------------
From: John Beardmore <[EMAIL PROTECTED]>
Crossposted-To: comp.os.linux.development.apps
Subject: Re: How to get a number of processors
Date: Tue, 8 May 2001 00:04:33 +0100
In message <[EMAIL PROTECTED]>, Stefaan A Eeckels
<[EMAIL PROTECTED]> writes
>In article <[EMAIL PROTECTED]>,
> John Beardmore <[EMAIL PROTECTED]> writes:
>> In message <[EMAIL PROTECTED]>, Stefaan A Eeckels
>> <[EMAIL PROTECTED]> writes
>>
>>>Maybe the OS should provide a service to ensure that certain
>>>processes get a minimum of CPU time. The approach you describe
>>>is a hack, and goes completely against the basic Unix concept
>>>of hiding the hardware differences and specifics from the
>>>applications.
>>
>> Guaranteeing resources to processes may be a good tool, but it isn't an
>> alternative to knowing how many threads to create for a big crunching
>> job. For that, you still need to know the number of available
>> processors.
>
>Why not tell the OS you want the number of parallel instances
>optimal for the kit and job mix on hand?
Good long term plan. Not a short term option.
Cheers, J/.
--
John Beardmore
------------------------------
From: John Beardmore <[EMAIL PROTECTED]>
Crossposted-To: comp.os.linux.development.apps
Subject: Re: How to get a number of processors
Date: Tue, 8 May 2001 00:01:53 +0100
In message <[EMAIL PROTECTED]>, Stefaan A Eeckels
<[EMAIL PROTECTED]> writes
>In article <[EMAIL PROTECTED]>,
> John Beardmore <[EMAIL PROTECTED]> writes:
>> In message <[EMAIL PROTECTED]>, Stefaan A Eeckels
>> <[EMAIL PROTECTED]> writes
>>>In article <[EMAIL PROTECTED]>,
>>> [EMAIL PROTECTED] (Dave Blake) writes:
>>>> Eric P. McCoy <[EMAIL PROTECTED]> wrote:
>>>>
>>>>> This strikes me as a battle of bad ideas: I hate writing a
>>>>> text parser to deal with /proc; I don't like using nonstandard
>>>>> pieces of code; and no program should ever need to know how
>>>>> many processors are in a given box. There are cases where
>>>>> you'd want to use one or all of these bad ideas, but I, for
>>>>> one, would need a pressing reason.
>>>>
>>>> Suppose I am writing a data crunching piece of software that
>>>> parallelizes easily, and wish to run a thread on each processor.
>>>>
>>>> I first parse /proc/stat, and then crunch away with a thread
>>>> on each CPU.
>>>>
>>>> For a web searching program, you may wish to know the number of
>>>> NICs and CPUs, and take the lower of the two as the number of
>>>> threads to run. And so on.
>>>
>>>In a Unix system, the application should not need to know
>>>anything about the hardware details.
>>
>> <dOGMA aLERT !>
>>
>> Well then, in that case, why don't YOU start the project to make gcc
>> exploit all possible opportunities for parallelism ?
>
>Maybe because I don't have a need for them?
Well, that's fair, but why preclude or compromise the use threads that
other people my choose to make in an SMP environment ?
>>> The recent obsession
>>>with threads violates that basic tenet.
>>
>> You sound if you find threads morally objectionable as opposed to 'just
>> another way to get the job done'.
>
>No. It's a step back from the level of hardware abstraction
>that set Unix aside from the OSes of its generation.
OK, but in many ways the world has moved on. SMP is common place, but
many unix implementations don't share processors well. Letting the
application dynamically select the number of threads it's using, while a
very incomplete solution, is better than having to configure this by
hand.
> Worrying
>about the differences between hardware is typically something
>that needs to be done by the OS;
That's right. TYPICALLY.
> putting it in an application
>is bound to yield suboptimal, or even wrong results when the
>hardware doesn't match the assumptions made by the application.
That's why it's better for the hardware to have good information through
an API rather than a compile time option.
>And as applications typically outlive hardware by many, many
>years (and maintenance programmers couldn't bother less), it's
>a bad strategy.
I agree that in many circumstances it may be poor, but it's still better
than guessing the number of threads at compile time.
>>> If one wants to
>>>squeeze the last ounce of performance from a box,
>>
>> But it's not the 'last ounce' ! On some boxes it's most of the ounces !
>>
>>
>>> don't
>>>use an OS.
>>
>> Oh balls.
>>
>> 'Use an OS and a compiler that knows about parallelism'
>
>Then you will get most of the ounces, but not the last ounce,
>agreed.
No. I have a three processor alpha box here, running a market
simulation that can exploit multiple threads. If I force it down to
single thread as it seem you'd like, it goes at less than half speed.
>And I also agree that the OS (and associated tools) should be
>capable of using the hardware to a reasonable extend,
Well, C and Linux are not. So what's wrong with helping them a bit ?
> and do
>so in a way that ensures that the application is not burdened
>with irrelevant details such as the number of processors, the
>size of the memory, or for that matter, the layout of the blocks
>on a disk drive.
Well that's a nice idea, but the number of processors is relevant to any
application where the compiler does not automatically parallelise the
task.
>COBOL has statements that allow some control over the layout
>and blocking factor of a file on disk. Leaving it to the OS
>wasn't an option, as OS and hardware _needed_ help in putting
>the file on disk in a not too inefficient manner. Suggesting
>that such should be possible on today's disks would be met with
>hoots of derision.
Yes, but efficient disk access is pretty much a solved problem.
By contrast, parallel algorithms are relatively rare, relatively hard to
implement, not well supported by most tools or OS.
Most Unix systems don't even balance loads dynamically. Has 'hot
process migration' got to Linux yet ?
>I'm merely suggesting that scheduling threads, worrying about
>synchronization, and adapting the behaviour of an application
>to the number of processors _should_ be handled by the OS,
Well it isn't, and reworking the app is quicker than reworking the OS !
> and
>that the Unix approach is exactly to hide details of hardware
>from the applications ( everything is a file, remember?).
Well, pretty much everything can be viewed through the file system.
So what's your big thing point ? Are threads files ? Are processes
files ? Is a CPU a file ?
Maybe the _problem_ is that insist on seeing everything as a file ?
>> might be a better assertion, but pausing briefly to live in the real
>> world, C is not terribly 'parallel aware', and gcc is what most us here
>> want to work with.
>
>Correct. And the solution is _not_ to add a syscall to Unix
>to get at the number of processors, but to find a way to
>avoid having to know how many processors we've got, like we
>don't want to know how fast a disk spins to decide how we'll
>open a file, or how many dots per inch a printer has before
>we generate PostScript.
Well, we may not want to know those things, but sometimes we have to.
Hell, it might even be handy for an OS to be able to tell a process that
it's going to have to swap if the process goes on allocating. Gives the
process a chance to make different space speed tradeoffs...
>> I don't see any moral problem with C and Linux supporting threads. Even
>> on single CPU machines this can speed up some IO operations with
>> simultaneous reads on more than one device for example. I really don't
>> see the point in coming over all 'closed minded' about it.
>
>Why do you interpret my remark as "close mindedness"?
You said:
"In a Unix system, the application should not need to know
anything about the hardware details."
That may be true of a single thread number crunching situation, but most
many apps and server processors are configured in the light of
which disks have space
which disks are fast
what ports are used for which devices
what network bandwidth is available on what card etc.
Good system management usually comes from an appreciation of the
hardware in the systems and using it appropriately.
You seem quite reluctant to let apps have any clues that might help to
automate any of that.
> Maybe
>this is an area where Linux can further the state of the art,
>and prove that the Unix paradigm can handle multi-processors
>and threading in a betterm more portable and more transparent
>way than POSIX threads and querying the number of CPUs in a box.
Well maybe it is, but for people who just want their app to be able to
pick an appropriate number of threads to run on a more or less dedicated
box, a way of knowing how many processors you have is a lot quicker to
implement than //Linux and //g++ !
>> Now if you're going to make the number of threads equal the number of
>> processors for some crunching task, WTF is wrong with the OS making that
>> info available ?? It's only one integer and nobody's forcing YOU to use
>> it if you object on religious grounds.
>
>Because on any multi-tasking box, the presence of persitent
>service processes, and other applications, this is an example
>in misguided micro-optimization.
Not if it makes the job you want to go faster !
I grant I'm assuming that the box is more or less dedicated to the task
in hand, but that's not uncommon. Loads of other tasks may run, but
they may not use much resource !
> What you should tell the OS
>is that you want to run as many threads as possible. And maybe
>you shouldn't have to worry about threads either; indicating
>what you want the OS to run simultaneously should suffice.
Yes - let's hope things evolve that way. In the mean time more skill
and 'dirty information' will be required to tune systems. But trust me,
people will tune the systems they've got rather than going off to write
//Linux etc...
Cheers, J/.
--
John Beardmore
------------------------------
Date: Tue, 08 May 2001 01:07:54 +0200
From: Luc Van Oostenryck <[EMAIL PROTECTED]>
Reply-To: [EMAIL PROTECTED]
Crossposted-To: comp.os.linux.development.apps
Subject: Re: How to get a number of processors
Stefaan A Eeckels wrote:
>
> In article <[EMAIL PROTECTED]>,
> John Beardmore <[EMAIL PROTECTED]> writes:
> > In message <[EMAIL PROTECTED]>, Stefaan A Eeckels
> > <[EMAIL PROTECTED]> writes
> >
> >>Maybe the OS should provide a service to ensure that certain
> >>processes get a minimum of CPU time. The approach you describe
> >>is a hack, and goes completely against the basic Unix concept
> >>of hiding the hardware differences and specifics from the
> >>applications.
> >
> > Guaranteeing resources to processes may be a good tool, but it isn't an
> > alternative to knowing how many threads to create for a big crunching
> > job. For that, you still need to know the number of available
> > processors.
>
> Why not tell the OS you want the number of parallel instances
> optimal for the kit and job mix on hand?
>
> --
> Stefaan
> --
> How's it supposed to get the respect of management if you've got just
> one guy working on the project? It's much more impressive to have a
> battery of programmers slaving away. -- Jeffrey Hobbs (comp.lang.tcl)
And how you "specify to the machine" what is kit and job mix ????
--
Luc Van Oostenryck
Do not use the address in the 'reply-to' field, but this instead:
<luc dot vanoostenryck at belgacom dot net>
------------------------------
From: "Philip Lee" <[EMAIL PROTECTED]>
Subject: SCSI emulation for CD-R
Date: Mon, 7 May 2001 20:33:02 -0400
I'm really in a bind because I have no way to tell linux to start my cd-r
with the ide-scsi driver. The only way I can find is with lilo and editing
the lilo.conf file to say append="hdd=ide-scsi". I use loadlin and have no
clue what to edit. Please help.
------------------------------
From: [EMAIL PROTECTED] (David Konerding)
Crossposted-To: comp.os.linux.alpha,comp.os.linux.setup
Subject: Re: Linux, streams and the standard library
Date: 8 May 2001 02:04:07 GMT
Reply-To: [EMAIL PROTECTED]
On 29 Apr 2001 20:54:46 +0100, Philip Armstrong <[EMAIL PROTECTED]> wrote:
> In article <[EMAIL PROTECTED]>,
> Neil Butterworth <[EMAIL PROTECTED]> wrote:
>>"John Beardmore" <[EMAIL PROTECTED]> wrote in message
>>news:[EMAIL PROTECTED]...
>>
>>[snip]
>>>
>>> So - a couple of questions:
>>>
>>> 1) is there a way to get ostringstream to work ?
>>
>>Yes - use STLport. You can download it from www.stlport.org.
>
> Or install gcc-2.95.3 which has ostringstream in its version
> of the STL.
>
>>> 2) is the Standard C++ library ready for prime time under Linux ?
>>> Isn't its
>>> use utterly routine these days ? Or am I trying to be too modern ?
>>
>>No, I've been using it for quite some time with absolutely no problems. Not
>>under RedHat though - I run SuSE.
>
> Indeed. With gcc-2.95.3 the STL support is pretty complete. gcc-3.0
> should finish off all the corner cases in the c++ support hopefully.
Not entirely true. STL support may be nearly complete (thanks to SGI)
but the standard library isn't. STL is just a part of the standard library
and ostringstream isn't part of STL (it's part of IOStreams). I've been playing with
gcc-2.95.3 ad gcc-3.0pre (from CVS) to test their standard C++ library. It's far
from as good as STLport. I get segfaults when using ostringstreams heavily--
I ran out of time figuring out the minimal test case which reproduces the problem,
mailing them in, waiting for them
to be patched, downloading new versions of the compiler, recompiling
the compiler and the library (1-2 hrs on a fast machine!), repeat. THat's why I
responded earlier
saying "I'm going to save you a lot of time" because 2.95.3 + STLport-4.0 is really
stable & solid.
I did spend a couple weeks chasing a really tricky bug which we were certain was
STLport-4.0 but
actually turned out to be a problem with the run time linker (if you make a C app that
dlopen()s two
shared object files, each of which is linked dynamically gainst STLport shared library,
you get crashes in IOStreams, because the run-time linker doesn't resolve symbols
properly when
it loads the second .so. The main C app (python) specifically does not like havig
RTLD_GLOBAL set
when it dlopen()s the .so's, so it's not surprising we have this behavior (what's
surprising
is that IRIX and Tru64 don't have this "problem". We had to work
around it using LD_PRELOAD-- yecch). But C++ dlls that get loaded
into C programs are fairly esoteric, hopefully not an issue for most programmers.
------------------------------
From: [EMAIL PROTECTED] (Kaz Kylheku)
Crossposted-To: comp.os.linux.development.apps
Subject: Re: How to get a number of processors
Reply-To: [EMAIL PROTECTED]
Date: Tue, 08 May 2001 02:15:36 GMT
On Mon, 7 May 2001 22:53:25 +0200, Stefaan A Eeckels <[EMAIL PROTECTED]>
wrote:
>> You sound if you find threads morally objectionable as opposed to 'just
>> another way to get the job done'.
>
>No. It's a step back from the level of hardware abstraction
>that set Unix aside from the OSes of its generation. Worrying
>about the differences between hardware is typically something
>that needs to be done by the OS; putting it in an application
>is bound to yield suboptimal, or even wrong results when the
>hardware doesn't match the assumptions made by the application.
But you said that if you want the last ounce of performance,
you should not use an OS.
>Because on any multi-tasking box, the presence of persitent
>service processes, and other applications, this is an example
>in misguided micro-optimization.
Dividing an application into tasks is not a form of hardware-specific
optimization. It's an algorithmic change. There are certain things the
application can do when it's threaded which it could not otherwise do, such as
interrupt low priority processing to take care of a high priority input.
Sometimes threading is done specifically with the intent that the software
abstraction of threading will be mapped to actual parallel hardware. This is
by no means the only way in which threading is used, because threads have other
benefits.
In such number crunching programs, it's best to leave the number of threads as
a parameter that can be tuned without recompiling the program. This way the
best number of threads can be discovered empirically for a given
multiprocessor. On a uniprocessor, it will invariably be found that
no benefit is derived in having more than one thread in such a number
crunching program, whereas on some it may be found that six is not
enough. So it would indeed be a hardware-specific assumption to hard-code that
number to six.
>What you should tell the OS
>is that you want to run as many threads as possible. And maybe
This is wishful thinking, because dividing a task into threads is not trivial.
If you tell the OS that you want to run as many threads as possible, the
problem is that it may just do exactly what you asked for.
>you shouldn't have to worry about threads either; indicating
>what you want the OS to run simultaneously should suffice.
If you indicate what you want to run simultaneously, you are then doing
explicit threading. What you really seem to be saying is that you don't want to
do explicit threading; in other words, leave it to the system to figure out
what should be independently scheduled.
The problem is that there are so many things that could be independently
scheduled, the hard part is deciding what *not* to schedule this way.
In principle, you could independently schedule each elementary expression in a
program, using synchronization to ensure that the minimum sequencing
constraints are met, and let the scheduler sort it out! ;)
------------------------------
** FOR YOUR REFERENCE **
The service address, to which questions about the list itself and requests
to be added to or deleted from it should be directed, is:
Internet: [EMAIL PROTECTED]
You can send mail to the entire list by posting to the
comp.os.linux.development.system newsgroup.
Linux may be obtained via one of these FTP sites:
ftp.funet.fi pub/Linux
tsx-11.mit.edu pub/linux
sunsite.unc.edu pub/Linux
End of Linux-Development-System Digest
******************************
