Was any of this done with OS X but using a file system other than HFS+? I doubt any of those other OSs would be using this file system but it's extremely likely that any OS X machine is using it.
Sent from my iPhone > On 2015/03/05, at 13:00, Brian Willoughby <[email protected]> wrote: > > Hi Paul, > > I cannot explain why your code works so well on Linux, Windows, and Virtual > Windows, but not OSX. I do have a lot of experience with third-party embedded > hardware platforms for audio recording. One in particular is ARM-based (I > believe) and acts as a USB Host for compatible USB hard drives. The thing I > noticed there is that once you get beyond 16 tracks, it becomes very > difficult to maintain I/O bandwidth on a typical USB drive, even with massive > RAM buffering. There are even examples of drives with periodic drops of > performance - and the hardware platform maker explained that this is a common > "feature" of USB drives, although I suppose it's really an aspect of the raw > drive and not the USB part. > > The only workable solution was to create multi-channel WAV or AIFF files. > This reduces the seek times because only one file is being written and all > samples are interleaved. This approach allows up to 24 or 32 channels of hard > disk recording to succeed. > > Their API allowed custom configuration of the number of files and the number > of channels per file. At one extreme, the Pro Tools standard of one WAV per > channel is configurable, but performance could not be maintained all the way > to 24 tracks. 16 tracks would work, but somewhere above that it would fail > and certainly before 24 tracks. This was all tested at 48 kHz, so higher > sample rates would pose more of an issue. > > The configuration also allowed multiple stereo files, multiple 8-channel > files, or a single file with all channels. Their only restriction was that > each file must contain contiguous channels, but there was no rule preventing > an assortment of mono, stereo, and multichannel files to be recorded > simultaneously. > > I realize that nothing I've written has much to do with Ardour on OSX, but it > sure seems like you're hitting the same limitations. Have you noticed any > correlation between the type of drive, transport (USB, FW, SCSI, PCI), or > make and model of drives? I suppose your Virtual Windows on OSX test was > necessarily using the same drive - but perhaps not. All I know is that modern > drives do tend to "check out" periodically, and this is causing digital audio > hardware developers no end of grief. > > Brian Willoughby > Sound Consulting > > p.s. There are Classic Mac programmers that I've spoken with who were adamant > that there are certain ways to handle these sorts of files, and the claim was > that typical Unix calls were exactly the wrong way. Those comments baffled me > at the time I heard them, since I know Unix servers can achieve high > bandwidth. But perhaps some of this DNA lives on in the OSX implementation, > and only the old-school Classic Mac programmers know how to access it. It's > certainly true that all of the major DAW titles that you've comparing against > started out as Classic Mac apps before being ported to OSX. Perhaps that old > DNA lives on and allows them to achieve the bandwidth that is so much easily > on the other operating systems. > > I'm hoping that one of those Classic Mac programmers will see your thread and > chime in with the right recipe. > > >> On Mar 3, 2015, at 4:20 PM, Paul Davis <[email protected]> wrote: >> But there's an area where we've really run into a brick wall, despite the >> extremely deep and amazingly talented pool of people we have as developers, >> people who know *nix operating systems inside out and upside down. >> >> That area is: getting acceptable disk I/O bandwidth on OS X when reading >> many files. For comparison: we can easily handle 1024 tracks on a single >> spinning disk on Linux. We can get close to similar performance on Windows. >> We can even (gasp!) get in the same ballpark on Windows running as a guest >> OS on OS X (i.e. the entire Windows filesystem is just one file from OS X's >> perspective). >> >> But on OS X itself, we are unable to get consistent, reliable performance >> when reading many files at once. Our code is, of course, entirely >> cross-platform, and so we are not using any Apple specific APIs (or >> Linux-specific or Windows-specific). The same code works well for recording >> (writing) files (though OS X does still perform worse than other platforms). >> >> We've used various dtrace-based tools to try to analyse what is going on: >> these only confirmed for us that we have a problem, but provided no insights >> into what. >> >> Things became so inexplicable that we decided to break out the problem from >> Ardour itself and write a small test application that would let us easily >> collect data from many different systems. We've now tested on the order of a >> half-dozen different OS X systems, and they all show the same basic bad >> behaviour: >> >> * heavy dependence of sustained streaming bandwidth on the number of >> files being read >> (i.e. sustained streaming bandwidth is high when reading 10 files, >> but can be very low >> when reading 128 files; This dependence is low on Windows >> and non-existent >> on Linux) >> >> * periodic drops of sustained streaming bandwidth of as much a factor of >> 50, which can >> last for several seconds (e.g a disk that can peak at 100MB/sec >> fails to deliver >> better than 5MB/sec for a noticeable period). >> >> * a requirement to read larger blocksizes to get the same bandwidth than >> on other platforms >> >> Our test application is small, less than 130 lines of code. It uses the >> POSIX API to read a specified blocksize from each of N files, and reports on >> the observed I/O bandwidth. It comes with a companion shell script (about 60 >> lines) which sets up the files to be read and then runs the executable with >> each of series of (specified) blocksizes. >> >> I've attached both files (source code for the executable, plus the shell >> script). The executable has an unfortunate reliance right now on glib in >> order to get the cross-platform g_get_monotonic_time(). If you want to run >> them, the build command is at the top of the executable, and then you would >> just do: >> >> ./run-readtest.sh -d FOLDER-FOR-TEST-FILES -n NUMBER-OF-FILES -f >> FILESIZE-IN-BYTES blocksize1 blocksize2 .... >> >> (You'll also need some pretty large-ish chunk of diskspace available for the >> test files). The blocksize list is typically powers of 2 starting at 16348 >> and going up to 4MB. We only see problems with NUMBER-OF-FILES is on the >> order of 128 or above. To be a useful test, the filesizes need to be at >> least 10MB or so. The full test takes several (2-20) minutes depending on >> the overall speed of your disk. >> >> But you don't have to run them: you could just read the source code >> executable to see if any lightbulbs go off. Why would this code fail so >> badly once the number of files gets up to 100+? Why can we predictably make >> any version of OS X able to get barely 5MB/sec from a disk that can sustain >> 100MB/sec? Are there some tricks to making this sort of thing work on OS X >> that anyone is aware of? Keep in mind that this same code works reliably, >> solidly and predictably on Windows and Linux (and even works reliably on >> Windows as a guest OS on OS X). >> >> We do know, by simple inspection, that other DAWs, from Reaper to Logic, >> appear to have solved this problem. Alas, unlike Ardour, they don't >> generally share their code, so we have no idea whether they did something >> clever, or whether we are doing something stupid. > > > _______________________________________________ > Do not post admin requests to the list. They will be ignored. > Coreaudio-api mailing list ([email protected]) > Help/Unsubscribe/Update your Subscription: > https://lists.apple.com/mailman/options/coreaudio-api/dangerwillrobinsondanger%40gmail.com > > This email sent to [email protected] _______________________________________________ Do not post admin requests to the list. They will be ignored. Coreaudio-api mailing list ([email protected]) Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/coreaudio-api/archive%40mail-archive.com This email sent to [email protected]
