All,
This message has generated a lot of responses, so I'm going to
address each of them here in an attempt to consolidate the thread.
============================================================
Matt,
- I'm very interested in the specifics of your setup.
- How much space is on the RAM disk?
Currently it is 10 GB. We are upgrading it to 16 GB.
- What kind of RAM drive is it?
The current RAM is 12 GB DDR2 400 MHz (12 x 1GB) Single Ranked
DIMMs. The details for each 1 GB DIMM can be seen here:
http://www.samsung.com/Products/Semiconductor/DRAM/DDR2SDRAM/DDR2SDRAMmodule/RegisteredDIMM/M393T2950CZ3/M393T2950CZ3.htm
The upgrade will involve adding 2 GB DIMMs to the system, but I
don't have the details on these yet.
The RAM disk is setup by adding the following kernel command line
option to grub.conf:
ramdisk_size=10485760
We are running Fedora Core 3 with the most up to date 2.6 SMP kernel.
By default the RAM disk's block size is 1024 bytes, so we are
formatting it as an ext2 file system with a block size of 1024 bytes
using the following command:
mke2fs -b 1024 -m 0 /dev/ram0
The block size can easily be changed from the kernel's view (using
the kernel command line option ramdisk_blocksize=XXXX) or from
mke2fs's view (using the -b XXXX argument), so please let me know if
I can make an easy optimization here.
Finally, the RAM disk is mounted using the command:
mount /dev/ram0 /digrec
A good RAMDISK howto exists at:
http://www.vanemery.com/Linux/Ramdisk/ramdisk.html
- What format are you recording to?
- What codec are the SIP calls being placed over?
We are recording to the PCM format and using the G711 uLaw codec.
High voice quality is essential to our application (we are a call
center) so we partnered with MCI to configure our network for the
required bandwidth and chose the highest quality, zero compression
codec. We noload all other codecs in order to avoid transcoding on
the switch, so we must record to PCM. Later (on a separate server)
the recordings are mixed to GSM which provides a 5 to 1 compression
ratio with very little artifacts.
- We've run into the "Avoided deadlock" recording issues several
times when trying to do
- more than 50 concurrent recordings. Changing the ast_channel_lock
loop from 10 to 20 has
- helped somewhat reduce the warnings and reduce audio gaps on the
recordings, but what is
- really needed for more robust recording is a configurable
recording buffer that wouldn't
- freak out if a 10ms delay occurs.
Are you saying that these messages indicate a gap in a digital
recording? If so, what is the duration of the gap? If it's
comparable to a CD skip, I think we can deal with it until a buffer
or another solution is implemented.
- Good luck and please keep us updated on your progress
Thank you. I'll be keeping the list informed of our progress.
============================================================
Zoa,
- I suppose you are the person from the digium forum
That was actually my boss's boss. We thank you all the way up and
down the line for your suggestion.
- The reason i recommended you to use a ramdisk is because i think the
- problem with recording to disk is saving 20ms of stream 1, then 20
ms of
- stream 2, then 20ms of stream 3 etc etc.... meaning you write
everytime
- very small things. (with a lot of seeking).
Agreed. This is why we hope that decoupling the copy (memory to
disk) from Asterisk itself and, more importantly, Asterisk's
real-time handling of the call being recorded will be sufficient.
For the record, when recording 512 simultaneous calls to the local
disk we saw a peek of about 13,000 blocks written per second.
- Our best test results were with:
-
- - buffering the recordings to a ramdisk, then
We're doing that, as per your suggestion.
- - on low load (at night) copy the files over the network (easy to
shape
- the pipe, so that you dont overload anything), This way, the memory
- buffer will take care of the 'fragmentation' and not your harddisk.
If you'll note the format of the recordings and that we'll be
recording up to 200,000 minutes of calls a day, with a little quick
math you'll realize that it would take 80 to 100 GBs of memory for
us to buffer a full day's recordings. Combined with the fact that a
server failure late in the day would cause us to lose them all, this
is not a desirable solution.
Instead, we plan to write an application to call from the
MONITOR_EXEC hook that will be executed at the end of each call.
This application will be niced down to the lowest priority, and
simply copy the leg files from memory to disk. Under normal
conditions (ie. our NFS server is up) this will actually be a copy
to a remote disk using an asynchronous NFS transfer. All actual
disk I/O will occur on our digital recording server and any handling
of the digital recordings will occur only after the call they are
bound to is completed.
Do you have any suggestions regarding this scenario?
- - on the remove server, do all the mixing / indexing etc. (i really
- don't mixing or converting between audio formats on the same
server as
- asterisk).
That exactly describes the functionality and purpose of our digital
recording server. We do no transcoding on the Asterisk server and
perform no DSP on it. It handles only SIP traffic, as our Ts are
terminated by a Cisco AS5400HPX Universal Gateway.
- If you want to go even freakier, run asterisk (or you complete
distro)
- from a ramdisk.
-
- Oh, another thing, for the people trying this the performance of
hdparm
- is not linear with the quality of your calls, tweaking your disks
to be
- faster will not help for asterisk when you do a copy. (in general).
Thanks. I will keep this information in mind.
- I thought over your suggestion to use a sniffer to do the recordings,
- you might pull it off, but will have to write your own to do so.
(or go
- to the expensive version of commercial sniffer applications).
Trust me, using a sniffer is not our ideal solution. But if push
comes to shove...
- Also when you do things over the network, disable your onboard
network
- card, and go for some more expensive network card.
- In our tests with small packets, we could increase the throughput
with a
- factor 2. (related to cpu load).
-
-...the dual nic gigabit intel card was a lot better than all the other
- combinations i tried.
Done. We were going to use a Level 5 Networks EF1-21022T NIC, but
it didn't like performing DMA with anything greater than 4 GB of
system RAM. Level 5 is working on resolving this issue with their
drivers, but until then we are using an Intel Pro 1000 MT NIC
installed in a 64-bit, 133 MHz PCI-X expansion slot. This Intel NIC
is actually the onboard NIC on our digital recording server (a Dell
PowerEdge 1850).
============================================================
Trixter,
- filesystems are also a consideration with larger scale projects.
- Different filesystems add different amounts of overheads on different
- types of operations. Some are faster at moving small files around
- others faster with large files. This adds to the disk latency.
- Removing the disk latency itself is a good thing, since that is
- typically slower, but to crank out that last little bit of
performance
- some research into the different filesystems under the specific
kernel
- that you are using could also be a consideration. The most obvious
- (and easiest to update a running system) is to remove things like
atime,
- whih with most linux distros is on by default. This causes a write
- operation for the read of a file to update the last time accessed. A
- couple little things can add up to a few percent improvement and
- generally make the cost go down.
We may need every percent we can get, so thanks for the
suggestions. As mentioned above, we're running Fedora Core 3 with
the latest 2.6 SMP kernel. Our physical disks are using the ext3
file system, while our RAM disk is using ext2.
- Or have a seperate network set up (dual nic card for example)
where the
- 2nd network is used just for NFS traffic. Although NFS generally is
- ugly network wise, it is standard and makes things easier. Just
gotta
- watch the IO on the system given that the network card itself will
cause
- cpu cycles to be used, but lets face it cpu is cheap now. Different
- drivers also work differently, and then with the 2.6 series
kernels you
- can use device polling instead of interupts which can help a little.
This seems like something we may pursue in the short term. We are
using NFS (which will be configuring for asynchronous transfers),
but not with a dedicated NIC. Our system has four 3.16 GHz Intel
Xeon processors. They are hyperthreaded, so the OS sees them as
eight separate processors. During our 256 call testing they hit 70%
idle, but during our 512 call testing they dropped to 10% idle.
Obviously, 512 calls is pushing the limits but we only really need
to hit around 300 for our current project. Nonetheless, the
additional CPU cycles consumed by another NIC are something to watch
for. Luckily, Intel will be releasing their dual-core processors soon.
- When you say ramdisk here I assume you mean using conventional
ram, its
- cheap yes but its volatile, do you have any plans for failure of the
- system or ram? Or is the data integrity itself not as critical? The
- reason that people like hard drives is because most of the time if
the
- system goes down for any reason the data is still intact.
Please see my responses to Matt regarding our RAM disk configuration.
- isnt vomit free? It was a voip sniffer that worked with some codecs
- many years ago (I wanna say mid-late 90s but I may be thinking of
- another back then). http://vomit.xtdnet.nl/ does G.711 only.
-
- The bigger prIoblem that I see is that sniffers dont always get
all the
- traffic that is on a network particularly when the network has more
- traffic on it. While this generally isnt a concern and I would
like to
- think that even a poorly configured network could allow for 512
calls,
- it is a factor to implement this type of a solution.
Hopefully, it won't come to this. With all of your help, I doubt it
will.
============================================================
Richard,
- Another project snatched from the Toilet bowl ahead of the flush.
With any luck, we'll be able to shine this turd.
============================================================
I apologize for the length of this post, but after consideration it
seemed to be the most efficient way to address each of you.
Thank you all,
Matthew Roth
InterMedia Marketing Solutions
Software Engineer and Systems Developer
_______________________________________________
--Bandwidth and Colocation sponsored by Easynews.com --
Asterisk-Users mailing list
[email protected]
http://lists.digium.com/mailman/listinfo/asterisk-users
To UNSUBSCRIBE or update options visit:
http://lists.digium.com/mailman/listinfo/asterisk-users
