Re: [vdr] making VDR ext4-ready
Marcel Witte wrote: > So ext4 seems to be perfect for a video-partition, but to make it more > perfect, it would be nice if VDR could use the fallocate()-systemcall as > mentioned in the article. This would prevent fragmentation in the file system. a) i wouldn't fully trust ext4 for a few more months at least (for a system partition, which you can toss and easily rebuild it's probably ready, for a large, not otherwise backuped, data collection it may not yet be) b) fallocate is relatively new, so you'd require a new libc and kernel (same reason i used fadvise and not sync_file_range) c) libc w/o proper kernel/fs support can emulate fallocate by prewriting zeros to every single block -- not likely what you'd want; this makes above (b) even more relevant. artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] sparse DVB adapter numbers
Klaus Schmidinger wrote: > On 21.03.2009 14:17, Artur Skawina wrote: >> Patrick Rother wrote: >>> For a reason not to go in details too much here, I would like to make >>> vdr recognise sparse DVB adapter numbers. >>> drwxr-xr-x 2 root root 200 Mar 19 11:59 adapter0/ >>> drwxr-xr-x 2 root root 120 Mar 19 11:59 adapter1/ >>> drwxr-xr-x 2 root root 120 Mar 19 11:59 adapter5/ >>> drwxr-xr-x 2 root root 120 Mar 19 11:59 adapter6/ >>> drwxr-xr-x 2 root root 120 Mar 19 11:59 adapter7/ >>> But vdr uses only the first two devices. >>> >>> Is there any easy possibility to patch the code accordingly? >> ./vdr -D 0 -D 1 -D 5 -D 6 -D 7 > > I'm afraid this won't work, because VDR stops scaning for DVB devices > as soon as it hits a gap. Good i didn't know that years ago when setting up my vdr boxes... :) IOW this approach has been working for me for quite a while w/o any problems. Note i'm still at 1.4.x and my 'gaps' actually contain adapters (unused by this vdr instance). If the gaps really are a problem you could always renumber the adapters using udev rules such as SUBSYSTEM=="dvb", SUBSYSTEMS=="pci", KERNELS==":00:0e.0", PROGRAM="/bin/sh -c 'K=%k; K=$${K#dvb}; printf dvb/adapter%%i/%%s 0 $${K#*.}'", NAME:="%c" SUBSYSTEM=="dvb", SUBSYSTEMS=="pci", KERNELS==":02:09.0", PROGRAM="/bin/sh -c 'K=%k; K=$${K#dvb}; printf dvb/adapter%%i/%%s 1 $${K#*.}'", NAME:="%c" etc artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] sparse DVB adapter numbers
Patrick Rother wrote: > For a reason not to go in details too much here, I would like to make > vdr recognise sparse DVB adapter numbers. > > I have: > > r...@vdr:/dev/dvb# ls -l > total 0 > drwxr-xr-x 2 root root 200 Mar 19 11:59 adapter0/ > drwxr-xr-x 2 root root 120 Mar 19 11:59 adapter1/ > drwxr-xr-x 2 root root 120 Mar 19 11:59 adapter5/ > drwxr-xr-x 2 root root 120 Mar 19 11:59 adapter6/ > drwxr-xr-x 2 root root 120 Mar 19 11:59 adapter7/ > r...@vdr:/dev/dvb# > > But vdr uses only the first two devices. > > Is there any easy possibility to patch the code accordingly? ./vdr -D 0 -D 1 -D 5 -D 6 -D 7 ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] channel-change-prediction, doable? stupid idea?
Udo Richter wrote: > On 21.01.2009 19:33, Gerald Dachs wrote: >> Assumed I have 2 free Tuners and I zapp through the channels of one >> tuner. Would it be possible to tune the 2. tuner to the same time to >> the after next channel in change direction, so that for the next >> channel change in the same direction it would be possible to >> channge to the 2. tuner instead of changing the channels of >> the 1. tuner > > Channel switching delay is caused by two delays: First, the time to tune > and lock the new transponder, second, the time till the next compressed > stream starting point (I-Frame or audio syncword) is reached. This > second time can be up to a second by its own. > > To get instant switching, you would have to also decode the second > stream in the background, so that the current frame is available the > moment it is needed. Or you can buffer the last GOP of the stream and go > back to that point on channel switch. You would have to ring-buffer up > to one complete GOP permanently in memory to keep the delay from there on. > > Both things are tricky to implement, require work on the output devices, > and are probably not worth the effort. Yes, the only case i think this could be useful is when the number of tuners >= transponders/muxes and you always receive all of them. Hmm, it might actually make sense for dvb-t (or -c) in some cases. Anyway, back when i was using a dvb-s ff card the tuning itself seemed to be slow, so what i did was cache some parms such as freq, srate, fec, voltage and tone in the driver, to avoid touching the hw/firmware for those that didn't change (the dvb api resulted in the driver always getting a new set of parms, even when some/most of them hadn't changed). That seemed to speed up the channel switches a bit, but as the card was never 100% reliable and would sometimes fail to correctly tune, i never went further with that. Now i think it was probably the pci host that was causing the trouble, and the changes may have been fine. artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] Frames per second PAL vs. NTSC
Klaus Schmidinger wrote: > On 05.01.2009 13:31, Artur Skawina wrote: >> Klaus Schmidinger wrote: >>> Detecting the frame rate is done by looking at the PTS values, so >>> it is independent of the actual broadcast system. >>> >>> Using this code for converting frame numbers into hh:mm:ss.ff... >>> >>> >>> #include >>> #include >>> >>> int main(void) >>> { >>> double FramesPerSecond = double(9) / 3003; >>> //FramesPerSecond = 25; >>> for (int Index = 0; Index < 1; Index++) { >>> double Seconds; >>> int f = round(modf(Index / FramesPerSecond, &Seconds) * >>> FramesPerSecond) + 1; >>> int s = int(Seconds); >>> int m = s / 60 % 60; >>> int h = s / 3600; >>> s %= 60; >>> printf("%3d ", Index); >>> printf("%15.9f ", Index / FramesPerSecond); >>> printf("%d:%02d:%02d.%02d", h, m, s, f); >>> if (f > 30) printf(" *"); >>> printf("\n"); >>> } >>> } >>> >>> >>> ... sometimes results in a 31st frame: >>> >>> 9978 332.93260 0:05:32.29 >>> 9979 332.96597 0:05:32.30 >>> 9980 332.99933 0:05:32.31 * >>> 9981 333.03270 0:05:33.02 >>> 9982 333.06607 0:05:33.03 >>> >>> >>> Any ideas how to fix this? >> eg >> >> - int f = round(modf(Index / FramesPerSecond, &Seconds) * FramesPerSecond) + >> 1; >> + int f = round(modf(Index / FramesPerSecond, &Seconds) * FramesPerSecond + >> 0.5 ); >> >> note that some 'seconds' will contain only 29 frames. (see index 510 in the >> original >> and 1019 in the fixed version) > > I'm afraid this isn't feasible. The '+1' is done to make the first frame > (at Index 0) have number '1'. With your change it would be numbered '0'. > > Klaus no, try it :) and you can also drop the round call: - int f = round(modf(Index / FramesPerSecond, &Seconds) * FramesPerSecond) + 1; + int f = modf((Index + 0.5) / FramesPerSecond, &Seconds) * FramesPerSecond + 1; which will move the 'missing' frames to the same locations as in the original. artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] Frames per second PAL vs. NTSC
Klaus Schmidinger wrote: > Detecting the frame rate is done by looking at the PTS values, so > it is independent of the actual broadcast system. > > Using this code for converting frame numbers into hh:mm:ss.ff... > > > #include > #include > > int main(void) > { > double FramesPerSecond = double(9) / 3003; > //FramesPerSecond = 25; > for (int Index = 0; Index < 1; Index++) { > double Seconds; > int f = round(modf(Index / FramesPerSecond, &Seconds) * > FramesPerSecond) + 1; > int s = int(Seconds); > int m = s / 60 % 60; > int h = s / 3600; > s %= 60; > printf("%3d ", Index); > printf("%15.9f ", Index / FramesPerSecond); > printf("%d:%02d:%02d.%02d", h, m, s, f); > if (f > 30) printf(" *"); > printf("\n"); > } > } > > > ... sometimes results in a 31st frame: > > 9978 332.93260 0:05:32.29 > 9979 332.96597 0:05:32.30 > 9980 332.99933 0:05:32.31 * > 9981 333.03270 0:05:33.02 > 9982 333.06607 0:05:33.03 > > > Any ideas how to fix this? eg - int f = round(modf(Index / FramesPerSecond, &Seconds) * FramesPerSecond) + 1; + int f = round(modf(Index / FramesPerSecond, &Seconds) * FramesPerSecond + 0.5 ); note that some 'seconds' will contain only 29 frames. (see index 510 in the original and 1019 in the fixed version) artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] VDR with S2API (update)
Udo Richter wrote: > Second: Priority and Lifetime of a recording IMHO don't belong to the > name part. This could easily fit into the info.vdr file instead. Or does > it make sense to have the same recording with different lifetime or > priority in separate folders? No and it actually causes problems; when you edit a currently active timer and eg lower the prio, the recording continues, but the next time it's restarted it is treated as a different one. Also, should vdr decide to do an emergency restart for whatever reason you end up with two recordings instead of one. artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] Tuning failures for transponder-locked tuners (VDR 1.4.7)
Richard F wrote: > Teemu, > > I had a similar problem with budget Freecom / WT-220U USB stick tuners. > I usually had to swap multiplex/transponder to get them to re-tune reliably. > I observed that if they lost tuning, vdr made them re-try (which I saw in the > log) > but eventually would bomb out if it couldn't tune, which is maybe what you're > seeing? > > Originally I logged it as a DVB driver bug, but the maintainer said it was an > application bug. > > I found the following, which requires you to patch/rebuild vdr from source. > Basically it just adds a small delay to the tuning process, and it works 100% > for me. > > I had to do this on 1.4.7 and carried it over in 1.6.0 also - so no, vanilla > 1.6 won't help you. > > It would be good if something equivalent was in vanilla VDR but I suppose it > slows > some systems by a fraction of a second when tuning. > Perhaps a "budget card" option? > > http://www.vdr-portal.de/board/thread.php?postid=639048 > The actual patch is at http://tafe.unkelhaeusser.de/vdr/dvbdevice.c.patch If the .1s delay helps it's probably some kind of race, and if restarting the app (ie close/open/tune sequence) does not make the device work it is clearly a driver bug. Adding random delays to apps in order to work around driver bugs is not the right solution. Fixing the driver would be, but if that's not an option, you could add the .1s delay to the open-demux path of the driver; that way only users of that driver would suffer until it's fixed properly. If the driver does recover after some time, you could do something like this instead of the above patch: int cDvbDevice::OpenFilter(u_short Pid, u_char Tid, u_char Mask) { const char *FileName = *cDvbName(DEV_DVB_DEMUX, CardIndex()); int f = open(FileName, O_RDWR | O_NONBLOCK); + if (f==-1) { + usleep(10); + f = open(FileName, O_RDWR | O_NONBLOCK); + } if (f >= 0) { dmx_sct_filter_params sctFilterParams; to limit the cost of the delays somewhat, but it won't work if the first open call messes up the driver... artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] PCI fun (RGB/PAL over VGA at variable frame rate)
Gavin Hamill wrote: > Over the last days, Thomas and I have been trying to sort out why my > nearly-identical machine couldn't run his VGA sync patches properly. > > The key difference is my Radeon 7000VE is PCI, whilst his is AGP. I > tried the PCI Radeon in two old Pentium-3 era machines, and on my modern > Pentium D930 desktop, all with the same behaviour - fullscreen video > over PCI causes huge CPU usage in the Xorg process, even when using xv > 'acceleration'. > > When I switch the PCI Radeon for a PCI Express X300 (the very lowest 'X' > series you can get), everything is glorious: Xorg CPU use is barely 1%. > > Unfortunately I don't have any machines with both AGP and PCI on which > I can try the same OS image but we both think it's safe to conclude that > PCI is just unsuitable for this task. PCI in general should be perfectly fine, for SDTV at least. While displaying SDTV (vdrsxfe) I see ~20% cpu use for X on AGP, ~44% on PCI (same machine, different heads, AGP is MGA450, PCI is MGA200). The huge difference is likely due to something else, like - display (X) driver (but even drivers which just memcpy the video data to the (xv) framebuffer should work on a modern machine) - PCI chipset (eg I had a VIA-based mobo, and it couldn't even keep up with SDTV on a PCI head, swapping the mobo for one w/ a real chipset made all problems suddenly disappear...) You could probably do some setpci tweaks to improve PCI throughput, but I doubt the gain would be enough (I'd expect 10% improvement or so). artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] Control VDR with RF remote
Wolfgang Fritz wrote: > Artur Skawina schrieb: >> sure, patch vs 2.6.25 below. These are just the minimal changes needed to >> make >> ati-remote work w/ one type of medion rf remotes (P/N 20018071). i did this > > Mine is P/N 20014751 afaict it looks exactly like the one i'm using. >> The keymap for medion remotes is different from the one for ati, but as i'm >> currently using mine just for vdr i never needed to fix this up (vdr does its >> own mapping anyway). > > I have added an additional table for Medion (selected by product ID), > but this might not work if the different Medion types have the same > product ID. I have at least four different types of medion remotes, each with a different layout and features; they all work w/ the same receiver so selecting based on usb ids won't work. I don't remember if the keymaps were incompatible though. In any case, i'm assuming the ati remote will work w/ a medion receiver and v/v... A per-channel module parameter is probably required. >> +/* >> + * Some remotes alternate codes, this makes it easier to detect >> autorepeat; >> + * to keep things simple we simply "undo" the code change here. >> + */ >> +if (d2 & 0x80) { >> +d1 += 0x80; >> +d2 -= 0x80; >> +} > > Interesting. I have to check if my Medion needs this. Autorepeat is > working OK but my solution is more complicated than yours, and this > might be the reason. well, the key code changes could be used to distinguish autorepeat from a repeated keypress, but i found the autorepeat rate to be high enough (for a remote) and just ignore them, as above. artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] Control VDR with RF remote
Wolfgang Fritz wrote: > Artur Skawina schrieb: >> Peer Oliver Schmidt wrote: >>> Hello Michael, >>> >>>> Does anybody use RF remote control with VDR instead of IR? How to >>>> configure such remotes? >>> I used the ATI RemoteWonder with it. LIRC has support for it. >> Or, if your remote is supported by the kernel input layer, you can use >> the remote plugin. eg the X10 remotes work nicely this way: >> >> $ modprobe -v ati-remote >> $ EV="/dev/input/"$( cd "/sys/class/input" && grep -l 'X10' >> event*/device/manufacturer | sed -e 's,/.*,,' ) >> $ vdr ... -P "remote -i $EV" >> >> (some remotes such as medion need a small kernel patch in order to handle >> all keys and detect autorepeat properly) > > Can you post this Kernel patch here (or send it to me via mail if it's > to big for the list)? I am using the Medion remote since quite a long > time with a patched ati-remote driver but I don't remember if I have > published the patch or not. And if this is not my patch, it might be > better than mine ;-) sure, patch vs 2.6.25 below. These are just the minimal changes needed to make ati-remote work w/ one type of medion rf remotes (P/N 20018071). i did this years ago, but haven't updated it since, because eventually i'll need to rewrite the driver to handle many remotes and map them to different input devices (each remote can be set to a different channel/id, but ati-remote lets you only choose which ones it listens to and provides a single /dev/input device for all of them). Other items on the TODO list: The keymap for medion remotes is different from the one for ati, but as i'm currently using mine just for vdr i never needed to fix this up (vdr does its own mapping anyway). There are other types of medion rf remotes, iirc i have 3 or 4 of them, quick tests showed they all seem to work w/ this patch, but may need some extra quirks (mostly trivial keymap mods, possibly handling of scroll wheel and touchpad). (if you have one that doesn't work w/ this patch and can't figure out how to make it work, post a link identifying it, and if i have that model i'll do the necessary changes) artur diff --git a/drivers/input/misc/ati_remote.c b/drivers/input/misc/ati_remote.c index f3b86c2..38f3183 100644 --- a/drivers/input/misc/ati_remote.c +++ b/drivers/input/misc/ati_remote.c @@ -281,6 +281,21 @@ static const struct { {KIND_FILTERED, 0xf4, 0x2F, EV_KEY, KEY_END, 1},/* END */ {KIND_FILTERED, 0xf5, 0x30, EV_KEY, KEY_SELECT, 1}, /* SELECT */ + /* Medion-remote keys */ + {KIND_FILTERED, 0xf1, 0x2c, EV_KEY, KEY_TV, 1}, /* TV */ + {KIND_FILTERED, 0xf6, 0x31, EV_KEY, KEY_VIDEO, 1}, /* (VIDEO DESKTOP) */ + + {KIND_FILTERED, 0xf7, 0x32, EV_KEY, KEY_RED, 1},/* */ + {KIND_FILTERED, 0xf8, 0x33, EV_KEY, KEY_GREEN, 1}, /* */ + {KIND_FILTERED, 0xf9, 0x34, EV_KEY, KEY_YELLOW, 1}, /* */ + {KIND_FILTERED, 0xfa, 0x35, EV_KEY, KEY_BLUE, 1}, /* */ + + {KIND_FILTERED, 0xfb, 0x36, EV_KEY, KEY_MOVE, 1}, /* RENAME */ + {KIND_FILTERED, 0xfc, 0x37, EV_KEY, KEY_NEW, 1},/* ACQUIRE IMAGE */ + {KIND_FILTERED, 0xfd, 0x38, EV_KEY, KEY_EDIT, 1}, /* EDIT */ + {KIND_FILTERED, 0xfe, 0x39, EV_KEY, KEY_SCREEN, 1}, /* FULL SCREEN */ + {KIND_FILTERED, 0xff, 0x3a, EV_KEY, KEY_AUDIO, 1}, /* DVD AUDIO */ + {KIND_END, 0x00, 0x00, EV_MAX + 1, 0, 0} }; @@ -405,6 +420,15 @@ static int ati_remote_event_lookup(int rem, unsigned char d1, unsigned char d2) { int i; + /* +* Some remotes alternate codes, this makes it easier to detect autorepeat; +* to keep things simple we simply "undo" the code change here. +*/ + if (d2 & 0x80) { + d1 += 0x80; + d2 -= 0x80; + } + for (i = 0; ati_remote_tbl[i].kind != KIND_END; i++) { /* * Decide if the table entry matches the remote input. @@ -524,6 +548,19 @@ static void ati_remote_input_report(struct urb *urb) ati_remote->old_data[1] = data[2]; ati_remote->old_jiffies = now; + /* +* There apparently are two types of X10 remotes -- some (newer?) models +* change the data sent for a repeated button press, others send the same +* bytes again. The following conditional makes the latter kind work; +* w/o this it's impossible to quickly enter eg. "11" because the driver +* assumes the button is held down (is autorepeating) and generates +* "11" -- the second button press r
Re: [vdr] Control VDR with RF remote
Peer Oliver Schmidt wrote: > Hello Michael, > >> Does anybody use RF remote control with VDR instead of IR? How to >> configure such remotes? > > I used the ATI RemoteWonder with it. LIRC has support for it. Or, if your remote is supported by the kernel input layer, you can use the remote plugin. eg the X10 remotes work nicely this way: $ modprobe -v ati-remote $ EV="/dev/input/"$( cd "/sys/class/input" && grep -l 'X10' event*/device/manufacturer | sed -e 's,/.*,,' ) $ vdr ... -P "remote -i $EV" (some remotes such as medion need a small kernel patch in order to handle all keys and detect autorepeat properly) The first time vdr is started with the remote plugin it will ask you to press all keys and learn them, just like with any other input source. artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] Problems playing ongoing recordings?
JJussi wrote: > On Wednesday, 2. Aprilta 2008 05:11:37 VDR User wrote: > >> Playback is faster then watching live tv?! Maybe if you skip past the >> commercials or something! I've started playing back a recording many >> times before it was finished and never had a problem with lockup or >> that the playback was going faster then live tv! > > Yes.. Check that attached message OR search subtitle "Playback to fast.." > from > archives. > Clements Kirchgatterer said that, that's normal... Small drift is expected -- the clock used as reference for playback (on dvb/dxr card etc) is not synced to the one used by the broadcaster. It's a similar problem to trying to record audio using one sound card and playing it back using another card in real time -- at some point you will get either over- or underruns because the sample rates are not 100% identical. I suspect some of the issues when playing ongoing recordings just after they are extended are caused by the recorder telling the system to drop the recently written data and this is interfering with the readahead done by the player, which then has to wait for the data to be refetched from disk. O_DIRECT helps here (by not using the cache at all), OTOH it could cause playback problems at the very end (due to the small delay and larger chunks used when extending the file) artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] Problems playing ongoing recordings?
Alexw wrote: Sometimes the player stops in the middle of a recording due to a zero size request. Here is the log: vdr: [3836] dvbplayer thread started (pid=3643, tid=3836) vdr: [3836] resuming replay at index 1950 (0:01:18.01) vdr: [3837] non blocking file reader thread started (pid=3643, tid=3837) vdr: [3836] SetBrokenLink: no GOP header found in video packet vdr: [3836] setting audio track to 1 (0) vdr: [3836] playing '/var/vdr/video/SERVER/recording/2008-03-28.18.58.50.50.rec/001.vdr' <<>> vdr: [3837] non blocking file reader thread ended (pid=3643, tid=3837) vdr: [3836] dvbplayer thread ended (pid=3643, tid=3836) vdr: [5618] WANT: fd: 25 1068536495 .. 1068722913 SIZE: 186418 vdr: [5618] READ: fd: 25 1068536495 .. 1068666704 SIZE: 130209 jump: 0 ra: 12582912 vdr: [5618] WANT: fd: 25 1068666704 .. 1068983331 SIZE: 316627 vdr: [5618] READ: fd: 25 1068666704 .. 1068680058 SIZE: 13354 jump: 0 ra: 12582912 vdr: [5618] READ: fd: 25 1068680058 .. 1068690344 SIZE: 10286 jump: 0 ra: 12582912 vdr: [5618] READ: fd: 25 1068690344 .. 1068721839 SIZE: 31495 jump: 0 ra: 12582912 vdr: [5618] READ: fd: 25 1068721839 .. 1069246127 SIZE: 524288 jump: 0 ra: 12582912 vdr: [5618] WANT: fd: 25 1069246127 .. 1070294703 SIZE: 1048576 vdr: [5618] READ: fd: 25 1069246127 .. 1069246127 SIZE: 0 jump: 0 ra: 12582912 vdr: [5618] non blocking file reader thread ended (pid=5563, tid=5618) vdr: [5617] dvbplayer thread ended (pid=5563, tid=5617) Weird, cUnbufferedFile::Read(Size=0). I'll try to reproduce this. Sometimes it take a long time to occur, sometimes not. Did this start after applying my patch, or did it happen in the past too? Does it always happen at a certain position? Specific stream or bitrate? I don't recall ever having a similar problem, the number 524288 looks a bit suspicious... As you can see the requested size is increasing until it reaches the max buf. This is also a period with freezes in the video (late delivery). Do these problems (0-sized reads) occur only near the end of a program being recorded? No, you can experience a stop in the middle of a recording. Also, I see from the above that the readahead code needs to be more aggressive: vdr: [5627] WANT: fd: 25 1188531493 .. 1188861741 SIZE: 330248 [... small reads...] vdr: [5627] READ: fd: 25 1188616808 .. 1189141096 SIZE: 524288 jump: 0 ra: 12582912 the readahead window does not cover the area which is being read later -- this certainly is likely to stall playback. I'll fix this (i did not expect such a large difference in read request sizes.) The attached patch makes the readahead window grow much faster, this will cause more I/O at the start of playback, but should handle cases like the one above better. If it works correctly all the ranges mentioned in "READ:" lines should be inside the preceding "WANT:" range and the playback shouldn't stall. Here the readahead window grows to ~5Mbytes just after starting playback, i still need to check that this is not too fast, doesn't saturate the disk and/or link and cause delays when jumping etc. Tested by playing a few files from an NFS mount, didn't notice any problems so far. An incremental patch would look like this (the attached one (vs 1.4.7) already includes it): diff --git a/tools.c b/tools.c index a14f799..e22614f 100644 --- a/tools.c +++ b/tools.c @@ -1186,13 +1186,13 @@ ssize_t cUnbufferedFile::Read(void *Data, size_t Size) // Trigger the readahead IO, but only if we've used at least some of the previously // requested area. This avoids calling fadvise() after every read() call. size_t cachedsize = cachedend - curpos; - size_t ra = cachedsize + Size*2 + (size_t)jumped*1; + size_t ra = cachedsize + Size*8 + (size_t)jumped*1; if (cutting) ra += KILOBYTE(64); ra = min(readahead, ra); // Start I/O if we A) used some of the data or B) can read sufficiently large new chunk. // (A) is important when starting w/ a small readahead. - if (cachedsize < (ra-ra/4) || cachedsize+KILOBYTE(256) <= ra) + if (cachedsize < (ra-ra/16) || cachedsize+KILOBYTE(256) <= ra) FadviseRead(curpos, ra); } else if (jumped >= 0) {// either large forward jump, or FF (jumps by ~4xSize) artur diff --git a/cutter.c b/cutter.c index 5170ae4..7e2e506 100644 --- a/cutter.c +++ b/cutter.c @@ -66,7 +66,8 @@ void cCuttingThread::Action(void) toFile = toFileName->Open(); if (!fromFile || !toFile) return; - fromFile->SetReadAhead(MEGABYTE(20)); + fromFile->CuttingSrc(); + toFile->CuttingDst(); int Index = Mark->position; Mark = fromMarks.Next(Mark); int FileSize = 0; @@ -91,7 +92,7 @@ void cCuttingThread::Action(void) if (fromIndex->Get(Index++, &FileNumber, &FileOffset, &PictureType, &Length))
Re: [vdr] Problems playing ongoing recordings?
alexw wrote: > On Friday 28 March 2008 14:52:38 alexw wrote: >> On Thursday 27 March 2008 18:22:35 Artur Skawina wrote: >>> VDR User wrote: >>>> On Thu, Mar 27, 2008 at 9:19 AM, Artur Skawina <[EMAIL PROTECTED]> wrote: >>>>> I have a similar setup, just with 100M ethernet instead of wifi and >>>>> NFS instead of samba. wifi could be a problem, but if you're able to >>>>> watch the same channel live you should also be able to view a >>>>> recording. >>>> Takes a lot more bandwidth to record & playback then just to record so >>>> the fact that live tv is fine doesn't amount to much I don't think. >>> I was referring to playing a finished recording and playing a file that >>> is currently being extended by the "server" vdr -- alexw said that >>> doesn't work well for him. It should, unless the disk and/or fs can't >>> handle the two data streams concurrently, while keeping the latency low >>> enough. I'm assuming the vdr server in powerful enough to handle the >>> load, yes. >> My setup is a little bit more complicated as it is using a share drive on >> both machine. The two local disks are only CF. The file server is compose >> of 4x SATA II 500GB in raid 5 (total ~1.5 TB available) piloted by a >> promise controller and a fully idle 3 GHz P4 CPU. The throughput or the >> sustained write/read access is not a bottleneck. Here is a quick ASCII art >> drawing: >> >> >> /-- 100M --[AP]~ WIFI 54Mb ~[vdr client/FF]-[TV] >> >> [switch]--- 100M ---[CIFS/fileserver] >> >> \-- 100M -- [vdr server/B2C2]-DVB-S+DVB-T >> >> >> This evening I will test the provided patch. > Using the patch provided by Artur, playing a finished/ongoing recording is > smoother than before. Thank you for testing (and for enabling the extra logging). Does "smoother than before" mean that there is some improvement, but the playback still isn't perfect? The fact that the recorder hangs on to the last 4M of the stream and the data is appended in large chunks can cause problems when timeshifting by just a few seconds -- the player could try to access the not yet written data. This should only happen at the very end of an ongoing recording, within 4M of EOF (depending on the bitrate usually a couple of seconds after the live program arrived). > Sometimes the player stops in the middle of a recording due to a zero size > request. Here is the log: > > > vdr: [3836] dvbplayer thread started (pid=3643, tid=3836) > vdr: [3836] resuming replay at index 1950 (0:01:18.01) > vdr: [3837] non blocking file reader thread started (pid=3643, tid=3837) > vdr: [3836] SetBrokenLink: no GOP header found in video packet > vdr: [3836] setting audio track to 1 (0) > vdr: [3836] playing > '/var/vdr/video/SERVER/recording/2008-03-28.18.58.50.50.rec/001.vdr' > > <<>> > > vdr: [3837] non blocking file reader thread ended (pid=3643, tid=3837) > vdr: [3836] dvbplayer thread ended (pid=3643, tid=3836) > > - > > vdr: [5618] WANT: fd: 25 1068536495 .. 1068722913 SIZE: 186418 > vdr: [5618] READ: fd: 25 1068536495 .. 1068666704 SIZE: 130209 jump: > 0 ra: 12582912 > vdr: [5618] WANT: fd: 25 1068666704 .. 1068983331 SIZE: 316627 > vdr: [5618] READ: fd: 25 1068666704 .. 1068680058 SIZE: 13354 jump: > 0 ra: 12582912 > vdr: [5618] READ: fd: 25 1068680058 .. 1068690344 SIZE: 10286 jump: > 0 ra: 12582912 > vdr: [5618] READ: fd: 25 1068690344 .. 1068721839 SIZE: 31495 jump: > 0 ra: 12582912 > vdr: [5618] READ: fd: 25 1068721839 .. 1069246127 SIZE: 524288 jump: > 0 ra: 12582912 > vdr: [5618] WANT: fd: 25 1069246127 .. 1070294703 SIZE: 1048576 > vdr: [5618] READ: fd: 25 1069246127 .. 1069246127 SIZE: 0 jump: > 0 ra: 12582912 > vdr: [5618] non blocking file reader thread ended (pid=5563, tid=5618) > vdr: [5617] dvbplayer thread ended (pid=5563, tid=5617) Weird, cUnbufferedFile::Read(Size=0). I'll try to reproduce this. > - > > vdr: [5627] READ: fd: 25 1188203807 .. 1188214024 SIZE: 10217 jump: > 0 ra: 12582912 > vdr: [5627] READ: fd: 25 1188214024 .. 1188226152 SIZE: 12128 jump: > 0 ra: 12582912 > vdr: [5627] READ: fd: 25 1188226152 .. 1188267411 SIZE: 41259 jump: > 0 ra: 12582912 > vdr: [5627] WANT: fd: 25 1188267411 .. 1188503927 SIZE: 236516 > vdr: [5627] READ: fd: 25 1188267411 .. 1188278781 SIZE: 11370 jump: > 0 ra: 12582912 > vdr: [5627] READ: fd: 25 118827
[vdr] [PATCH] Direct I/O support v1
This change makes vdr write the video stream directly to disk, bypassing OS caches. O_DIRECT is used if it is supported, falling back to "normal" I/O and fadvise() if not. Only recording is affected, playback remains unchanged. I did this to see if using direct i/o made sense for vdr, wasn't convinced before. Avoiding all the fadvise hacks and especially the fdatasyncs on close seems to be worth it though. Only lightly tested, but seems to work, i tried a few recordings on both ext3 and NFS and didn't notice any missing data yet ;) Upgraded all vdrs and will keep testing for some time, but in case anyone else would like to try (or even better improve) it, patch is below. This patch is based on top of the one I posted in the "Problems playing ongoing recordings?" thread, so you'll need to apply that one before applying this one. Should work on both 1.4 and 1.6. artur diff --git a/tools.c b/tools.c index a14f799..b7281b0 100644 --- a/tools.c +++ b/tools.c @@ -1054,6 +1054,11 @@ bool cSafeFile::Close(void) // --- cUnbufferedFile --- #define USE_FADVISE +#define USE_DIRECTIO + +// O_DIRECT can have various alignment restrictions, usually at most +// the block size of the filesystem. 4096 bytes should be enough. +#define ALIGN_DIO 4096 //#define dfsyslog dsyslog // uncomment to turn on fadvise related logging #define dfsyslog(a...) do {} while (0) @@ -1071,7 +1076,17 @@ cUnbufferedFile::~cUnbufferedFile() int cUnbufferedFile::Open(const char *FileName, int Flags, mode_t Mode) { Close(); - fd = open(FileName, Flags, Mode); +#ifdef USE_DIRECTIO + if (Flags&(O_WRONLY|O_RDWR)) { + fd = open(FileName, Flags|O_DIRECT, Mode); + directio = 1; + } +#endif + if (fd==-1) { + directio = 0; + fd = open(FileName, Flags, Mode); + } + dsyslog("Using %s IO to access %s", directio?"DIRECT":"normal", FileName); curpos = 0; #ifdef USE_FADVISE lastpos = 0; @@ -1101,7 +1116,7 @@ int cUnbufferedFile::Close(void) free(wbuf); } #ifdef USE_FADVISE - if (fd >= 0) { + if (fd >= 0 || !directio) { if (totwritten)// if we wrote anything make sure the data has hit the disk before fdatasync(fd); // calling fadvise, as this is our last chance to un-cache it. posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED); @@ -1243,7 +1258,36 @@ ssize_t cUnbufferedFile::Read(void *Data, size_t Size) ssize_t cUnbufferedFile::WriteBuf(const void *Data, size_t Size) { if (fd >=0) { - ssize_t bytesWritten = safe_write(fd, Data, Size); + ssize_t bytesWritten; + +#ifdef USE_DIRECTIO + if (directio) { +// write properly sized buffers directly. +if ((Size & (ALIGN_DIO-1)) == 0) { + bytesWritten = safe_write(fd, Data, Size); + curpos += bytesWritten; + return bytesWritten; +} +// in the unlikely case of a short write (inevitable when closing) +// pad the data with zeros, write it, then truncate the file. +int padding = ALIGN_DIO - (Size & (ALIGN_DIO-1)); +memset((char *)Data+Size, 0, padding); +bytesWritten = safe_write(fd, Data, Size+padding); + +padding = bytesWritten - Size; +if (padding<0) + padding = 0; +bytesWritten -= padding; +curpos += bytesWritten; + +lseek(fd, -padding, SEEK_CUR); +ftruncate(fd, curpos); +// note: past this point the file offset is likely unaligned +// so further directio shouldn't happen. +return min(bytesWritten, (ssize_t)Size); + } +#endif + bytesWritten = safe_write(fd, Data, Size); //dsyslog("WRIT: fd:%3d %9zd .. %9zd SIZE: %6zd", fd, curpos, curpos+Size, Size); #ifdef USE_FADVISE if (bytesWritten > 0) { @@ -1296,18 +1340,23 @@ ssize_t cUnbufferedFile::Write(const void *Data, size_t Size) { if (!wbuf) { wbuf_chunk = cutting?MEGABYTE(8):MEGABYTE(4); - wbuf = MALLOC(uchar,wbuf_chunk); - if (!wbuf) + if (posix_memalign(&wbuf, ALIGN_DIO, wbuf_chunk)) { +directio = 0; return WriteBuf(Data, Size); + } wbuf_len = 0; } if (Size <= wbuf_chunk-wbuf_len) { memcpy(wbuf+wbuf_len, Data, Size); wbuf_len += Size; } else { - WriteBuf(wbuf, wbuf_len); - memcpy(wbuf, Data, Size); - wbuf_len = Size; + unsigned l = wbuf_chunk-wbuf_len; + if (l) +memcpy(wbuf+wbuf_len, Data, l); + WriteBuf(wbuf, wbuf_chunk); + + memcpy(wbuf, (char *)Data+l, Size-l); + wbuf_len = Size-l; } return Size; } diff --git a/tools.h b/tools.h index ce7283c..c2ff493 100644 --- a/tools.h +++ b/tools.h @@ -255,6 +255,7 @@ private: size_t written; size_t totwritten; int cutting; + int directio; size_t writebuffer; int FadviseDrop(off_t Offset, off_t Len); int FadviseRead(off_t Offset, off_t Len); ___ vdr mailing list v
Re: [vdr] Problems playing ongoing recordings?
VDR User wrote: > On Thu, Mar 27, 2008 at 9:19 AM, Artur Skawina <[EMAIL PROTECTED]> wrote: >> I have a similar setup, just with 100M ethernet instead of wifi and NFS >> instead >> of samba. wifi could be a problem, but if you're able to watch the same >> channel >> live you should also be able to view a recording. > > Takes a lot more bandwidth to record & playback then just to record so > the fact that live tv is fine doesn't amount to much I don't think. I was referring to playing a finished recording and playing a file that is currently being extended by the "server" vdr -- alexw said that doesn't work well for him. It should, unless the disk and/or fs can't handle the two data streams concurrently, while keeping the latency low enough. I'm assuming the vdr server in powerful enough to handle the load, yes. artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] Problems playing ongoing recordings?
alexw wrote: I _think_ the issue was some kind of stuttering playback; it didn't happen always, but often enough that one day i decided to see if writing the stream in much larger chunks would help. It did; never seen it since. I am using the following VDR setup: 2 VDR machines in client(FF)/server(budget) mode (thank to streamdev) over wifi. The video folder is shared over the network in CIFS (aka Samba). Actually replaying VDR I have a similar setup, just with 100M ethernet instead of wifi and NFS instead of samba. wifi could be a problem, but if you're able to watch the same channel live you should also be able to view a recording. recording is not smooth all the time and the timeshifting on the client side is not working at all. Timeshifting on the client requires 3x the bandwith and a lot of data going in both directions, so may not work all that well over wireless. Recording on the server and playing on the client at the same time result in smooth and freeze cycle with lip sync issue. Watching live programs is perfect. But this sounds very much like the problem i was seeing. In my case it was dependent on the bitrate and number of simultaneous recordings IIRC. If you'd like to try a patch, attached is what i've using since May 2007. It could help, but might also not change anything at all; the main reason i'm interested if it helps is because i have O_DIRECT support done on top of this, and am not sure if i should keep the write coalescing for the case where O_DIRECT isn't available. It contains: - various code cleanups (that i made after Klaus merged the fadvise-related modifications, they make the patch a bit harder to read, but should be otherwise harmless) - tweaks to improve playback (smarter readahead) - tweaks to improve cutting performance (didn't help much iirc) - write coalescing - the stream is written in multi-megabyte chunks instead of a dozen or so K at a time. This is the interesting part; it adds another data copy, but seems to have made the stuttering playback disappear. (this change will only make a difference when applied to the vdr that does the recording, ie the server; the fadvise tweaks mentioned above could also improve things when applied to the client) Patch made against vdr-1.4.7, but applies to 1.6.0 too. artur diff --git a/cutter.c b/cutter.c index 5170ae4..7e2e506 100644 --- a/cutter.c +++ b/cutter.c @@ -66,7 +66,8 @@ void cCuttingThread::Action(void) toFile = toFileName->Open(); if (!fromFile || !toFile) return; - fromFile->SetReadAhead(MEGABYTE(20)); + fromFile->CuttingSrc(); + toFile->CuttingDst(); int Index = Mark->position; Mark = fromMarks.Next(Mark); int FileSize = 0; @@ -91,7 +92,7 @@ void cCuttingThread::Action(void) if (fromIndex->Get(Index++, &FileNumber, &FileOffset, &PictureType, &Length)) { if (FileNumber != CurrentFileNumber) { fromFile = fromFileName->SetOffset(FileNumber, FileOffset); - fromFile->SetReadAhead(MEGABYTE(20)); + fromFile->CuttingSrc(); CurrentFileNumber = FileNumber; } if (fromFile) { @@ -124,6 +125,7 @@ void cCuttingThread::Action(void) error = "toFile 1"; break; } + toFile->CuttingDst(); FileSize = 0; } LastIFrame = 0; @@ -164,6 +166,7 @@ void cCuttingThread::Action(void) error = "toFile 2"; break; } +toFile->CuttingDst(); FileSize = 0; } } diff --git a/tools.c b/tools.c index 255c806..a14f799 100644 --- a/tools.c +++ b/tools.c @@ -1055,7 +1055,8 @@ bool cSafeFile::Close(void) #define USE_FADVISE -#define WRITE_BUFFER KILOBYTE(800) +//#define dfsyslog dsyslog // uncomment to turn on fadvise related logging +#define dfsyslog(a...) do {} while (0) cUnbufferedFile::cUnbufferedFile(void) { @@ -1073,12 +1074,19 @@ int cUnbufferedFile::Open(const char *FileName, int Flags, mode_t Mode) fd = open(FileName, Flags, Mode); curpos = 0; #ifdef USE_FADVISE - begin = lastpos = ahead = 0; - cachedstart = 0; - cachedend = 0; - readahead = KILOBYTE(128); + lastpos = 0; + lastjump = 0; + cachedstart = cachedend = 0; + // 4M readahead seems to work for playback while cutting, + // but isn't quite enough for FF while cutting... + readahead = MEGABYTE(12); + writebuffer = KILOBYTE(2000); written = 0; totwritten = 0; + cutting = 0; + + wbuf = NULL; + if (fd >= 0) posix_fadvise(fd, 0, 0, POSIX_FADV_RANDOM); // we could use POSIX_FADV_SEQUENTIAL, but we do our own readahead, disabling the kernel one. #endif @@ -1087,6 +1095,11 @@ int cUnbufferedFile::Open(const char *FileName, int Flags, mode_t Mode)
Re: [vdr] Problems playing ongoing recordings?
VDR User wrote: > On Wed, Mar 26, 2008 at 10:27 AM, Artur Skawina <[EMAIL PROTECTED]> wrote: >> Is anyone seeing some kind of problem when playing back files, but only when >> the >> recording has not yet finished? Possibly just when replaying a file that is >> currently >> being extended, maybe by another VDR and/or over NFS. >> >> [i had a problem like that many months ago, attempted to fix it, but it >> seems i was >> too successful ;) - the problem went away, but i didn't verify that the >> change >> actually helped. By now i don't remember exactly what the symptoms were, >> and what >> the vdr setup looked like at the time; hence i can't easily reproduce it. >> I'm planning on basing another feature on top of the fix and wondering how >> to >> handle that...] > > It would help if you mentioned what the problem actually was so people > actually know what they're supposed to be looking for. I've played > recordings back before they're finished many times and never noticed > any problems. *shrug* kind of hard to describe something you no longer remember... ;) I _think_ the issue was some kind of stuttering playback; it didn't happen always, but often enough that one day i decided to see if writing the stream in much larger chunks would help. It did; never seen it since. artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
[vdr] Problems playing ongoing recordings?
Is anyone seeing some kind of problem when playing back files, but only when the recording has not yet finished? Possibly just when replaying a file that is currently being extended, maybe by another VDR and/or over NFS. [i had a problem like that many months ago, attempted to fix it, but it seems i was too successful ;) - the problem went away, but i didn't verify that the change actually helped. By now i don't remember exactly what the symptoms were, and what the vdr setup looked like at the time; hence i can't easily reproduce it. I'm planning on basing another feature on top of the fix and wondering how to handle that...] artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] astra und hotbird ueber diseqc schalter?
Oliver Joa wrote: > Hi, > > ich habe mir heute einen zweiten LNB auf die Schuessel gebaut um Hotbird > empfangen zu koennen. Jetzt weiss ich nur noch nicht wie ich beide LNB > am geschicktesten mit meinem VDR verbinde (Beide LNB jeweils Quad). Ich > habe 2 DVB Karten. Die professionelle Variante waere wohl einen > Multiswitch einzubauen. Dagegen spricht der Preis und dass er extra the easiest way would be two 2>1 diseqc switches - no vdr patches required, just the usual diseqc config. artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] change make install to be really useful
Udo Richter wrote: > Matthias Schwarzott wrote: >> makefile-destdir.diff: >> It add variable DESTDIR to makefile to install under $(DESTDIR) instead of / > > I would prefer something like DESTDIR=/usr/local that can be redirected > to /usr or /tmp/newpackage/usr or similar, just as many configure > scripts do it. On the downside, this wouldn't work for > $(DESTDIR)$(VIDEODIR). the usual convention is PREFIX, which defaults to '/usr/local' and can be changed to eg '/usr/' or '/sw/vdr-1.6.0' etc. DESTDIR is completely independent -- it adds _another_ prefix; this is used eg when building packages. So "make install DESTDIR=/tmp/BLD-38746 PREFIX=/sw/vdr-1.6.0" means to assume the program will be installed in /sw/vdr-1.6.0 (and look for config files in /sw/vdr-1.6.0/etc/, binaries in /sw/vdr-1.6.0/bin/ etc), but to copy all the files to /tmp/BLD-38746/sw/vdr-1.6.0. This way you can build a package w/o disturbing the /sw/ tree and install using a package manager later. ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
[vdr] [PATCH] softdevice segfaulting on stream errors
Some time ago the softdevice plugin started crashing when playing video streams containing errors, both while live viewing and watching recordings. Today I found a transponder where it reliably occurred within seconds and decided to investigate. The picture buffers used for video were 8-byte aligned, which happened to work for "normal" mpeg decoding, but wasn't enough. ffmpeg was executing SSE instructions when doing error concealment and those were segfaulting. Fix below. artur diff -uwp softdevice.noalign/PicBuffer.c softdevice/PicBuffer.c --- softdevice.noalign/PicBuffer.c 2007-03-13 02:57:19.0 +0100 +++ softdevice/PicBuffer.c 2007-07-10 15:25:38.0 +0200 @@ -316,9 +316,8 @@ bool AllocatePicBuffer(sPicBuffer *buf,P if ( !isPlanar(pix_fmt) ) { buf->stride[0]=ALIGN(pixel_size*w,16); -buf->pixel[0]=(uint8_t*)malloc((buf->stride[0]*h)+16); -if (buf->pixel[0]==NULL) { +if (posix_memalign(&buf->pixel[0], 16, buf->stride[0]*h+16)) { printf("could not allocate memory for picture buffer!\n") ; exit(-1); return false; @@ -337,9 +336,7 @@ bool AllocatePicBuffer(sPicBuffer *buf,P buf->stride[i]= ALIGN(pixel_size*w>>h_shift, STRIDE_ALIGN<<(h_chroma_shift-h_shift)); -buf->pixel[i]= (uint8_t*)malloc((buf->stride[i]*h>>v_shift)+16); //FIXME 16 - -if(buf->pixel[i]==NULL) { +if(posix_memalign(&buf->pixel[i], 16, (buf->stride[i]*h>>v_shift)+16)) { //FIXME 16 printf("could not allocate memory for picture buffer!\n") ; exit(-1); return false; ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] [PATCH] Auto sized ringbuffers v3
VDR User wrote: > On 5/20/07, Artur Skawina <[EMAIL PROTECTED]> wrote: >> VDR User wrote: >> > I've recorded/watched a lot of tv in the last couple days and so far I >> > haven't noticed any problems with this patch. I am currently using v3 >> > (the patch I attached to my previous post) with vdr-1.5.2. >> >> Do you know how large the buffers usually grow? >> If you haven't reduced vdrs log level (-l) this info should be >> somewhere in /var/log/ and you can take a look at the buffer >> usage w/ eg: > > I use level 3 logging for vdr. I emailed you a copy of my logs so you > should see that by the time you read this. Thanks. It seems that 648k was the largest size ever used for a buffer, which is less than 1/3 of the full size (2M for Result,Transfer and TS). The Recorder buffer was typically at 5% of its max size, ie 300k instead of 5M. No overflows w/ v3 of the patch (two with the previous version, which was starting out with just 128k). I get a lot worse results here (buffers grow to a few M), but that's probably because the vdr box is not a dedicated one, but also handles quite a few other services; IO stalls are not uncommon and several simultaneous recordings are sometimes done while streaming via streamdev to another vdr. Still, no overflows. artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] [PATCH] Auto sized ringbuffers v3
VDR User wrote: > I've recorded/watched a lot of tv in the last couple days and so far I > haven't noticed any problems with this patch. I am currently using v3 > (the patch I attached to my previous post) with vdr-1.5.2. Do you know how large the buffers usually grow? If you haven't reduced vdrs log level (-l) this info should be somewhere in /var/log/ and you can take a look at the buffer usage w/ eg: $ grep vdr /var/log/* | grep 'ring buffer' | less Did any overflows happen (you can search for 'overflow' in the output of above commands to find them)? I've now checked my vdr logs here (starting ~ a week ago) and not a single buffer overflowed. I would have expected a few problems, but it looks like the tuning worked. (However that vdr has all data moving threads running at real time priority and this may have helped too.) Thanks, artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] [PATCH] Auto sized ringbuffers v3
Jaroslaw Swierczynski wrote: > It seems your patch is against VDR 1.5.x. Could you please backport it > to VDR 1.4.x? actually it's vs 1.4.6, but applies to 1.5.2 too. ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
[vdr] [PATCH] Auto sized ringbuffers v3
This is still in a just-for-testing phase. Found a way to better stress test this code, hence these changes: - decreased the growth threshold from 1/2 to 1/3. IOW the goal is always for 66% of the buffer to be free; this increases the working set, but the alternative (handling overflows by spilling over to the unused portion) makes thing more complex. - made buffers grow by 1.5x, not 2x. Related to above. - increased the initial bufsize from 128k to 192k. - removed some now redundant Read() code. - dropped the controversial cRecorder::Receive hunk (I'm keeping this locally as it prevents (very rare) data corruption while testing. Overflows will be logged even w/o this change, so it makes no difference otherwise). - added verbose cRecorder::Action sleep logging. Shorter delays are a side effect, but do not help all that much (from the few times this triggered here so far less than half of the delays were <<100ms). Patch attached (apparently this makes extracting it easier for gmail users). artur diff --git a/dvbdevice.c b/dvbdevice.c index 955483e..20dcf29 100644 --- a/dvbdevice.c +++ b/dvbdevice.c @@ -1184,7 +1184,7 @@ bool cDvbDevice::OpenDvr(void) CloseDvr(); fd_dvr = DvbOpen(DEV_DVB_DVR, CardIndex(), O_RDONLY | O_NONBLOCK, true); if (fd_dvr >= 0) - tsBuffer = new cTSBuffer(fd_dvr, MEGABYTE(2), CardIndex() + 1); + tsBuffer = new cTSBuffer(fd_dvr, MEGABYTE(8), CardIndex() + 1); return fd_dvr >= 0; } diff --git a/recorder.c b/recorder.c index 8bb1621..0b625d6 100644 --- a/recorder.c +++ b/recorder.c @@ -171,8 +171,22 @@ void cRecorder::Action(void) int Count = remux->Put(b, r); if (Count) ringBuffer->Del(Count); - else - cCondWait::SleepMs(100); // avoid busy loop when resultBuffer is full in cRemux::Put() - } + else { // avoid busy loop when resultBuffer is full in cRemux::Put() + int ms; + for (ms=5; Running() && ms<100; ms+=ms) { + cCondWait::SleepMs(ms); + if (!Running()) +return; + Count = remux->Put(b, r); + if (Count) { +ringBuffer->Del(Count); +dsyslog("cRecorder::Action() ring buffer consumer slept %d ms", ms); +break; +} + } + if (ms>=100) + dsyslog("cRecorder::Action() ring buffer consumer slept >100 ms"); + } + } } } diff --git a/remux.c b/remux.c index da805b7..bd29b2f 100644 --- a/remux.c +++ b/remux.c @@ -1851,7 +1851,7 @@ void cTS2PES::ts_to_pes(const uint8_t *Buf) // don't need count (=188) // --- cRemux -#define RESULTBUFFERSIZE KILOBYTE(256) +#define RESULTBUFFERSIZE MEGABYTE(2) cRemux::cRemux(int VPid, const int *APids, const int *DPids, const int *SPids, bool ExitOnFailure) { diff --git a/ringbuffer.c b/ringbuffer.c index 0633bd3..68ce081 100644 --- a/ringbuffer.c +++ b/ringbuffer.c @@ -151,13 +151,31 @@ void cRingBufferLinear::PrintDebugRBL(void) } #endif +// cRingBufferLinear are dynamically sized, or at least we can pretend they are ;) +// We treat 'Size' as the maximum size, but start with a small buffer, which can +// grow later as it fills up. Memory is always allocated for the full buffer, but +// the unused RAM portion remains untouched until (if at all) it is actually needed. +// Note that we can not start with a larger than requested buffer because there are +// ring buffer users that cause crashes then (eg softdevice mpegdecoder absolutely +// needs 32/64k). + +// Startup size. 64k still causes overflows before buffer starts to grow, 128k doesn't. +// The buffers grow to 200..500k anyway, so maybe increasing this a bit more would +// make sense, but let's first see if it's really needed. +// In fact 128k is on the low side, but let's try not going for 256k yet. +#define DEFRBSIZE KILOBYTE(192) + cRingBufferLinear::cRingBufferLinear(int Size, int Margin, bool Statistics, const char *Description) -:cRingBuffer(Size, Statistics) +:cRingBuffer(min(Size,DEFRBSIZE), Statistics) { description = Description ? strdup(Description) : NULL; tail = head = margin = Margin; gotten = 0; buffer = NULL; + maxsize = Size; + growthresh = size 1) { // 'Size - 1' must not be 0! if (Margin <= Size / 2) { buffer = MALLOC(uchar, Size); @@ -183,6 +201,8 @@ cRingBufferLinear::~cRingBufferLinear() #ifdef DEBUGRINGBUFFERS DelDebugRBL(this); #endif + dsyslog("Deleting ring buffer \"%s\" size: %d / %d (used %g%% of requested size)", description, + size, maxsize, size/(double)maxsize*100.0 ); free(buffer); free(description); } @@ -205,8 +225,20 @@ void cRingBufferLinear::Clear(void) EnablePut(); } +// Must only be called by the p
Re: [vdr] [PATCH] dynamically sized ringbuffers v2
Klaus Schmidinger wrote: > If this "auto sized ringbuffers" change (which, from what I can > see so far - haven't tried it myself - looks like a good idea) > is ever to make its way into the official VDR source, you'll need > to get rid of the above waiting. It says in receiver.h: > > ...the call must return as soon as possible, without any unnecessary delay. > > Maybe I should change this to > > ...the call must return immediately. or "must not sleep". I did mention in the patch description, which you snipped, that this was just a debugging tool. IOW that change can be dropped entirely, the condition that makes it sleep has never triggered since I made the buffers grow faster anyway. I'd be great if somebody with a FF card could test the patch, the code has worked flawlessly on my vdr server, not a single overflow since v2. However on the streamdev/softdevice client I had two transfer buffer overflows (153 bytes each) just after switching to a new channel, while both machines were under significant (io) load, cutting aot. I don't think this is reason enough to introduce a minsize rb parameter, but if it also happens w/ a simple FF setup, that would be an option. I don't want to raise the default minimum, because 128k is often enough and i'd like to avoid shrinking, to keep things simple. The patch is for 1.4.6, but applies also, with a few harmless offsets, to 1.5.2. artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] [PATCH] dynamically sized ringbuffers v2
Jouni Karvo wrote: > Stone writes: > > > > > > It still wouldn't surprise me if this version caused a few overflows, > > > but hopefully these will be very rare. > > > > I'm curious how streamdev will function with these buffer changes. > > And since I am not convinced that this memory footprint issue is > significant, I am concerned if this patch is accepted before all the > overflows and problems with live viewing have been solved first. It shouldn't. The whole point of posting it is so that other are able to test and review it. That is why I always mention the overflows. Right now it's stable enough for production for me, haven't seen any overflow since adjusting the growth rate, but it would be nice if somebody with a different vdr setup tested it too (eg a FF card based vdr). It will take weeks to verify that the code works as expected (the goal is to have _fewer_ overflows (compared to the static buffers) while at the same time a much smaller working set). artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] [PATCH] dynamically sized ringbuffers v2
Stone wrote: > > It still wouldn't surprise me if this version caused a few overflows, > but hopefully these will be very rare. > > I'm curious how streamdev will function with these buffer changes. it works fine -- i'm using a headless vdr server and streamdev+softdevice clients, so this actually gets tested fairly well. Example buffer usage from server: Deleting ring buffer "streamdev-streamer" size: 262144 / 4194304 (used 6.25% of requested size) Deleting ring buffer "streamdev-streamer" size: 131072 / 4194304 (used 3.125% of requested size) Deleting ring buffer "streamdev-streamer" size: 262144 / 4194304 (used 6.25% of requested size) Deleting ring buffer "streamdev-streamer" size: 131072 / 4194304 (used 3.125% of requested size) Deleting ring buffer "streamdev-streamer" size: 131072 / 4194304 (used 3.125% of requested size) Deleting ring buffer "streamdev-streamer" size: 524288 / 4194304 (used 12.5% of requested size) Deleting ring buffer "streamdev-streamer" size: 262144 / 4194304 (used 6.25% of requested size) Deleting ring buffer "streamdev-streamer" size: 262144 / 4194304 (used 6.25% of requested size) Deleting ring buffer "streamdev-streamer" size: 262144 / 4194304 (used 6.25% of requested size) Deleting ring buffer "streamdev-streamer" size: 131072 / 4194304 (used 3.125% of requested size) Deleting ring buffer "streamdev-streamer" size: 262144 / 4194304 (used 6.25% of requested size) Deleting ring buffer "streamdev-streamer" size: 131072 / 4194304 (used 3.125% of requested size) Deleting ring buffer "streamdev-streamer" size: 262144 / 4194304 (used 6.25% of requested size) Deleting ring buffer "streamdev-streamer" size: 262144 / 4194304 (used 6.25% of requested size) Deleting ring buffer "streamdev-streamer" size: 262144 / 4194304 (used 6.25% of requested size) Deleting ring buffer "streamdev-streamer" size: 524288 / 4194304 (used 12.5% of requested size) Deleting ring buffer "streamdev-streamer" size: 524288 / 4194304 (used 12.5% of requested size) Deleting ring buffer "streamdev-streamer" size: 131072 / 4194304 (used 3.125% of requested size) Deleting ring buffer "streamdev-streamer" size: 262144 / 4194304 (used 6.25% of requested size) Deleting ring buffer "streamdev-streamer" size: 131072 / 4194304 (used 3.125% of requested size) Deleting ring buffer "streamdev-streamer" size: 131072 / 4194304 (used 3.125% of requested size) Deleting ring buffer "streamdev-streamer" size: 131072 / 4194304 (used 3.125% of requested size) Deleting ring buffer "streamdev-streamer" size: 131072 / 4194304 (used 3.125% of requested size) and the streamdev client TS buffer usually grows to 12.5..25% (ie 256K/512K out of 4M max). artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
[vdr] [PATCH] dynamically sized ringbuffers v2
Auto sized ringbuffers, changes since v1: - increased maximum sizes for a few rb users. Most of the time just a small part will be used, but there will be more room for times when more is required. - a little smarter cRingBufferLinear::Read(), handles the TS buffer better. - faster buffer growth. Also makes it less likely that there won't be an opportunity to resize before an overflow. - a potentially sleeping cRecorder:Receive(). Mostly to find out how much more buffer/time would have been needed, ie debugging. - more verbose logging. It still wouldn't surprise me if this version caused a few overflows, but hopefully these will be very rare. artur diff --git a/dvbdevice.c b/dvbdevice.c index 955483e..20dcf29 100644 --- a/dvbdevice.c +++ b/dvbdevice.c @@ -1184,7 +1184,7 @@ bool cDvbDevice::OpenDvr(void) CloseDvr(); fd_dvr = DvbOpen(DEV_DVB_DVR, CardIndex(), O_RDONLY | O_NONBLOCK, true); if (fd_dvr >= 0) - tsBuffer = new cTSBuffer(fd_dvr, MEGABYTE(2), CardIndex() + 1); + tsBuffer = new cTSBuffer(fd_dvr, MEGABYTE(8), CardIndex() + 1); return fd_dvr >= 0; } diff --git a/recorder.c b/recorder.c index 8bb1621..3c0e002 100644 --- a/recorder.c +++ b/recorder.c @@ -157,8 +157,20 @@ void cRecorder::Receive(uchar *Data, int Length) { if (Running()) { int p = ringBuffer->Put(Data, Length); - if (p != Length && Running()) + if (p != Length && Running()) { +for (int ms=20; ms<1000; ms+=ms) { + cCondWait::SleepMs(ms); + if (!Running()) + return; + int r = ringBuffer->Put(Data+p, Length-p); + p += r; + if (r) + dsyslog("saved extra %d bytes in recorder ring buffer after %d ms delay", r, ms); + if (p == Length || !Running()) + return; + } ringBuffer->ReportOverflow(Length - p); +} } } diff --git a/remux.c b/remux.c index da805b7..bd29b2f 100644 --- a/remux.c +++ b/remux.c @@ -1851,7 +1851,7 @@ void cTS2PES::ts_to_pes(const uint8_t *Buf) // don't need count (=188) // --- cRemux -#define RESULTBUFFERSIZE KILOBYTE(256) +#define RESULTBUFFERSIZE MEGABYTE(2) cRemux::cRemux(int VPid, const int *APids, const int *DPids, const int *SPids, bool ExitOnFailure) { diff --git a/ringbuffer.c b/ringbuffer.c index 0633bd3..39cc02e 100644 --- a/ringbuffer.c +++ b/ringbuffer.c @@ -151,13 +151,30 @@ void cRingBufferLinear::PrintDebugRBL(void) } #endif +// cRingBufferLinear are dynamically sized, or at least we can pretend they are ;) +// We treat 'Size' as the maximum size, but start with a small buffer, which can +// grow later as it fills up. Memory is always allocated for the full buffer, but +// the unused RAM portion remains untouched until (if at all) it is actually needed. +// Note that we can not start with a larger than requested buffer because there are +// ring buffer users that cause crashes then (eg softdevice mpegdecoder absolutely +// needs 32/64k). + +// Startup size. 64k still causes overflows before buffer starts to grow, 128k doesn't. +// The buffers grow to 200..500k anyway, so maybe increasing this a bit more would +// make sense, but let's first see if it's really needed. +// In fact 128k is on the low side, but let's try not going for 256k yet. +#define DEFRBSIZE KILOBYTE(128) + cRingBufferLinear::cRingBufferLinear(int Size, int Margin, bool Statistics, const char *Description) -:cRingBuffer(Size, Statistics) +:cRingBuffer(min(Size,DEFRBSIZE), Statistics) { description = Description ? strdup(Description) : NULL; tail = head = margin = Margin; gotten = 0; buffer = NULL; + maxsize = Size; + growbuf = 0; + dsyslog("New ring buffer \"%s\" size: %d margin: %d", description, Size, Margin ); if (Size > 1) { // 'Size - 1' must not be 0! if (Margin <= Size / 2) { buffer = MALLOC(uchar, Size); @@ -183,6 +200,8 @@ cRingBufferLinear::~cRingBufferLinear() #ifdef DEBUGRINGBUFFERS DelDebugRBL(this); #endif + dsyslog("Deleting ring buffer \"%s\" size: %d / %d (used %g%% of requested size)", description, + size, maxsize, size/(double)maxsize*100.0 ); free(buffer); free(description); } @@ -205,8 +224,21 @@ void cRingBufferLinear::Clear(void) EnablePut(); } +// Must only be called by the producer (ie Read()/Put()), not the consumer (Get()). +void cRingBufferLinear::GrowBuffer(void) +{ + size = min(size+size, maxsize); + dsyslog("Enlarging ring buffer \"%s\": %d bytes (trigger %d)", + description, size, growbuf); + growbuf = 0; +} + int cRingBufferLinear::Read(int FileHandle, int Max) { + if (Available() >= size/2) + growbuf = 1; + if (growbuf && head>=tail && size 0) ? diff - 1 : Size() - head; @@ -219,6 +251,10 @@ int cRingBufferLinear::Read(int FileHandle, int Max) Count = safe_read(FileHandle, buffer
Re: [vdr] [PATCH] dynamically sized ringbuffers v1
Klaus Schmidinger wrote: > On 05/09/07 20:56, Artur Skawina wrote: >>> void cRecorder::Receive(uchar *Data, int Length) >>> it simply drops any data that does not fit into the buffer, which would be >>> fine >>> for live viewing, but isn't ideal when recording. >>> >>> Can we try harder not to loose data here? IOW can this function sleep (and >>> retry)? >> It took a while to trigger the condition again, but now it happened and >> trying a bit >> harder payed off. Running vdr /w following patch resulted in this log (and >> no overflow): >>From receiver.h: > > virtual void Receive(uchar *Data, int Length) = 0; >///< This function is called from the cDevice we are attached > to, and >///< delivers one TS packet from the set of PIDs the cReceiver > has requested. >///< The data packet must be accepted immediately, and the > call must return > > ** >///< as soon as possible, without any unnecessary delay. Each > TS packet > ** >///< will be delivered only ONCE, so the cReceiver must make > sure that >///< it will be able to buffer the data if necessary. > Yes, saw that, that's why asked if it couldn't sleep after all... And is why i made it sleep only for for short periods. ASAP is relative ;) Avoiding corrupting recordings is worth some small glitches when live viewing. Now, the max 1.26s sleep is probably too much, but i'll need to run with this for a few weeks to gather some meaningful statistics. So far the overflow has triggered twice in three days, first time (vanilla Receive()) some data was lost, second time the "unnecessary" delays rescued a few bytes. Another way to look at this is -- given how rarely this happens i don't think it's statistically significant ;) There will likely be many more errors due to other factors during these 36h or so of recordings. So this change isn't necessary, but i'd like to find out if the data loss can be avoided and it lets me monitor that. The advantage of auto tuning the buffer sizes is that the max size can be increased cheaply, as the unneeded parts won't be used at all; i'll do this for the TS buffer; whether that will be enough to cope with the extra delays i'll probably find out in a few days... artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] [PATCH] dynamically sized ringbuffers v1
> void cRecorder::Receive(uchar *Data, int Length) > { > if (Running()) { > int p = ringBuffer->Put(Data, Length); > if (p != Length && Running()) > ringBuffer->ReportOverflow(Length - p); > } > } > > it simply drops any data that does not fit into the buffer, which would be > fine > for live viewing, but isn't ideal when recording. > > Can we try harder not to loose data here? IOW can this function sleep (and > retry)? It took a while to trigger the condition again, but now it happened and trying a bit harder payed off. Running vdr /w following patch resulted in this log (and no overflow): 20:31:35 vdr: [16328] buffer usage: 70% (tid=16327) 20:31:35 vdr: [16328] buffer usage: 80% (tid=16327) 20:31:35 vdr: [16328] buffer usage: 90% (tid=16327) 20:31:35 vdr: [16328] buffer usage: 100% (tid=16327) 20:31:35 vdr: [16327] Enlarging ring buffer "Result": 262144 bytes (trigger 3) 20:31:35 vdr: [16329] Enlarging ring buffer "TS": 262144 bytes (trigger 2) 20:31:35 vdr: [16328] Enlarging ring buffer "Recorder": 262144 bytes (trigger 3) 20:31:35 vdr: [16328] buffer usage: 0% (tid=16327) 20:31:35 vdr: [16328] saved extra 153 bytes in recorder ring buffer after 80 ms delay artur diff --git a/recorder.c b/recorder.c index 8bb1621..3c0e002 100644 --- a/recorder.c +++ b/recorder.c @@ -157,8 +157,20 @@ void cRecorder::Receive(uchar *Data, int Length) { if (Running()) { int p = ringBuffer->Put(Data, Length); - if (p != Length && Running()) + if (p != Length && Running()) { +for (int ms=20; ms<1000; ms+=ms) { + cCondWait::SleepMs(ms); + if (!Running()) + return; + int r = ringBuffer->Put(Data+p, Length-p); + p += r; + if (r) + dsyslog("saved extra %d bytes in recorder ring buffer after %d ms delay", r, ms); + if (p == Length || !Running()) + return; + } ringBuffer->ReportOverflow(Length - p); +} } } ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] [PATCH] dynamically sized ringbuffers v1
> Patch below changes the interpretation of the 'Size' parm given when creating > a buffer from a > fixed length to a max limit. The buffers start out much smaller (currently > 128k, instead of many > megabytes) and grow automatically when they become almost full. This way all > callers benefit, > and they don't have to be modified at all. > I've tested this on two vdrs for a few hours and it seems to work, but it > needs a lot > more testing. I did a few test recordings (total duration likely more than 24h) and the new code seems mostly fine; just some tweaking of the sizes and grow heuristics left. However one problem did appear after many hours of simultaneous recordings: 08:17:38 vdr: [24052] buffer usage: 70% (tid=24051) 08:17:39 vdr: [24052] buffer usage: 80% (tid=24051) 08:17:39 vdr: [24052] buffer usage: 90% (tid=24051) 08:17:39 vdr: [24052] buffer usage: 100% (tid=24051) 08:17:39 vdr: [24052] ERROR: 1 ring buffer overflow (61 bytes dropped) 08:17:45 vdr: [24052] ERROR: 9053 ring buffer overflows (1701964 bytes dropped) 08:17:47 vdr: [24052] Enlarging ring buffer "Recorder": 442368 bytes 08:17:47 vdr: [24052] buffer usage: 0% (tid=24051) Apparently the 'Recorder' consumer stalled for several seconds at a point when growing the ringbuffer wasn't possible - to keep Get/Put lockless resizing is only possible when the ringbuffer data is continuous; and if the consumer stops making progress at the wrong moment, the buffer will not be enlarged and overflow. In this case the buffer was ~300k and ~2m would have been needed. I can think of a few ways to make the ringbuffer growing more aggressive, but they all make the buffer handling much more complex, so i'd like to avoid that. Looking at the caller: void cRecorder::Receive(uchar *Data, int Length) { if (Running()) { int p = ringBuffer->Put(Data, Length); if (p != Length && Running()) ringBuffer->ReportOverflow(Length - p); } } it simply drops any data that does not fit into the buffer, which would be fine for live viewing, but isn't ideal when recording. Can we try harder not to loose data here? IOW can this function sleep (and retry)? (not that i think doing this would be enough; it could help for short stalls, but for the longer ones a much larger buffer size is necessary, i'll have to find a clean way to handle that anyway. It's kind of hard to test though as the overflow only happened once during ~24h of recording...) artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
[vdr] [PATCH] dynamically sized ringbuffers v1
vdr uses quite a lot of memory, RSS here used to be ~70M, so i decided to see if it could be reduced a bit. First suspects became the ringbuffers, cause i was seeing a lot of log lines such as "buffer stats: 96068 (1%) used". Turns out many buffers are way over sized, only a few percent of the memory are typically used, yet because of how a ringbuffer works _all_ of it is constantly accessed. I first tried to make the ringbuffer users smarter about picking the size and eg after changing the transfer buffer to start out small and use bigger sizes only when the current one isn't large enough it seemed a more reasonable size for that vdr instance was 384k instead of 2M. Similarly the TS buffer could be just a few hundred K instead of 2M. But that approach was not ideal as there isn't anything one could do after realizing the buffer is too small, it could be grown next time, after channel switch etc, but that's it. Also some of the huge buffers came from plugins, and I wasn't all that excited about fixing all of them (one example w/ the one-digit % fill was streamdev-server). It is hard to estimate what the proper size would be under load, with many simultaneous recordings. So I did attack this issue differently -- by making the ringbuffers themselves smarter. Patch below changes the interpretation of the 'Size' parm given when creating a buffer from a fixed length to a max limit. The buffers start out much smaller (currently 128k, instead of many megabytes) and grow automatically when they become almost full. This way all callers benefit, and they don't have to be modified at all. The buffers are actually always allocated using the max size, but only the necessary buffer part is used, the rest of the memory is never accessed; this is done this way so that no extra locking is necessary. I've tested this on two vdrs for a few hours and it seems to work, but it needs a lot more testing. There is some extra logging in this patch so that you can see what's going on; every time a ring buffer gets deleted it logs a line which tells you exactly how much memory was requested and how much was really needed. vdr: [19729] Deleting ring buffer "TS" size: 196608 / 2097152 (used 9.375% of requested size) vdr: [19718] Deleting ring buffer "Result" size: 262144 / 262144 (used 100% of requested size) vdr: [19718] Deleting ring buffer "Recorder" size: 442368 / 5242880 (used 8.4375% of requested size) vdr: [19718] Deleting ring buffer "Result" size: 262144 / 262144 (used 100% of requested size) vdr: [19718] Deleting ring buffer "Recorder" size: 131072 / 5242880 (used 2.5% of requested size) vdr: [19733] Deleting ring buffer "TS" size: 131072 / 2097152 (used 6.25% of requested size) vdr: [19718] Deleting ring buffer "Result" size: 262144 / 262144 (used 100% of requested size) vdr: [19718] Deleting ring buffer "Recorder" size: 442368 / 5242880 (used 8.4375% of requested size) So each of the several 'Recorder' buffers was an order of magnitude smaller than w/o this patch and several megabytes less ram was used. artur diff --git a/ringbuffer.c b/ringbuffer.c index 0633bd3..8dd1efe 100644 --- a/ringbuffer.c +++ b/ringbuffer.c @@ -151,13 +151,29 @@ void cRingBufferLinear::PrintDebugRBL(void) } #endif +// cRingBufferLinear are dynamically sized, or at least we can pretend they are ;) +// We treat 'Size' as the maximum size, but start with a small buffer, which can +// grow later as it fills up. Memory is always allocated for the full buffer, but +// the unused RAM portion remains untouched until (if at all) it is actually needed. +// Note that we can not start with a larger than requested buffer because there are +// ring buffer users that cause crashes then (eg softdevice mpegdecoder absolutely +// needs 32/64k). + +// Startup size. 64k still causes overflows before buffer starts to grow, 128k doesn't. +// The buffers grow to 200..500k anyway, so maybe increasing this a bit more would +// make sense, but let's first see if it's really needed. +#define DEFRBSIZE KILOBYTE(128) + cRingBufferLinear::cRingBufferLinear(int Size, int Margin, bool Statistics, const char *Description) -:cRingBuffer(Size, Statistics) +:cRingBuffer(min(Size,DEFRBSIZE), Statistics) { description = Description ? strdup(Description) : NULL; tail = head = margin = Margin; gotten = 0; buffer = NULL; + maxsize = Size; + growbuf = 0; + dsyslog("New ring buffer \"%s\" size: %d margin: %d", description, Size, Margin ); if (Size > 1) { // 'Size - 1' must not be 0! if (Margin <= Size / 2) { buffer = MALLOC(uchar, Size); @@ -183,6 +199,8 @@ cRingBufferLinear::~cRingBufferLinear() #ifdef DEBUGRINGBUFFERS DelDebugRBL(this); #endif + dsyslog("Deleting ring buffer \"%s\" size: %d / %d (used %g%% of requested size)", description, + size, maxsize, size/(double)maxsize*100.0 ); free(buffer); free(description); } @@ -207,6 +225,13 @@ v
[vdr] remote plugin ignoring some keys
A while ago the remote plugins "telnet" server started acting strangely here; it would sometimes ignore all cursor and function keys, yet other keys would work perfectly. I'm not sure when exactly this began, most likely a newer kernel triggered this. It does not happen always, but when it does it makes navigating the OSD practically impossible. Turns out the remote plugin gets confused when a read() call returns not a full escape sequence, but just the first \0x27 byte and the next read() receives the remaining two or three bytes. And this is exactly what happens (the client for some reason actually sends two tcp packets containing just eg "\0x1b" and "\0x5b\0x43"). This quick patch made the remote osd usable again; it isn't perfect as keys can still be lost (see comments), but it's a huge improvement - only some key repeats are dropped, not all multibyte sequences. artur diff -urNp remote-0.3.9.org/remote.c remote-0.3.9/remote.c --- remote-0.3.9.org/remote.c 2006-12-03 14:46:55.0 + +++ remote-0.3.9/remote.c 2007-04-29 20:45:13.0 + @@ -517,6 +517,27 @@ uint64_t cRemoteDevTty::getKey(void) uint64_t code = 0; n = read(fh, &code, sizeof code); + +if (code==27 && n==1) +{ + // Ugh. What happened: user pressed a key that generates an escape + // sequence (eg up/down) and we got only the first '\e'; now we have + // to fetch the remaining bytes as otherwise we end up ignoring almost + // every such key press and navigating the osd becomes impossible. + // We try to read more bytes and hope that the result makes sense; + // this often works, but not always. When eg holding down a key we + // can receive several (repeated) sequences and these are then ignored. + // Dropping a few key presses (mostly repeats) is still much better + // than dropping nearly _all_ of them... + // (Unfortunately the sequences do differ in length, hence either + // parsing them or matching them w/ the expected ones would be needed + // to avoid loosing sync here) + int m = read(fh, ((char*)&code)+1, sizeof code-1); + if (m > 0) + n += m; +} +//dsyslog("cRemoteDevTty::getKey: %d %lld", n, code); + return (n > 0) ? code : INVALID_KEY; } ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] Freezes with Reiserfs
Sven Schwyn wrote: > I'm having strange issues with my VDR box lately. However, I'm not sure > at all it has anything to do with VDR as other services are running on > it, too. > > The Gentoo box freezed four times in the past month, twice of which the > very minute a VDR timer stopped. (Maybe even all four times, but I one thing that comes to mind is that vdr calls fdatasync when closing an mpg file, maybe this trips up reiserfs somehow. You could try to comment out that call (in tools.c cUnbufferedFile::Close). Also apparently there is some reiserfs option to reduce the writeout rate, maybe turning that off would help. artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] [PATCH] limit streamdev client's ringbuffer timeouts to more sane values
Stone wrote: > > Is this patch for streamdev a direct result from selecting the new > "Tickless System (Dynamic Ticks)" option in kernel 2.6.21? If yes, then CONFIG_HIGH_RES_TIMERS "High Resolution Timer Support". Whether NO_HZ makes a difference I'm not sure, i ran into this w/ both options on. (launching "nice --15 vdr" w/streamdev+softdevice caused X to become unresponsive; i had to reset the machine. X was barely usable even with the vdr client niced) > does the new "dynamic" timer effect vdr's performance as well? After increasing that streamdev 1us usleep I don't see much difference in vdr (+streamdev+softdevice) performance. Haven't updated to highres timers on the vdr server yet, but i doubt it will change much. (iirc vdr itself limits the delays to >=3ms). artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] [PATCH] limit streamdev client's ringbuffer timeouts to more sane values
Stone wrote: > > sure, just changing it to 'usleep(1)' works too. Is there a > reason to avoid > the ringbuffer infrastructure? > > No reason in particular, I just wanted to test which one worked best > with performance. Thanks for the patch. :) I tried the 1us -> (1000|1)us sleep approach first, before using the ringbuffer timeouts -- saw no noticeable difference wrt performance. 10ms timeouts seemed to be enough (it's the resolution of a HZ==100 kernel) and gave similar interrupt and cs numbers as w/ low-res timers. artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] [PATCH] limit streamdev client's ringbuffer timeouts to more sane values
Stone wrote: > On 4/5/07, *Artur Skawina* <[EMAIL PROTECTED] <mailto:[EMAIL PROTECTED]>> > wrote: > With the high-res timers in kernel 2.6.21+ usleep(1) is no longer > treated as > usleep(1) and the streamdev client is almost unusable; it uses > most of the cpu > and causes hundreds of thousands context switches per second. > This gets rid of the almost-busy-loop. > How about his approach? > > --- streamdev/client/filter.c.orig 2007-04-05 20:45:04.0 -0700 > +++ streamdev/client/filter.c 2007-04-05 20:45: 44.0 -0700 > @@ -135,7 +135,7 @@ > } > m_RingBuffer->Del(TS_SIZE); > } else > - usleep(1); > + cCondWait::SleepMs(1); > } > } sure, just changing it to 'usleep(1)' works too. Is there a reason to avoid the ringbuffer infrastructure? artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
[vdr] [PATCH] limit streamdev client's ringbuffer timeouts to more sane values
Well, the streamdev-client reads data from a ringbuffer and when there isn't anything to read it tries to sleep for 1us and loops. This wasn't a problem when the timer resolution was in the 1000..1us range (1000..100Hz); the usleep(1) call slept for one or more milliseconds. With the high-res timers in kernel 2.6.21+ usleep(1) is no longer treated as usleep(1) and the streamdev client is almost unusable; it uses most of the cpu and causes hundreds of thousands context switches per second. This gets rid of the almost-busy-loop. artur diff -urNp streamdev.org/client/filter.c streamdev/client/filter.c --- streamdev.org/client/filter.c 2005-11-06 17:43:58.0 +0100 +++ streamdev/client/filter.c 2007-04-05 23:25:11.0 +0200 @@ -57,6 +57,7 @@ cStreamdevFilters::cStreamdevFilters(voi cThread("streamdev-client: sections assembler") { m_Active = false; m_RingBuffer = new cRingBufferLinear(MEGABYTE(1), TS_SIZE * 2, true); + m_RingBuffer->SetTimeouts(10, 10); Start(); } @@ -111,8 +112,7 @@ void cStreamdevFilters::Action(void) { } } m_RingBuffer->Del(TS_SIZE); - } else - usleep(1); + } } } ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] Too many open files - error
Kartsa wrote: >>> ERROR: /dev/dvb/adapter0/demux0: Too many open files >>> >>> I do not recall seeing this earlier. This came when fourth simultaneous >>> recording started. >> dvb/dvb-core/dmxdev.c- if (i == dmxdev->filternum) { >> dvb/dvb-core/dmxdev.c- mutex_unlock(&dmxdev->mutex); >> dvb/dvb-core/dmxdev.c: return -EMFILE; >> dvb/dvb-core/dmxdev.c-} >> >> IOW you ran out of filters. >> yes, the error code should probably be different (eg EBUSY). >> > So, why did I ran out of filters? Why did it happen? Why doesn't it > happen on my other vdr box? And what does it cause? The recording did > succeed. you (well, vdr) used more filters than were available (in hw)? less simultaneous recordings, different hw and/or config? ("hw_sections" module parm?). Apparently vdr recovered somehow (eg another recording ended, the filters were opened in a different order etc). Can't help you much more, I haven't used a FF card in years -- and since then all dvb-related problems vanished... It wasn't 100% reliable when live viewing, not to mention a single recording; a few novas have absolutely no problems w/ ~ 4-6 simultaneous recordings (never counted, but i believe it could go even higher) and this is on really low end hw (P2)... ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] Too many open files - error
Kartsa wrote: > I was about to test the performance of vdr when I stumbled on this message > > ERROR: /dev/dvb/adapter0/demux0: Too many open files > > I do not recall seeing this earlier. This came when fourth simultaneous > recording started. > > Is this a vdr, dvb, or firmware issue? Seems like dvb but I really do > not know. > > This is what was in the log after the fourth recording > > Feb 19 21:53:00 vdr: [2226] timer 4 (7 2153-2200 'TestRec4') start > Feb 19 21:53:00 vdr: [2226] record > /srv/vdr/TestRec4/2007-02-19.21.53.50.99.rec > Feb 19 21:53:00 vdr: [2226] ERROR: /dev/dvb/adapter0/demux0: Too many > open files > Feb 19 21:53:00 vdr: [2226] ERROR (dvbdevice.c,673): Too many open files > Feb 19 21:53:00 vdr: [2226] ERROR: can't set PID 680 on device 1 > Feb 19 21:53:00 vdr: [2226] ERROR (dvbdevice.c,688): Bad file descriptor grep -20 -r EMFILE * dvb/dvb-core/dmxdev.c-static int dvb_demux_open(struct inode *inode, struct file *file) dvb/dvb-core/dmxdev.c-{ dvb/dvb-core/dmxdev.c- struct dvb_device *dvbdev = file->private_data; dvb/dvb-core/dmxdev.c- struct dmxdev *dmxdev = dvbdev->priv; dvb/dvb-core/dmxdev.c- int i; dvb/dvb-core/dmxdev.c- struct dmxdev_filter *dmxdevfilter; dvb/dvb-core/dmxdev.c- dvb/dvb-core/dmxdev.c- if (!dmxdev->filter) dvb/dvb-core/dmxdev.c- return -EINVAL; dvb/dvb-core/dmxdev.c- dvb/dvb-core/dmxdev.c- if (mutex_lock_interruptible(&dmxdev->mutex)) dvb/dvb-core/dmxdev.c- return -ERESTARTSYS; dvb/dvb-core/dmxdev.c- dvb/dvb-core/dmxdev.c- for (i = 0; i < dmxdev->filternum; i++) dvb/dvb-core/dmxdev.c- if (dmxdev->filter[i].state == DMXDEV_STATE_FREE) dvb/dvb-core/dmxdev.c- break; dvb/dvb-core/dmxdev.c- dvb/dvb-core/dmxdev.c- if (i == dmxdev->filternum) { dvb/dvb-core/dmxdev.c- mutex_unlock(&dmxdev->mutex); dvb/dvb-core/dmxdev.c: return -EMFILE; dvb/dvb-core/dmxdev.c- } dvb/dvb-core/dmxdev.c- dvb/dvb-core/dmxdev.c- dmxdevfilter = &dmxdev->filter[i]; dvb/dvb-core/dmxdev.c- mutex_init(&dmxdevfilter->mutex); dvb/dvb-core/dmxdev.c- file->private_data = dmxdevfilter; dvb/dvb-core/dmxdev.c- dvb/dvb-core/dmxdev.c- dvb_ringbuffer_init(&dmxdevfilter->buffer, NULL, 8192); dvb/dvb-core/dmxdev.c- dmxdevfilter->type = DMXDEV_TYPE_NONE; dvb/dvb-core/dmxdev.c- dvb_dmxdev_filter_state_set(dmxdevfilter, DMXDEV_STATE_ALLOCATED); dvb/dvb-core/dmxdev.c- dmxdevfilter->feed.ts = NULL; dvb/dvb-core/dmxdev.c- init_timer(&dmxdevfilter->timer); dvb/dvb-core/dmxdev.c- dvb/dvb-core/dmxdev.c- mutex_unlock(&dmxdev->mutex); dvb/dvb-core/dmxdev.c- return 0; dvb/dvb-core/dmxdev.c-} IOW you ran out of filters. yes, the error code should probably be different (eg EBUSY). ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] Is there a way to give inode entries a higher cache priority than data in the linux kernel?
Carsten Koch wrote: > I have a distributed VDR system in my house with a lot of disks > that are NFS mounted by VDR PCs in two rooms. > In order to conserve energy, I have used hdparm to set a spin down > delay after which the disks turn themselves off. > When /video/.update is touched (one of the VDR PCs creates/deletes/ > edits a recording, I move a recording into a folder, etc.) or when > the vdr program is started, it reads all directories from all disks. > Most of these directories are unchanged, so there really is no need > to spin up the disk just to read a few inode entries. > However, my observation is that they are always spun up. > So my questions are: > > 1) Is there a bug in the linux kernel that makes it spin up >the disk needlessly even if the data are still in the cache? > > 2) Is there a way to configure the kernel, so the inode entries >are locked in the cache or at least get a much higher >cache priority than ordinary data? The problem is that the nfs server reads (and writes, when editing a recording on a client) large amounts of data and the metadata (which hasn't been accessed since the last /video scan) gets evicted from the cache. This usually does not happen w/ a local vdr and on the client vdrs because fadvice() makes sure that the cached video data gets freed before the system experiences any memory pressure. But fadvise() only helps locally, ie the nfs client drops the data, but the nfs server does not. I've gotten used to this; at one point, when doing the fadvise changes, i was going to implement O_DIRECT-based i/o, but decided it wasn't worth it, because of the complexity and small gains (you can't really avoid a data copy w/ the current vdr design (alignment restrictions) and the code still has to handle all the cases where O_DIRECT and/or AIO isn't available, older kernels, other filesystems etc). What you could try is: o play with /proc/sys/vm/vfs_cache_pressure; try decreasing it. significantly. what you describe as "2)" above would be "vfs_cache_pressure=0". Haven't tried this myself; does it help? If not the other options would probably be a bit more complex (like modifying nfsd or writing a dedicated fuse fs). artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr