Re: [vdr] making VDR ext4-ready
Marcel Witte wrote: So ext4 seems to be perfect for a video-partition, but to make it more perfect, it would be nice if VDR could use the fallocate()-systemcall as mentioned in the article. This would prevent fragmentation in the file system. a) i wouldn't fully trust ext4 for a few more months at least (for a system partition, which you can toss and easily rebuild it's probably ready, for a large, not otherwise backuped, data collection it may not yet be) b) fallocate is relatively new, so you'd require a new libc and kernel (same reason i used fadvise and not sync_file_range) c) libc w/o proper kernel/fs support can emulate fallocate by prewriting zeros to every single block -- not likely what you'd want; this makes above (b) even more relevant. artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] sparse DVB adapter numbers
Klaus Schmidinger wrote:
On 21.03.2009 14:17, Artur Skawina wrote:
Patrick Rother wrote:
For a reason not to go in details too much here, I would like to make
vdr recognise sparse DVB adapter numbers.
drwxr-xr-x 2 root root 200 Mar 19 11:59 adapter0/
drwxr-xr-x 2 root root 120 Mar 19 11:59 adapter1/
drwxr-xr-x 2 root root 120 Mar 19 11:59 adapter5/
drwxr-xr-x 2 root root 120 Mar 19 11:59 adapter6/
drwxr-xr-x 2 root root 120 Mar 19 11:59 adapter7/
But vdr uses only the first two devices.
Is there any easy possibility to patch the code accordingly?
./vdr -D 0 -D 1 -D 5 -D 6 -D 7
I'm afraid this won't work, because VDR stops scaning for DVB devices
as soon as it hits a gap.
Good i didn't know that years ago when setting up my vdr boxes... :)
IOW this approach has been working for me for quite a while w/o any
problems. Note i'm still at 1.4.x and my 'gaps' actually contain
adapters (unused by this vdr instance).
If the gaps really are a problem you could always renumber the adapters
using udev rules such as
SUBSYSTEM==dvb, SUBSYSTEMS==pci, KERNELS==:00:0e.0, PROGRAM=/bin/sh
-c 'K=%k; K=$${K#dvb}; printf dvb/adapter%%i/%%s 0 $${K#*.}',
NAME:=%c
SUBSYSTEM==dvb, SUBSYSTEMS==pci, KERNELS==:02:09.0, PROGRAM=/bin/sh
-c 'K=%k; K=$${K#dvb}; printf dvb/adapter%%i/%%s 1 $${K#*.}',
NAME:=%c
etc
artur
___
vdr mailing list
vdr@linuxtv.org
http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] channel-change-prediction, doable? stupid idea?
Udo Richter wrote: On 21.01.2009 19:33, Gerald Dachs wrote: Assumed I have 2 free Tuners and I zapp through the channels of one tuner. Would it be possible to tune the 2. tuner to the same time to the after next channel in change direction, so that for the next channel change in the same direction it would be possible to channge to the 2. tuner instead of changing the channels of the 1. tuner Channel switching delay is caused by two delays: First, the time to tune and lock the new transponder, second, the time till the next compressed stream starting point (I-Frame or audio syncword) is reached. This second time can be up to a second by its own. To get instant switching, you would have to also decode the second stream in the background, so that the current frame is available the moment it is needed. Or you can buffer the last GOP of the stream and go back to that point on channel switch. You would have to ring-buffer up to one complete GOP permanently in memory to keep the delay from there on. Both things are tricky to implement, require work on the output devices, and are probably not worth the effort. Yes, the only case i think this could be useful is when the number of tuners = transponders/muxes and you always receive all of them. Hmm, it might actually make sense for dvb-t (or -c) in some cases. Anyway, back when i was using a dvb-s ff card the tuning itself seemed to be slow, so what i did was cache some parms such as freq, srate, fec, voltage and tone in the driver, to avoid touching the hw/firmware for those that didn't change (the dvb api resulted in the driver always getting a new set of parms, even when some/most of them hadn't changed). That seemed to speed up the channel switches a bit, but as the card was never 100% reliable and would sometimes fail to correctly tune, i never went further with that. Now i think it was probably the pci host that was causing the trouble, and the changes may have been fine. artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] Frames per second PAL vs. NTSC
Klaus Schmidinger wrote:
Detecting the frame rate is done by looking at the PTS values, so
it is independent of the actual broadcast system.
Using this code for converting frame numbers into hh:mm:ss.ff...
#include math.h
#include stdio.h
int main(void)
{
double FramesPerSecond = double(9) / 3003;
//FramesPerSecond = 25;
for (int Index = 0; Index 1; Index++) {
double Seconds;
int f = round(modf(Index / FramesPerSecond, Seconds) *
FramesPerSecond) + 1;
int s = int(Seconds);
int m = s / 60 % 60;
int h = s / 3600;
s %= 60;
printf(%3d , Index);
printf(%15.9f , Index / FramesPerSecond);
printf(%d:%02d:%02d.%02d, h, m, s, f);
if (f 30) printf( *);
printf(\n);
}
}
... sometimes results in a 31st frame:
9978 332.93260 0:05:32.29
9979 332.96597 0:05:32.30
9980 332.99933 0:05:32.31 *
9981 333.03270 0:05:33.02
9982 333.06607 0:05:33.03
Any ideas how to fix this?
eg
- int f = round(modf(Index / FramesPerSecond, Seconds) * FramesPerSecond) + 1;
+ int f = round(modf(Index / FramesPerSecond, Seconds) * FramesPerSecond + 0.5
);
note that some 'seconds' will contain only 29 frames. (see index 510 in the
original
and 1019 in the fixed version)
artur
___
vdr mailing list
vdr@linuxtv.org
http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] Frames per second PAL vs. NTSC
Klaus Schmidinger wrote:
On 05.01.2009 13:31, Artur Skawina wrote:
Klaus Schmidinger wrote:
Detecting the frame rate is done by looking at the PTS values, so
it is independent of the actual broadcast system.
Using this code for converting frame numbers into hh:mm:ss.ff...
#include math.h
#include stdio.h
int main(void)
{
double FramesPerSecond = double(9) / 3003;
//FramesPerSecond = 25;
for (int Index = 0; Index 1; Index++) {
double Seconds;
int f = round(modf(Index / FramesPerSecond, Seconds) *
FramesPerSecond) + 1;
int s = int(Seconds);
int m = s / 60 % 60;
int h = s / 3600;
s %= 60;
printf(%3d , Index);
printf(%15.9f , Index / FramesPerSecond);
printf(%d:%02d:%02d.%02d, h, m, s, f);
if (f 30) printf( *);
printf(\n);
}
}
... sometimes results in a 31st frame:
9978 332.93260 0:05:32.29
9979 332.96597 0:05:32.30
9980 332.99933 0:05:32.31 *
9981 333.03270 0:05:33.02
9982 333.06607 0:05:33.03
Any ideas how to fix this?
eg
- int f = round(modf(Index / FramesPerSecond, Seconds) * FramesPerSecond) +
1;
+ int f = round(modf(Index / FramesPerSecond, Seconds) * FramesPerSecond +
0.5 );
note that some 'seconds' will contain only 29 frames. (see index 510 in the
original
and 1019 in the fixed version)
I'm afraid this isn't feasible. The '+1' is done to make the first frame
(at Index 0) have number '1'. With your change it would be numbered '0'.
Klaus
no, try it :)
and you can also drop the round call:
- int f = round(modf(Index / FramesPerSecond, Seconds) * FramesPerSecond) + 1;
+ int f = modf((Index + 0.5) / FramesPerSecond, Seconds) * FramesPerSecond + 1;
which will move the 'missing' frames to the same locations as in the original.
artur
___
vdr mailing list
vdr@linuxtv.org
http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] Tuning failures for transponder-locked tuners (VDR 1.4.7)
Richard F wrote:
Teemu,
I had a similar problem with budget Freecom / WT-220U USB stick tuners.
I usually had to swap multiplex/transponder to get them to re-tune reliably.
I observed that if they lost tuning, vdr made them re-try (which I saw in the
log)
but eventually would bomb out if it couldn't tune, which is maybe what you're
seeing?
Originally I logged it as a DVB driver bug, but the maintainer said it was an
application bug.
I found the following, which requires you to patch/rebuild vdr from source.
Basically it just adds a small delay to the tuning process, and it works 100%
for me.
I had to do this on 1.4.7 and carried it over in 1.6.0 also - so no, vanilla
1.6 won't help you.
It would be good if something equivalent was in vanilla VDR but I suppose it
slows
some systems by a fraction of a second when tuning.
Perhaps a budget card option?
http://www.vdr-portal.de/board/thread.php?postid=639048
The actual patch is at http://tafe.unkelhaeusser.de/vdr/dvbdevice.c.patch
If the .1s delay helps it's probably some kind of race, and if restarting
the app (ie close/open/tune sequence) does not make the device work it is
clearly a driver bug. Adding random delays to apps in order to work around
driver bugs is not the right solution.
Fixing the driver would be, but if that's not an option, you could add the
.1s delay to the open-demux path of the driver; that way only users of that
driver would suffer until it's fixed properly.
If the driver does recover after some time, you could do something like this
instead of the above patch:
int cDvbDevice::OpenFilter(u_short Pid, u_char Tid, u_char Mask)
{
const char *FileName = *cDvbName(DEV_DVB_DEMUX, CardIndex());
int f = open(FileName, O_RDWR | O_NONBLOCK);
+ if (f==-1) {
+ usleep(10);
+ f = open(FileName, O_RDWR | O_NONBLOCK);
+ }
if (f = 0) {
dmx_sct_filter_params sctFilterParams;
to limit the cost of the delays somewhat, but it won't work if the first
open call messes up the driver...
artur
___
vdr mailing list
vdr@linuxtv.org
http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] PCI fun (RGB/PAL over VGA at variable frame rate)
Gavin Hamill wrote: Over the last days, Thomas and I have been trying to sort out why my nearly-identical machine couldn't run his VGA sync patches properly. The key difference is my Radeon 7000VE is PCI, whilst his is AGP. I tried the PCI Radeon in two old Pentium-3 era machines, and on my modern Pentium D930 desktop, all with the same behaviour - fullscreen video over PCI causes huge CPU usage in the Xorg process, even when using xv 'acceleration'. When I switch the PCI Radeon for a PCI Express X300 (the very lowest 'X' series you can get), everything is glorious: Xorg CPU use is barely 1%. Unfortunately I don't have any machines with both AGP and PCI on which I can try the same OS image but we both think it's safe to conclude that PCI is just unsuitable for this task. PCI in general should be perfectly fine, for SDTV at least. While displaying SDTV (vdrsxfe) I see ~20% cpu use for X on AGP, ~44% on PCI (same machine, different heads, AGP is MGA450, PCI is MGA200). The huge difference is likely due to something else, like - display (X) driver (but even drivers which just memcpy the video data to the (xv) framebuffer should work on a modern machine) - PCI chipset (eg I had a VIA-based mobo, and it couldn't even keep up with SDTV on a PCI head, swapping the mobo for one w/ a real chipset made all problems suddenly disappear...) You could probably do some setpci tweaks to improve PCI throughput, but I doubt the gain would be enough (I'd expect 10% improvement or so). artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] Control VDR with RF remote
Wolfgang Fritz wrote:
Artur Skawina schrieb:
sure, patch vs 2.6.25 below. These are just the minimal changes needed to
make
ati-remote work w/ one type of medion rf remotes (P/N 20018071). i did this
Mine is P/N 20014751
afaict it looks exactly like the one i'm using.
The keymap for medion remotes is different from the one for ati, but as i'm
currently using mine just for vdr i never needed to fix this up (vdr does its
own mapping anyway).
I have added an additional table for Medion (selected by product ID),
but this might not work if the different Medion types have the same
product ID.
I have at least four different types of medion remotes, each with a different
layout and features; they all work w/ the same receiver so selecting based on
usb ids won't work. I don't remember if the keymaps were incompatible though.
In any case, i'm assuming the ati remote will work w/ a medion receiver and
v/v...
A per-channel module parameter is probably required.
+/*
+ * Some remotes alternate codes, this makes it easier to detect
autorepeat;
+ * to keep things simple we simply undo the code change here.
+ */
+if (d2 0x80) {
+d1 += 0x80;
+d2 -= 0x80;
+}
Interesting. I have to check if my Medion needs this. Autorepeat is
working OK but my solution is more complicated than yours, and this
might be the reason.
well, the key code changes could be used to distinguish autorepeat from a
repeated keypress, but i found the autorepeat rate to be high enough
(for a remote) and just ignore them, as above.
artur
___
vdr mailing list
vdr@linuxtv.org
http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] Control VDR with RF remote
Peer Oliver Schmidt wrote: Hello Michael, Does anybody use RF remote control with VDR instead of IR? How to configure such remotes? I used the ATI RemoteWonder with it. LIRC has support for it. Or, if your remote is supported by the kernel input layer, you can use the remote plugin. eg the X10 remotes work nicely this way: $ modprobe -v ati-remote $ EV=/dev/input/$( cd /sys/class/input grep -l 'X10' event*/device/manufacturer | sed -e 's,/.*,,' ) $ vdr ... -P remote -i $EV (some remotes such as medion need a small kernel patch in order to handle all keys and detect autorepeat properly) The first time vdr is started with the remote plugin it will ask you to press all keys and learn them, just like with any other input source. artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] Control VDR with RF remote
Wolfgang Fritz wrote:
Artur Skawina schrieb:
Peer Oliver Schmidt wrote:
Hello Michael,
Does anybody use RF remote control with VDR instead of IR? How to
configure such remotes?
I used the ATI RemoteWonder with it. LIRC has support for it.
Or, if your remote is supported by the kernel input layer, you can use
the remote plugin. eg the X10 remotes work nicely this way:
$ modprobe -v ati-remote
$ EV=/dev/input/$( cd /sys/class/input grep -l 'X10'
event*/device/manufacturer | sed -e 's,/.*,,' )
$ vdr ... -P remote -i $EV
(some remotes such as medion need a small kernel patch in order to handle
all keys and detect autorepeat properly)
Can you post this Kernel patch here (or send it to me via mail if it's
to big for the list)? I am using the Medion remote since quite a long
time with a patched ati-remote driver but I don't remember if I have
published the patch or not. And if this is not my patch, it might be
better than mine ;-)
sure, patch vs 2.6.25 below. These are just the minimal changes needed to make
ati-remote work w/ one type of medion rf remotes (P/N 20018071). i did this
years ago, but haven't updated it since, because eventually i'll need to rewrite
the driver to handle many remotes and map them to different input devices (each
remote can be set to a different channel/id, but ati-remote lets you only choose
which ones it listens to and provides a single /dev/input device for all of
them).
Other items on the TODO list:
The keymap for medion remotes is different from the one for ati, but as i'm
currently using mine just for vdr i never needed to fix this up (vdr does its
own mapping anyway).
There are other types of medion rf remotes, iirc i have 3 or 4 of them, quick
tests
showed they all seem to work w/ this patch, but may need some extra quirks
(mostly
trivial keymap mods, possibly handling of scroll wheel and touchpad). (if you
have
one that doesn't work w/ this patch and can't figure out how to make it work,
post
a link identifying it, and if i have that model i'll do the necessary changes)
artur
diff --git a/drivers/input/misc/ati_remote.c b/drivers/input/misc/ati_remote.c
index f3b86c2..38f3183 100644
--- a/drivers/input/misc/ati_remote.c
+++ b/drivers/input/misc/ati_remote.c
@@ -281,6 +281,21 @@ static const struct {
{KIND_FILTERED, 0xf4, 0x2F, EV_KEY, KEY_END, 1},/* END */
{KIND_FILTERED, 0xf5, 0x30, EV_KEY, KEY_SELECT, 1}, /* SELECT */
+ /* Medion-remote keys */
+ {KIND_FILTERED, 0xf1, 0x2c, EV_KEY, KEY_TV, 1}, /* TV */
+ {KIND_FILTERED, 0xf6, 0x31, EV_KEY, KEY_VIDEO, 1}, /* (VIDEO
DESKTOP) */
+
+ {KIND_FILTERED, 0xf7, 0x32, EV_KEY, KEY_RED, 1},/* RED */
+ {KIND_FILTERED, 0xf8, 0x33, EV_KEY, KEY_GREEN, 1}, /* GREEN */
+ {KIND_FILTERED, 0xf9, 0x34, EV_KEY, KEY_YELLOW, 1}, /* YELLOW */
+ {KIND_FILTERED, 0xfa, 0x35, EV_KEY, KEY_BLUE, 1}, /* BLUE */
+
+ {KIND_FILTERED, 0xfb, 0x36, EV_KEY, KEY_MOVE, 1}, /* RENAME */
+ {KIND_FILTERED, 0xfc, 0x37, EV_KEY, KEY_NEW, 1},/* ACQUIRE
IMAGE */
+ {KIND_FILTERED, 0xfd, 0x38, EV_KEY, KEY_EDIT, 1}, /* EDIT */
+ {KIND_FILTERED, 0xfe, 0x39, EV_KEY, KEY_SCREEN, 1}, /* FULL SCREEN
*/
+ {KIND_FILTERED, 0xff, 0x3a, EV_KEY, KEY_AUDIO, 1}, /* DVD AUDIO */
+
{KIND_END, 0x00, 0x00, EV_MAX + 1, 0, 0}
};
@@ -405,6 +420,15 @@ static int ati_remote_event_lookup(int rem, unsigned char
d1, unsigned char d2)
{
int i;
+ /*
+* Some remotes alternate codes, this makes it easier to detect
autorepeat;
+* to keep things simple we simply undo the code change here.
+*/
+ if (d2 0x80) {
+ d1 += 0x80;
+ d2 -= 0x80;
+ }
+
for (i = 0; ati_remote_tbl[i].kind != KIND_END; i++) {
/*
* Decide if the table entry matches the remote input.
@@ -524,6 +548,19 @@ static void ati_remote_input_report(struct urb *urb)
ati_remote-old_data[1] = data[2];
ati_remote-old_jiffies = now;
+ /*
+* There apparently are two types of X10 remotes -- some
(newer?) models
+* change the data sent for a repeated button press, others
send the same
+* bytes again. The following conditional makes the latter kind
work;
+* w/o this it's impossible to quickly enter eg. 11 because
the driver
+* assumes the button is held down (is autorepeating) and
generates
+* 11 -- the second button press results in five events.
+* This also lowers the autorepeat rate, but as the alternative
seems to
+* be spurious key presses...
+*/
+ if (ati_remote-repeat_count 4)
+ ati_remote-repeat_count = 0;
+
/* Ensure we skip at least the 4 first duplicate
Re: [vdr] Problems playing ongoing recordings?
JJussi wrote: On Wednesday, 2. Aprilta 2008 05:11:37 VDR User wrote: Playback is faster then watching live tv?! Maybe if you skip past the commercials or something! I've started playing back a recording many times before it was finished and never had a problem with lockup or that the playback was going faster then live tv! Yes.. Check that attached message OR search subtitle Playback to fast.. from archives. Clements Kirchgatterer said that, that's normal... Small drift is expected -- the clock used as reference for playback (on dvb/dxr card etc) is not synced to the one used by the broadcaster. It's a similar problem to trying to record audio using one sound card and playing it back using another card in real time -- at some point you will get either over- or underruns because the sample rates are not 100% identical. I suspect some of the issues when playing ongoing recordings just after they are extended are caused by the recorder telling the system to drop the recently written data and this is interfering with the readahead done by the player, which then has to wait for the data to be refetched from disk. O_DIRECT helps here (by not using the cache at all), OTOH it could cause playback problems at the very end (due to the small delay and larger chunks used when extending the file) artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] Problems playing ongoing recordings?
alexw wrote: On Friday 28 March 2008 14:52:38 alexw wrote: On Thursday 27 March 2008 18:22:35 Artur Skawina wrote: VDR User wrote: On Thu, Mar 27, 2008 at 9:19 AM, Artur Skawina [EMAIL PROTECTED] wrote: I have a similar setup, just with 100M ethernet instead of wifi and NFS instead of samba. wifi could be a problem, but if you're able to watch the same channel live you should also be able to view a recording. Takes a lot more bandwidth to record playback then just to record so the fact that live tv is fine doesn't amount to much I don't think. I was referring to playing a finished recording and playing a file that is currently being extended by the server vdr -- alexw said that doesn't work well for him. It should, unless the disk and/or fs can't handle the two data streams concurrently, while keeping the latency low enough. I'm assuming the vdr server in powerful enough to handle the load, yes. My setup is a little bit more complicated as it is using a share drive on both machine. The two local disks are only CF. The file server is compose of 4x SATA II 500GB in raid 5 (total ~1.5 TB available) piloted by a promise controller and a fully idle 3 GHz P4 CPU. The throughput or the sustained write/read access is not a bottleneck. Here is a quick ASCII art drawing: /-- 100M --[AP]~ WIFI 54Mb ~[vdr client/FF]-[TV] [switch]--- 100M ---[CIFS/fileserver] \-- 100M -- [vdr server/B2C2]-DVB-S+DVB-T This evening I will test the provided patch. Using the patch provided by Artur, playing a finished/ongoing recording is smoother than before. Thank you for testing (and for enabling the extra logging). Does smoother than before mean that there is some improvement, but the playback still isn't perfect? The fact that the recorder hangs on to the last 4M of the stream and the data is appended in large chunks can cause problems when timeshifting by just a few seconds -- the player could try to access the not yet written data. This should only happen at the very end of an ongoing recording, within 4M of EOF (depending on the bitrate usually a couple of seconds after the live program arrived). Sometimes the player stops in the middle of a recording due to a zero size request. Here is the log: vdr: [3836] dvbplayer thread started (pid=3643, tid=3836) vdr: [3836] resuming replay at index 1950 (0:01:18.01) vdr: [3837] non blocking file reader thread started (pid=3643, tid=3837) vdr: [3836] SetBrokenLink: no GOP header found in video packet vdr: [3836] setting audio track to 1 (0) vdr: [3836] playing '/var/vdr/video/SERVER/recording/2008-03-28.18.58.50.50.rec/001.vdr' unexpect stop of replay vdr: [3837] non blocking file reader thread ended (pid=3643, tid=3837) vdr: [3836] dvbplayer thread ended (pid=3643, tid=3836) - vdr: [5618] WANT: fd: 25 1068536495 .. 1068722913 SIZE: 186418 vdr: [5618] READ: fd: 25 1068536495 .. 1068666704 SIZE: 130209 jump: 0 ra: 12582912 vdr: [5618] WANT: fd: 25 1068666704 .. 1068983331 SIZE: 316627 vdr: [5618] READ: fd: 25 1068666704 .. 1068680058 SIZE: 13354 jump: 0 ra: 12582912 vdr: [5618] READ: fd: 25 1068680058 .. 1068690344 SIZE: 10286 jump: 0 ra: 12582912 vdr: [5618] READ: fd: 25 1068690344 .. 1068721839 SIZE: 31495 jump: 0 ra: 12582912 vdr: [5618] READ: fd: 25 1068721839 .. 1069246127 SIZE: 524288 jump: 0 ra: 12582912 vdr: [5618] WANT: fd: 25 1069246127 .. 1070294703 SIZE: 1048576 vdr: [5618] READ: fd: 25 1069246127 .. 1069246127 SIZE: 0 jump: 0 ra: 12582912 vdr: [5618] non blocking file reader thread ended (pid=5563, tid=5618) vdr: [5617] dvbplayer thread ended (pid=5563, tid=5617) Weird, cUnbufferedFile::Read(Size=0). I'll try to reproduce this. - vdr: [5627] READ: fd: 25 1188203807 .. 1188214024 SIZE: 10217 jump: 0 ra: 12582912 vdr: [5627] READ: fd: 25 1188214024 .. 1188226152 SIZE: 12128 jump: 0 ra: 12582912 vdr: [5627] READ: fd: 25 1188226152 .. 1188267411 SIZE: 41259 jump: 0 ra: 12582912 vdr: [5627] WANT: fd: 25 1188267411 .. 1188503927 SIZE: 236516 vdr: [5627] READ: fd: 25 1188267411 .. 1188278781 SIZE: 11370 jump: 0 ra: 12582912 vdr: [5627] READ: fd: 25 1188278781 .. 1188292400 SIZE: 13619 jump: 0 ra: 12582912 vdr: [5627] READ: fd: 25 1188292400 .. 1188335024 SIZE: 42624 jump: 0 ra: 12582912 vdr: [5627] WANT: fd: 25 1188335024 .. 1188589175 SIZE: 254151 vdr: [5627] READ: fd: 25 1188335024 .. 1188344461 SIZE: 9437 jump: 0 ra: 12582912 vdr: [5627] READ: fd: 25 1188344461 .. 1188365793 SIZE: 21332 jump: 0 ra: 12582912 vdr: [5627] READ: fd: 25 1188365793 .. 1188459990 SIZE: 94197 jump: 0 ra: 12582912 vdr: [5627] WANT: fd: 25 1188459990 .. 1188777569 SIZE: 317579 vdr: [5627] READ: fd: 25 1188459990 .. 1188473626 SIZE: 13636
Re: [vdr] Problems playing ongoing recordings?
alexw wrote:
I _think_ the issue was some kind of stuttering playback; it didn't
happen always, but often enough that one day i decided to see if writing
the stream in much larger chunks would help. It did; never seen it since.
I am using the following VDR setup: 2 VDR machines in
client(FF)/server(budget) mode (thank to streamdev) over wifi. The video
folder is shared over the network in CIFS (aka Samba). Actually replaying VDR
I have a similar setup, just with 100M ethernet instead of wifi and NFS instead
of samba. wifi could be a problem, but if you're able to watch the same channel
live you should also be able to view a recording.
recording is not smooth all the time and the timeshifting on the client side
is not working at all.
Timeshifting on the client requires 3x the bandwith and a lot of data going in
both directions, so may not work all that well over wireless.
Recording on the server and playing on the client at
the same time result in smooth and freeze cycle with lip sync issue. Watching
live programs is perfect.
But this sounds very much like the problem i was seeing. In my case it was
dependent on the bitrate and number of simultaneous recordings IIRC.
If you'd like to try a patch, attached is what i've using since May 2007.
It could help, but might also not change anything at all; the main reason
i'm interested if it helps is because i have O_DIRECT support done on top
of this, and am not sure if i should keep the write coalescing for the case
where O_DIRECT isn't available.
It contains:
- various code cleanups (that i made after Klaus merged the fadvise-related
modifications, they make the patch a bit harder to read, but should be
otherwise harmless)
- tweaks to improve playback (smarter readahead)
- tweaks to improve cutting performance (didn't help much iirc)
- write coalescing - the stream is written in multi-megabyte chunks instead
of a dozen or so K at a time. This is the interesting part; it adds
another data copy, but seems to have made the stuttering playback disappear.
(this change will only make a difference when applied to the vdr that does
the recording, ie the server; the fadvise tweaks mentioned above could also
improve things when applied to the client)
Patch made against vdr-1.4.7, but applies to 1.6.0 too.
artur
diff --git a/cutter.c b/cutter.c
index 5170ae4..7e2e506 100644
--- a/cutter.c
+++ b/cutter.c
@@ -66,7 +66,8 @@ void cCuttingThread::Action(void)
toFile = toFileName-Open();
if (!fromFile || !toFile)
return;
- fromFile-SetReadAhead(MEGABYTE(20));
+ fromFile-CuttingSrc();
+ toFile-CuttingDst();
int Index = Mark-position;
Mark = fromMarks.Next(Mark);
int FileSize = 0;
@@ -91,7 +92,7 @@ void cCuttingThread::Action(void)
if (fromIndex-Get(Index++, FileNumber, FileOffset, PictureType,
Length)) {
if (FileNumber != CurrentFileNumber) {
fromFile = fromFileName-SetOffset(FileNumber, FileOffset);
- fromFile-SetReadAhead(MEGABYTE(20));
+ fromFile-CuttingSrc();
CurrentFileNumber = FileNumber;
}
if (fromFile) {
@@ -124,6 +125,7 @@ void cCuttingThread::Action(void)
error = toFile 1;
break;
}
+ toFile-CuttingDst();
FileSize = 0;
}
LastIFrame = 0;
@@ -164,6 +166,7 @@ void cCuttingThread::Action(void)
error = toFile 2;
break;
}
+toFile-CuttingDst();
FileSize = 0;
}
}
diff --git a/tools.c b/tools.c
index 255c806..a14f799 100644
--- a/tools.c
+++ b/tools.c
@@ -1055,7 +1055,8 @@ bool cSafeFile::Close(void)
#define USE_FADVISE
-#define WRITE_BUFFER KILOBYTE(800)
+//#define dfsyslog dsyslog // uncomment to turn on fadvise
related logging
+#define dfsyslog(a...) do {} while (0)
cUnbufferedFile::cUnbufferedFile(void)
{
@@ -1073,12 +1074,19 @@ int cUnbufferedFile::Open(const char *FileName, int
Flags, mode_t Mode)
fd = open(FileName, Flags, Mode);
curpos = 0;
#ifdef USE_FADVISE
- begin = lastpos = ahead = 0;
- cachedstart = 0;
- cachedend = 0;
- readahead = KILOBYTE(128);
+ lastpos = 0;
+ lastjump = 0;
+ cachedstart = cachedend = 0;
+ // 4M readahead seems to work for playback while cutting,
+ // but isn't quite enough for FF while cutting...
+ readahead = MEGABYTE(12);
+ writebuffer = KILOBYTE(2000);
written = 0;
totwritten = 0;
+ cutting = 0;
+
+ wbuf = NULL;
+
if (fd = 0)
posix_fadvise(fd, 0, 0, POSIX_FADV_RANDOM); // we could use
POSIX_FADV_SEQUENTIAL, but we do our own readahead, disabling the kernel one.
#endif
@@ -1087,6 +1095,11 @@ int cUnbufferedFile::Open(const char *FileName, int
Flags, mode_t Mode)
int
Re: [vdr] Problems playing ongoing recordings?
VDR User wrote: On Thu, Mar 27, 2008 at 9:19 AM, Artur Skawina [EMAIL PROTECTED] wrote: I have a similar setup, just with 100M ethernet instead of wifi and NFS instead of samba. wifi could be a problem, but if you're able to watch the same channel live you should also be able to view a recording. Takes a lot more bandwidth to record playback then just to record so the fact that live tv is fine doesn't amount to much I don't think. I was referring to playing a finished recording and playing a file that is currently being extended by the server vdr -- alexw said that doesn't work well for him. It should, unless the disk and/or fs can't handle the two data streams concurrently, while keeping the latency low enough. I'm assuming the vdr server in powerful enough to handle the load, yes. artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
[vdr] [PATCH] Direct I/O support v1
This change makes vdr write the video stream directly to disk, bypassing
OS caches. O_DIRECT is used if it is supported, falling back to normal
I/O and fadvise() if not. Only recording is affected, playback remains
unchanged.
I did this to see if using direct i/o made sense for vdr, wasn't convinced
before. Avoiding all the fadvise hacks and especially the fdatasyncs on
close seems to be worth it though.
Only lightly tested, but seems to work, i tried a few recordings on both
ext3 and NFS and didn't notice any missing data yet ;) Upgraded all vdrs
and will keep testing for some time, but in case anyone else would like
to try (or even better improve) it, patch is below.
This patch is based on top of the one I posted in the Problems playing
ongoing recordings? thread, so you'll need to apply that one before
applying this one. Should work on both 1.4 and 1.6.
artur
diff --git a/tools.c b/tools.c
index a14f799..b7281b0 100644
--- a/tools.c
+++ b/tools.c
@@ -1054,6 +1054,11 @@ bool cSafeFile::Close(void)
// --- cUnbufferedFile ---
#define USE_FADVISE
+#define USE_DIRECTIO
+
+// O_DIRECT can have various alignment restrictions, usually at most
+// the block size of the filesystem. 4096 bytes should be enough.
+#define ALIGN_DIO 4096
//#define dfsyslog dsyslog // uncomment to turn on fadvise
related logging
#define dfsyslog(a...) do {} while (0)
@@ -1071,7 +1076,17 @@ cUnbufferedFile::~cUnbufferedFile()
int cUnbufferedFile::Open(const char *FileName, int Flags, mode_t Mode)
{
Close();
- fd = open(FileName, Flags, Mode);
+#ifdef USE_DIRECTIO
+ if (Flags(O_WRONLY|O_RDWR)) {
+ fd = open(FileName, Flags|O_DIRECT, Mode);
+ directio = 1;
+ }
+#endif
+ if (fd==-1) {
+ directio = 0;
+ fd = open(FileName, Flags, Mode);
+ }
+ dsyslog(Using %s IO to access %s, directio?DIRECT:normal, FileName);
curpos = 0;
#ifdef USE_FADVISE
lastpos = 0;
@@ -1101,7 +1116,7 @@ int cUnbufferedFile::Close(void)
free(wbuf);
}
#ifdef USE_FADVISE
- if (fd = 0) {
+ if (fd = 0 || !directio) {
if (totwritten)// if we wrote anything make sure the data has hit the
disk before
fdatasync(fd); // calling fadvise, as this is our last chance to
un-cache it.
posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED);
@@ -1243,7 +1258,36 @@ ssize_t cUnbufferedFile::Read(void *Data, size_t Size)
ssize_t cUnbufferedFile::WriteBuf(const void *Data, size_t Size)
{
if (fd =0) {
- ssize_t bytesWritten = safe_write(fd, Data, Size);
+ ssize_t bytesWritten;
+
+#ifdef USE_DIRECTIO
+ if (directio) {
+// write properly sized buffers directly.
+if ((Size (ALIGN_DIO-1)) == 0) {
+ bytesWritten = safe_write(fd, Data, Size);
+ curpos += bytesWritten;
+ return bytesWritten;
+}
+// in the unlikely case of a short write (inevitable when closing)
+// pad the data with zeros, write it, then truncate the file.
+int padding = ALIGN_DIO - (Size (ALIGN_DIO-1));
+memset((char *)Data+Size, 0, padding);
+bytesWritten = safe_write(fd, Data, Size+padding);
+
+padding = bytesWritten - Size;
+if (padding0)
+ padding = 0;
+bytesWritten -= padding;
+curpos += bytesWritten;
+
+lseek(fd, -padding, SEEK_CUR);
+ftruncate(fd, curpos);
+// note: past this point the file offset is likely unaligned
+// so further directio shouldn't happen.
+return min(bytesWritten, (ssize_t)Size);
+ }
+#endif
+ bytesWritten = safe_write(fd, Data, Size);
//dsyslog(WRIT: fd:%3d %9zd .. %9zd SIZE: %6zd, fd, curpos,
curpos+Size, Size);
#ifdef USE_FADVISE
if (bytesWritten 0) {
@@ -1296,18 +1340,23 @@ ssize_t cUnbufferedFile::Write(const void *Data, size_t
Size)
{
if (!wbuf) {
wbuf_chunk = cutting?MEGABYTE(8):MEGABYTE(4);
- wbuf = MALLOC(uchar,wbuf_chunk);
- if (!wbuf)
+ if (posix_memalign(wbuf, ALIGN_DIO, wbuf_chunk)) {
+directio = 0;
return WriteBuf(Data, Size);
+ }
wbuf_len = 0;
}
if (Size = wbuf_chunk-wbuf_len) {
memcpy(wbuf+wbuf_len, Data, Size);
wbuf_len += Size;
} else {
- WriteBuf(wbuf, wbuf_len);
- memcpy(wbuf, Data, Size);
- wbuf_len = Size;
+ unsigned l = wbuf_chunk-wbuf_len;
+ if (l)
+memcpy(wbuf+wbuf_len, Data, l);
+ WriteBuf(wbuf, wbuf_chunk);
+
+ memcpy(wbuf, (char *)Data+l, Size-l);
+ wbuf_len = Size-l;
}
return Size;
}
diff --git a/tools.h b/tools.h
index ce7283c..c2ff493 100644
--- a/tools.h
+++ b/tools.h
@@ -255,6 +255,7 @@ private:
size_t written;
size_t totwritten;
int cutting;
+ int directio;
size_t writebuffer;
int FadviseDrop(off_t Offset, off_t Len);
int FadviseRead(off_t Offset, off_t Len);
___
vdr mailing list
vdr@linuxtv.org
[vdr] Problems playing ongoing recordings?
Is anyone seeing some kind of problem when playing back files, but only when the recording has not yet finished? Possibly just when replaying a file that is currently being extended, maybe by another VDR and/or over NFS. [i had a problem like that many months ago, attempted to fix it, but it seems i was too successful ;) - the problem went away, but i didn't verify that the change actually helped. By now i don't remember exactly what the symptoms were, and what the vdr setup looked like at the time; hence i can't easily reproduce it. I'm planning on basing another feature on top of the fix and wondering how to handle that...] artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] Problems playing ongoing recordings?
VDR User wrote: On Wed, Mar 26, 2008 at 10:27 AM, Artur Skawina [EMAIL PROTECTED] wrote: Is anyone seeing some kind of problem when playing back files, but only when the recording has not yet finished? Possibly just when replaying a file that is currently being extended, maybe by another VDR and/or over NFS. [i had a problem like that many months ago, attempted to fix it, but it seems i was too successful ;) - the problem went away, but i didn't verify that the change actually helped. By now i don't remember exactly what the symptoms were, and what the vdr setup looked like at the time; hence i can't easily reproduce it. I'm planning on basing another feature on top of the fix and wondering how to handle that...] It would help if you mentioned what the problem actually was so people actually know what they're supposed to be looking for. I've played recordings back before they're finished many times and never noticed any problems. *shrug* kind of hard to describe something you no longer remember... ;) I _think_ the issue was some kind of stuttering playback; it didn't happen always, but often enough that one day i decided to see if writing the stream in much larger chunks would help. It did; never seen it since. artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] astra und hotbird ueber diseqc schalter?
Oliver Joa wrote: Hi, ich habe mir heute einen zweiten LNB auf die Schuessel gebaut um Hotbird empfangen zu koennen. Jetzt weiss ich nur noch nicht wie ich beide LNB am geschicktesten mit meinem VDR verbinde (Beide LNB jeweils Quad). Ich habe 2 DVB Karten. Die professionelle Variante waere wohl einen Multiswitch einzubauen. Dagegen spricht der Preis und dass er extra the easiest way would be two 21 diseqc switches - no vdr patches required, just the usual diseqc config. artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] change make install to be really useful
Udo Richter wrote: Matthias Schwarzott wrote: makefile-destdir.diff: It add variable DESTDIR to makefile to install under $(DESTDIR) instead of / I would prefer something like DESTDIR=/usr/local that can be redirected to /usr or /tmp/newpackage/usr or similar, just as many configure scripts do it. On the downside, this wouldn't work for $(DESTDIR)$(VIDEODIR). the usual convention is PREFIX, which defaults to '/usr/local' and can be changed to eg '/usr/' or '/sw/vdr-1.6.0' etc. DESTDIR is completely independent -- it adds _another_ prefix; this is used eg when building packages. So make install DESTDIR=/tmp/BLD-38746 PREFIX=/sw/vdr-1.6.0 means to assume the program will be installed in /sw/vdr-1.6.0 (and look for config files in /sw/vdr-1.6.0/etc/, binaries in /sw/vdr-1.6.0/bin/ etc), but to copy all the files to /tmp/BLD-38746/sw/vdr-1.6.0. This way you can build a package w/o disturbing the /sw/ tree and install using a package manager later. ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
[vdr] [PATCH] softdevice segfaulting on stream errors
Some time ago the softdevice plugin started crashing when playing video streams
containing errors, both while live viewing and watching recordings.
Today I found a transponder where it reliably occurred within seconds and
decided
to investigate. The picture buffers used for video were 8-byte aligned, which
happened to work for normal mpeg decoding, but wasn't enough. ffmpeg was
executing SSE instructions when doing error concealment and those were
segfaulting.
Fix below.
artur
diff -uwp softdevice.noalign/PicBuffer.c softdevice/PicBuffer.c
--- softdevice.noalign/PicBuffer.c 2007-03-13 02:57:19.0 +0100
+++ softdevice/PicBuffer.c 2007-07-10 15:25:38.0 +0200
@@ -316,9 +316,8 @@ bool AllocatePicBuffer(sPicBuffer *buf,P
if ( !isPlanar(pix_fmt) ) {
buf-stride[0]=ALIGN(pixel_size*w,16);
-buf-pixel[0]=(uint8_t*)malloc((buf-stride[0]*h)+16);
-if (buf-pixel[0]==NULL) {
+if (posix_memalign(buf-pixel[0], 16, buf-stride[0]*h+16)) {
printf(could not allocate memory for picture buffer!\n) ;
exit(-1);
return false;
@@ -337,9 +336,7 @@ bool AllocatePicBuffer(sPicBuffer *buf,P
buf-stride[i]= ALIGN(pixel_size*wh_shift,
STRIDE_ALIGN(h_chroma_shift-h_shift));
-buf-pixel[i]= (uint8_t*)malloc((buf-stride[i]*hv_shift)+16);
//FIXME 16
-
-if(buf-pixel[i]==NULL) {
+if(posix_memalign(buf-pixel[i], 16,
(buf-stride[i]*hv_shift)+16)) { //FIXME 16
printf(could not allocate memory for picture buffer!\n) ;
exit(-1);
return false;
___
vdr mailing list
vdr@linuxtv.org
http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] [PATCH] Auto sized ringbuffers v3
VDR User wrote: I've recorded/watched a lot of tv in the last couple days and so far I haven't noticed any problems with this patch. I am currently using v3 (the patch I attached to my previous post) with vdr-1.5.2. Do you know how large the buffers usually grow? If you haven't reduced vdrs log level (-l) this info should be somewhere in /var/log/ and you can take a look at the buffer usage w/ eg: $ grep vdr /var/log/* | grep 'ring buffer' | less Did any overflows happen (you can search for 'overflow' in the output of above commands to find them)? I've now checked my vdr logs here (starting ~ a week ago) and not a single buffer overflowed. I would have expected a few problems, but it looks like the tuning worked. (However that vdr has all data moving threads running at real time priority and this may have helped too.) Thanks, artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] [PATCH] Auto sized ringbuffers v3
VDR User wrote: On 5/20/07, Artur Skawina [EMAIL PROTECTED] wrote: VDR User wrote: I've recorded/watched a lot of tv in the last couple days and so far I haven't noticed any problems with this patch. I am currently using v3 (the patch I attached to my previous post) with vdr-1.5.2. Do you know how large the buffers usually grow? If you haven't reduced vdrs log level (-l) this info should be somewhere in /var/log/ and you can take a look at the buffer usage w/ eg: I use level 3 logging for vdr. I emailed you a copy of my logs so you should see that by the time you read this. Thanks. It seems that 648k was the largest size ever used for a buffer, which is less than 1/3 of the full size (2M for Result,Transfer and TS). The Recorder buffer was typically at 5% of its max size, ie 300k instead of 5M. No overflows w/ v3 of the patch (two with the previous version, which was starting out with just 128k). I get a lot worse results here (buffers grow to a few M), but that's probably because the vdr box is not a dedicated one, but also handles quite a few other services; IO stalls are not uncommon and several simultaneous recordings are sometimes done while streaming via streamdev to another vdr. Still, no overflows. artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] [PATCH] Auto sized ringbuffers v3
Jaroslaw Swierczynski wrote: It seems your patch is against VDR 1.5.x. Could you please backport it to VDR 1.4.x? actually it's vs 1.4.6, but applies to 1.5.2 too. ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] [PATCH] dynamically sized ringbuffers v2
Klaus Schmidinger wrote: If this auto sized ringbuffers change (which, from what I can see so far - haven't tried it myself - looks like a good idea) is ever to make its way into the official VDR source, you'll need to get rid of the above waiting. It says in receiver.h: ...the call must return as soon as possible, without any unnecessary delay. Maybe I should change this to ...the call must return immediately. or must not sleep. I did mention in the patch description, which you snipped, that this was just a debugging tool. IOW that change can be dropped entirely, the condition that makes it sleep has never triggered since I made the buffers grow faster anyway. I'd be great if somebody with a FF card could test the patch, the code has worked flawlessly on my vdr server, not a single overflow since v2. However on the streamdev/softdevice client I had two transfer buffer overflows (153 bytes each) just after switching to a new channel, while both machines were under significant (io) load, cutting aot. I don't think this is reason enough to introduce a minsize rb parameter, but if it also happens w/ a simple FF setup, that would be an option. I don't want to raise the default minimum, because 128k is often enough and i'd like to avoid shrinking, to keep things simple. The patch is for 1.4.6, but applies also, with a few harmless offsets, to 1.5.2. artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] [PATCH] dynamically sized ringbuffers v2
Stone wrote: It still wouldn't surprise me if this version caused a few overflows, but hopefully these will be very rare. I'm curious how streamdev will function with these buffer changes. it works fine -- i'm using a headless vdr server and streamdev+softdevice clients, so this actually gets tested fairly well. Example buffer usage from server: Deleting ring buffer streamdev-streamer size: 262144 / 4194304 (used 6.25% of requested size) Deleting ring buffer streamdev-streamer size: 131072 / 4194304 (used 3.125% of requested size) Deleting ring buffer streamdev-streamer size: 262144 / 4194304 (used 6.25% of requested size) Deleting ring buffer streamdev-streamer size: 131072 / 4194304 (used 3.125% of requested size) Deleting ring buffer streamdev-streamer size: 131072 / 4194304 (used 3.125% of requested size) Deleting ring buffer streamdev-streamer size: 524288 / 4194304 (used 12.5% of requested size) Deleting ring buffer streamdev-streamer size: 262144 / 4194304 (used 6.25% of requested size) Deleting ring buffer streamdev-streamer size: 262144 / 4194304 (used 6.25% of requested size) Deleting ring buffer streamdev-streamer size: 262144 / 4194304 (used 6.25% of requested size) Deleting ring buffer streamdev-streamer size: 131072 / 4194304 (used 3.125% of requested size) Deleting ring buffer streamdev-streamer size: 262144 / 4194304 (used 6.25% of requested size) Deleting ring buffer streamdev-streamer size: 131072 / 4194304 (used 3.125% of requested size) Deleting ring buffer streamdev-streamer size: 262144 / 4194304 (used 6.25% of requested size) Deleting ring buffer streamdev-streamer size: 262144 / 4194304 (used 6.25% of requested size) Deleting ring buffer streamdev-streamer size: 262144 / 4194304 (used 6.25% of requested size) Deleting ring buffer streamdev-streamer size: 524288 / 4194304 (used 12.5% of requested size) Deleting ring buffer streamdev-streamer size: 524288 / 4194304 (used 12.5% of requested size) Deleting ring buffer streamdev-streamer size: 131072 / 4194304 (used 3.125% of requested size) Deleting ring buffer streamdev-streamer size: 262144 / 4194304 (used 6.25% of requested size) Deleting ring buffer streamdev-streamer size: 131072 / 4194304 (used 3.125% of requested size) Deleting ring buffer streamdev-streamer size: 131072 / 4194304 (used 3.125% of requested size) Deleting ring buffer streamdev-streamer size: 131072 / 4194304 (used 3.125% of requested size) Deleting ring buffer streamdev-streamer size: 131072 / 4194304 (used 3.125% of requested size) and the streamdev client TS buffer usually grows to 12.5..25% (ie 256K/512K out of 4M max). artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] [PATCH] dynamically sized ringbuffers v2
Jouni Karvo wrote: Stone writes: It still wouldn't surprise me if this version caused a few overflows, but hopefully these will be very rare. I'm curious how streamdev will function with these buffer changes. And since I am not convinced that this memory footprint issue is significant, I am concerned if this patch is accepted before all the overflows and problems with live viewing have been solved first. It shouldn't. The whole point of posting it is so that other are able to test and review it. That is why I always mention the overflows. Right now it's stable enough for production for me, haven't seen any overflow since adjusting the growth rate, but it would be nice if somebody with a different vdr setup tested it too (eg a FF card based vdr). It will take weeks to verify that the code works as expected (the goal is to have _fewer_ overflows (compared to the static buffers) while at the same time a much smaller working set). artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] [PATCH] dynamically sized ringbuffers v1
void cRecorder::Receive(uchar *Data, int Length)
{
if (Running()) {
int p = ringBuffer-Put(Data, Length);
if (p != Length Running())
ringBuffer-ReportOverflow(Length - p);
}
}
it simply drops any data that does not fit into the buffer, which would be
fine
for live viewing, but isn't ideal when recording.
Can we try harder not to loose data here? IOW can this function sleep (and
retry)?
It took a while to trigger the condition again, but now it happened and trying
a bit
harder payed off. Running vdr /w following patch resulted in this log (and no
overflow):
20:31:35 vdr: [16328] buffer usage: 70% (tid=16327)
20:31:35 vdr: [16328] buffer usage: 80% (tid=16327)
20:31:35 vdr: [16328] buffer usage: 90% (tid=16327)
20:31:35 vdr: [16328] buffer usage: 100% (tid=16327)
20:31:35 vdr: [16327] Enlarging ring buffer Result: 262144 bytes (trigger 3)
20:31:35 vdr: [16329] Enlarging ring buffer TS: 262144 bytes (trigger 2)
20:31:35 vdr: [16328] Enlarging ring buffer Recorder: 262144 bytes (trigger 3)
20:31:35 vdr: [16328] buffer usage: 0% (tid=16327)
20:31:35 vdr: [16328] saved extra 153 bytes in recorder ring buffer after 80 ms
delay
artur
diff --git a/recorder.c b/recorder.c
index 8bb1621..3c0e002 100644
--- a/recorder.c
+++ b/recorder.c
@@ -157,8 +157,20 @@ void cRecorder::Receive(uchar *Data, int Length)
{
if (Running()) {
int p = ringBuffer-Put(Data, Length);
- if (p != Length Running())
+ if (p != Length Running()) {
+for (int ms=20; ms1000; ms+=ms) {
+ cCondWait::SleepMs(ms);
+ if (!Running())
+ return;
+ int r = ringBuffer-Put(Data+p, Length-p);
+ p += r;
+ if (r)
+ dsyslog(saved extra %d bytes in recorder ring buffer after %d
ms delay, r, ms);
+ if (p == Length || !Running())
+ return;
+ }
ringBuffer-ReportOverflow(Length - p);
+}
}
}
___
vdr mailing list
vdr@linuxtv.org
http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] [PATCH] dynamically sized ringbuffers v1
Patch below changes the interpretation of the 'Size' parm given when creating
a buffer from a
fixed length to a max limit. The buffers start out much smaller (currently
128k, instead of many
megabytes) and grow automatically when they become almost full. This way all
callers benefit,
and they don't have to be modified at all.
I've tested this on two vdrs for a few hours and it seems to work, but it
needs a lot
more testing.
I did a few test recordings (total duration likely more than 24h) and the new
code
seems mostly fine; just some tweaking of the sizes and grow heuristics left.
However one problem did appear after many hours of simultaneous recordings:
08:17:38 vdr: [24052] buffer usage: 70% (tid=24051)
08:17:39 vdr: [24052] buffer usage: 80% (tid=24051)
08:17:39 vdr: [24052] buffer usage: 90% (tid=24051)
08:17:39 vdr: [24052] buffer usage: 100% (tid=24051)
08:17:39 vdr: [24052] ERROR: 1 ring buffer overflow (61 bytes dropped)
08:17:45 vdr: [24052] ERROR: 9053 ring buffer overflows (1701964 bytes dropped)
08:17:47 vdr: [24052] Enlarging ring buffer Recorder: 442368 bytes
08:17:47 vdr: [24052] buffer usage: 0% (tid=24051)
Apparently the 'Recorder' consumer stalled for several seconds at a point when
growing the ringbuffer wasn't possible - to keep Get/Put lockless resizing is
only possible when the ringbuffer data is continuous; and if the consumer stops
making progress at the wrong moment, the buffer will not be enlarged and
overflow.
In this case the buffer was ~300k and ~2m would have been needed.
I can think of a few ways to make the ringbuffer growing more aggressive, but
they all make the buffer handling much more complex, so i'd like to avoid that.
Looking at the caller:
void cRecorder::Receive(uchar *Data, int Length)
{
if (Running()) {
int p = ringBuffer-Put(Data, Length);
if (p != Length Running())
ringBuffer-ReportOverflow(Length - p);
}
}
it simply drops any data that does not fit into the buffer, which would be fine
for live viewing, but isn't ideal when recording.
Can we try harder not to loose data here? IOW can this function sleep (and
retry)?
(not that i think doing this would be enough; it could help for short stalls,
but
for the longer ones a much larger buffer size is necessary, i'll have to find a
clean way to handle that anyway. It's kind of hard to test though as the
overflow
only happened once during ~24h of recording...)
artur
___
vdr mailing list
vdr@linuxtv.org
http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
[vdr] [PATCH] dynamically sized ringbuffers v1
vdr uses quite a lot of memory, RSS here used to be ~70M, so i decided to see
if it could be
reduced a bit. First suspects became the ringbuffers, cause i was seeing a lot
of log lines
such as buffer stats: 96068 (1%) used. Turns out many buffers are way over
sized, only a few
percent of the memory are typically used, yet because of how a ringbuffer works
_all_ of it is
constantly accessed. I first tried to make the ringbuffer users smarter about
picking the size
and eg after changing the transfer buffer to start out small and use bigger
sizes only when the
current one isn't large enough it seemed a more reasonable size for that vdr
instance was 384k
instead of 2M. Similarly the TS buffer could be just a few hundred K instead of
2M.
But that approach was not ideal as there isn't anything one could do after
realizing the buffer
is too small, it could be grown next time, after channel switch etc, but that's
it. Also some
of the huge buffers came from plugins, and I wasn't all that excited about
fixing all of them
(one example w/ the one-digit % fill was streamdev-server).
It is hard to estimate what the proper size would be under load, with many
simultaneous recordings.
So I did attack this issue differently -- by making the ringbuffers themselves
smarter.
Patch below changes the interpretation of the 'Size' parm given when creating a
buffer from a
fixed length to a max limit. The buffers start out much smaller (currently
128k, instead of many
megabytes) and grow automatically when they become almost full. This way all
callers benefit,
and they don't have to be modified at all.
The buffers are actually always allocated using the max size, but only the
necessary
buffer part is used, the rest of the memory is never accessed; this is done
this way
so that no extra locking is necessary.
I've tested this on two vdrs for a few hours and it seems to work, but it needs
a lot
more testing.
There is some extra logging in this patch so that you can see what's going on;
every
time a ring buffer gets deleted it logs a line which tells you exactly how much
memory
was requested and how much was really needed.
vdr: [19729] Deleting ring buffer TS size: 196608 / 2097152 (used 9.375% of
requested size)
vdr: [19718] Deleting ring buffer Result size: 262144 / 262144 (used 100% of
requested size)
vdr: [19718] Deleting ring buffer Recorder size: 442368 / 5242880 (used
8.4375% of requested size)
vdr: [19718] Deleting ring buffer Result size: 262144 / 262144 (used 100% of
requested size)
vdr: [19718] Deleting ring buffer Recorder size: 131072 / 5242880 (used 2.5%
of requested size)
vdr: [19733] Deleting ring buffer TS size: 131072 / 2097152 (used 6.25% of
requested size)
vdr: [19718] Deleting ring buffer Result size: 262144 / 262144 (used 100% of
requested size)
vdr: [19718] Deleting ring buffer Recorder size: 442368 / 5242880 (used
8.4375% of requested size)
So each of the several 'Recorder' buffers was an order of magnitude smaller
than w/o
this patch and several megabytes less ram was used.
artur
diff --git a/ringbuffer.c b/ringbuffer.c
index 0633bd3..8dd1efe 100644
--- a/ringbuffer.c
+++ b/ringbuffer.c
@@ -151,13 +151,29 @@ void cRingBufferLinear::PrintDebugRBL(void)
}
#endif
+// cRingBufferLinear are dynamically sized, or at least we can pretend they
are ;)
+// We treat 'Size' as the maximum size, but start with a small buffer, which
can
+// grow later as it fills up. Memory is always allocated for the full buffer,
but
+// the unused RAM portion remains untouched until (if at all) it is actually
needed.
+// Note that we can not start with a larger than requested buffer because
there are
+// ring buffer users that cause crashes then (eg softdevice mpegdecoder
absolutely
+// needs 32/64k).
+
+// Startup size. 64k still causes overflows before buffer starts to grow, 128k
doesn't.
+// The buffers grow to 200..500k anyway, so maybe increasing this a bit more
would
+// make sense, but let's first see if it's really needed.
+#define DEFRBSIZE KILOBYTE(128)
+
cRingBufferLinear::cRingBufferLinear(int Size, int Margin, bool Statistics,
const char *Description)
-:cRingBuffer(Size, Statistics)
+:cRingBuffer(min(Size,DEFRBSIZE), Statistics)
{
description = Description ? strdup(Description) : NULL;
tail = head = margin = Margin;
gotten = 0;
buffer = NULL;
+ maxsize = Size;
+ growbuf = 0;
+ dsyslog(New ring buffer \%s\ size: %d margin: %d, description, Size,
Margin );
if (Size 1) { // 'Size - 1' must not be 0!
if (Margin = Size / 2) {
buffer = MALLOC(uchar, Size);
@@ -183,6 +199,8 @@ cRingBufferLinear::~cRingBufferLinear()
#ifdef DEBUGRINGBUFFERS
DelDebugRBL(this);
#endif
+ dsyslog(Deleting ring buffer \%s\ size: %d / %d (used %g%% of requested
size), description,
+ size, maxsize,
size/(double)maxsize*100.0 );
free(buffer);
free(description);
}
@@ -207,6 +225,13 @@ void
[vdr] remote plugin ignoring some keys
A while ago the remote plugins telnet server started acting strangely here;
it would
sometimes ignore all cursor and function keys, yet other keys would work
perfectly.
I'm not sure when exactly this began, most likely a newer kernel triggered this.
It does not happen always, but when it does it makes navigating the OSD
practically
impossible.
Turns out the remote plugin gets confused when a read() call returns not a full
escape sequence, but just the first \0x27 byte and the next read() receives the
remaining two or three bytes. And this is exactly what happens (the client for
some
reason actually sends two tcp packets containing just eg \0x1b and
\0x5b\0x43).
This quick patch made the remote osd usable again; it isn't perfect as keys can
still
be lost (see comments), but it's a huge improvement - only some key repeats are
dropped,
not all multibyte sequences.
artur
diff -urNp remote-0.3.9.org/remote.c remote-0.3.9/remote.c
--- remote-0.3.9.org/remote.c 2006-12-03 14:46:55.0 +
+++ remote-0.3.9/remote.c 2007-04-29 20:45:13.0 +
@@ -517,6 +517,27 @@ uint64_t cRemoteDevTty::getKey(void)
uint64_t code = 0;
n = read(fh, code, sizeof code);
+
+if (code==27 n==1)
+{
+ // Ugh. What happened: user pressed a key that generates an escape
+ // sequence (eg up/down) and we got only the first '\e'; now we have
+ // to fetch the remaining bytes as otherwise we end up ignoring almost
+ // every such key press and navigating the osd becomes impossible.
+ // We try to read more bytes and hope that the result makes sense;
+ // this often works, but not always. When eg holding down a key we
+ // can receive several (repeated) sequences and these are then ignored.
+ // Dropping a few key presses (mostly repeats) is still much better
+ // than dropping nearly _all_ of them...
+ // (Unfortunately the sequences do differ in length, hence either
+ // parsing them or matching them w/ the expected ones would be needed
+ // to avoid loosing sync here)
+ int m = read(fh, ((char*)code)+1, sizeof code-1);
+ if (m 0)
+ n += m;
+}
+//dsyslog(cRemoteDevTty::getKey: %d %lld, n, code);
+
return (n 0) ? code : INVALID_KEY;
}
___
vdr mailing list
vdr@linuxtv.org
http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] [PATCH] limit streamdev client's ringbuffer timeouts to more sane values
Stone wrote: On 4/5/07, *Artur Skawina* [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: With the high-res timers in kernel 2.6.21+ usleep(1) is no longer treated as usleep(1) and the streamdev client is almost unusable; it uses most of the cpu and causes hundreds of thousands context switches per second. This gets rid of the almost-busy-loop. How about his approach? --- streamdev/client/filter.c.orig 2007-04-05 20:45:04.0 -0700 +++ streamdev/client/filter.c 2007-04-05 20:45: 44.0 -0700 @@ -135,7 +135,7 @@ } m_RingBuffer-Del(TS_SIZE); } else - usleep(1); + cCondWait::SleepMs(1); } } sure, just changing it to 'usleep(1)' works too. Is there a reason to avoid the ringbuffer infrastructure? artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] [PATCH] limit streamdev client's ringbuffer timeouts to more sane values
Stone wrote: sure, just changing it to 'usleep(1)' works too. Is there a reason to avoid the ringbuffer infrastructure? No reason in particular, I just wanted to test which one worked best with performance. Thanks for the patch. :) I tried the 1us - (1000|1)us sleep approach first, before using the ringbuffer timeouts -- saw no noticeable difference wrt performance. 10ms timeouts seemed to be enough (it's the resolution of a HZ==100 kernel) and gave similar interrupt and cs numbers as w/ low-res timers. artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
[vdr] [PATCH] limit streamdev client's ringbuffer timeouts to more sane values
Well, the streamdev-client reads data from a ringbuffer and when there isn't
anything
to read it tries to sleep for 1us and loops. This wasn't a problem when the
timer
resolution was in the 1000..1us range (1000..100Hz); the usleep(1) call
slept
for one or more milliseconds.
With the high-res timers in kernel 2.6.21+ usleep(1) is no longer treated as
usleep(1) and the streamdev client is almost unusable; it uses most of the
cpu
and causes hundreds of thousands context switches per second.
This gets rid of the almost-busy-loop.
artur
diff -urNp streamdev.org/client/filter.c streamdev/client/filter.c
--- streamdev.org/client/filter.c 2005-11-06 17:43:58.0 +0100
+++ streamdev/client/filter.c 2007-04-05 23:25:11.0 +0200
@@ -57,6 +57,7 @@ cStreamdevFilters::cStreamdevFilters(voi
cThread(streamdev-client: sections assembler) {
m_Active = false;
m_RingBuffer = new cRingBufferLinear(MEGABYTE(1), TS_SIZE * 2, true);
+ m_RingBuffer-SetTimeouts(10, 10);
Start();
}
@@ -111,8 +112,7 @@ void cStreamdevFilters::Action(void) {
}
}
m_RingBuffer-Del(TS_SIZE);
- } else
- usleep(1);
+ }
}
}
___
vdr mailing list
vdr@linuxtv.org
http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] Too many open files - error
Kartsa wrote:
I was about to test the performance of vdr when I stumbled on this message
ERROR: /dev/dvb/adapter0/demux0: Too many open files
I do not recall seeing this earlier. This came when fourth simultaneous
recording started.
Is this a vdr, dvb, or firmware issue? Seems like dvb but I really do
not know.
This is what was in the log after the fourth recording
Feb 19 21:53:00 vdr: [2226] timer 4 (7 2153-2200 'TestRec4') start
Feb 19 21:53:00 vdr: [2226] record
/srv/vdr/TestRec4/2007-02-19.21.53.50.99.rec
Feb 19 21:53:00 vdr: [2226] ERROR: /dev/dvb/adapter0/demux0: Too many
open files
Feb 19 21:53:00 vdr: [2226] ERROR (dvbdevice.c,673): Too many open files
Feb 19 21:53:00 vdr: [2226] ERROR: can't set PID 680 on device 1
Feb 19 21:53:00 vdr: [2226] ERROR (dvbdevice.c,688): Bad file descriptor
grep -20 -r EMFILE *
dvb/dvb-core/dmxdev.c-static int dvb_demux_open(struct inode *inode, struct
file *file)
dvb/dvb-core/dmxdev.c-{
dvb/dvb-core/dmxdev.c- struct dvb_device *dvbdev = file-private_data;
dvb/dvb-core/dmxdev.c- struct dmxdev *dmxdev = dvbdev-priv;
dvb/dvb-core/dmxdev.c- int i;
dvb/dvb-core/dmxdev.c- struct dmxdev_filter *dmxdevfilter;
dvb/dvb-core/dmxdev.c-
dvb/dvb-core/dmxdev.c- if (!dmxdev-filter)
dvb/dvb-core/dmxdev.c- return -EINVAL;
dvb/dvb-core/dmxdev.c-
dvb/dvb-core/dmxdev.c- if (mutex_lock_interruptible(dmxdev-mutex))
dvb/dvb-core/dmxdev.c- return -ERESTARTSYS;
dvb/dvb-core/dmxdev.c-
dvb/dvb-core/dmxdev.c- for (i = 0; i dmxdev-filternum; i++)
dvb/dvb-core/dmxdev.c- if (dmxdev-filter[i].state ==
DMXDEV_STATE_FREE)
dvb/dvb-core/dmxdev.c- break;
dvb/dvb-core/dmxdev.c-
dvb/dvb-core/dmxdev.c- if (i == dmxdev-filternum) {
dvb/dvb-core/dmxdev.c- mutex_unlock(dmxdev-mutex);
dvb/dvb-core/dmxdev.c: return -EMFILE;
dvb/dvb-core/dmxdev.c- }
dvb/dvb-core/dmxdev.c-
dvb/dvb-core/dmxdev.c- dmxdevfilter = dmxdev-filter[i];
dvb/dvb-core/dmxdev.c- mutex_init(dmxdevfilter-mutex);
dvb/dvb-core/dmxdev.c- file-private_data = dmxdevfilter;
dvb/dvb-core/dmxdev.c-
dvb/dvb-core/dmxdev.c- dvb_ringbuffer_init(dmxdevfilter-buffer, NULL, 8192);
dvb/dvb-core/dmxdev.c- dmxdevfilter-type = DMXDEV_TYPE_NONE;
dvb/dvb-core/dmxdev.c- dvb_dmxdev_filter_state_set(dmxdevfilter,
DMXDEV_STATE_ALLOCATED);
dvb/dvb-core/dmxdev.c- dmxdevfilter-feed.ts = NULL;
dvb/dvb-core/dmxdev.c- init_timer(dmxdevfilter-timer);
dvb/dvb-core/dmxdev.c-
dvb/dvb-core/dmxdev.c- mutex_unlock(dmxdev-mutex);
dvb/dvb-core/dmxdev.c- return 0;
dvb/dvb-core/dmxdev.c-}
IOW you ran out of filters.
yes, the error code should probably be different (eg EBUSY).
___
vdr mailing list
vdr@linuxtv.org
http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
Re: [vdr] Is there a way to give inode entries a higher cache priority than data in the linux kernel?
Carsten Koch wrote: I have a distributed VDR system in my house with a lot of disks that are NFS mounted by VDR PCs in two rooms. In order to conserve energy, I have used hdparm to set a spin down delay after which the disks turn themselves off. When /video/.update is touched (one of the VDR PCs creates/deletes/ edits a recording, I move a recording into a folder, etc.) or when the vdr program is started, it reads all directories from all disks. Most of these directories are unchanged, so there really is no need to spin up the disk just to read a few inode entries. However, my observation is that they are always spun up. So my questions are: 1) Is there a bug in the linux kernel that makes it spin up the disk needlessly even if the data are still in the cache? 2) Is there a way to configure the kernel, so the inode entries are locked in the cache or at least get a much higher cache priority than ordinary data? The problem is that the nfs server reads (and writes, when editing a recording on a client) large amounts of data and the metadata (which hasn't been accessed since the last /video scan) gets evicted from the cache. This usually does not happen w/ a local vdr and on the client vdrs because fadvice() makes sure that the cached video data gets freed before the system experiences any memory pressure. But fadvise() only helps locally, ie the nfs client drops the data, but the nfs server does not. I've gotten used to this; at one point, when doing the fadvise changes, i was going to implement O_DIRECT-based i/o, but decided it wasn't worth it, because of the complexity and small gains (you can't really avoid a data copy w/ the current vdr design (alignment restrictions) and the code still has to handle all the cases where O_DIRECT and/or AIO isn't available, older kernels, other filesystems etc). What you could try is: o play with /proc/sys/vm/vfs_cache_pressure; try decreasing it. significantly. what you describe as 2) above would be vfs_cache_pressure=0. Haven't tried this myself; does it help? If not the other options would probably be a bit more complex (like modifying nfsd or writing a dedicated fuse fs). artur ___ vdr mailing list vdr@linuxtv.org http://www.linuxtv.org/cgi-bin/mailman/listinfo/vdr
