Send Motion-user mailing list submissions to motion-user@lists.sourceforge.net
To subscribe or unsubscribe via the World Wide Web, visit https://lists.sourceforge.net/lists/listinfo/motion-user or, via email, send a message with subject or body 'help' to motion-user-requ...@lists.sourceforge.net You can reach the person managing the list at motion-user-ow...@lists.sourceforge.net When replying, please edit your Subject line so it is more specific than "Re: Contents of Motion-user digest..." Today's Topics: 1. Re: motion crashing or stops working (S Andreason) 2. Re: motion crashing or stops working (rmbusy) 3. Re: motion crashing or stops working (S Andreason) ---------------------------------------------------------------------- Message: 1 Date: Thu, 25 Feb 2021 10:11:36 -0800 From: S Andreason <sandrea...@gmail.com> To: Motion-user <motion-user@lists.sourceforge.net> Subject: Re: [Motion-user] motion crashing or stops working Message-ID: <c8259eec-d868-3ba3-d2b5-2b095c15e...@gmail.com> Content-Type: text/plain; charset=UTF-8; format=flowed Hi, I have results to report from testing the git version. Feb.24 booted running Motion 4.3.1+git20210125-34091a7 issue #1. After 28 minutes of running, it started having events with corrupted frames. ?73 from 206_E and 20 from 204_W were my total for yesterday. ?I tried restarting motion, which worked for a while, then the problem returned after 30 minutes. ?I tried swapping hubs and cables, but no change. ?Only the Alptop cameras #204 and 206 are having this happen. The same cameras involved with crashing running Motion 4.3.2. I think there is a connection here. ?The symptoms are either a grey frame (identical to when I press forward x seconds in mpv), or a rainbow smear of corrupted changes, P-frames or B-frames. Either way, something was lost. ?On average this was happening once every 2 minutes, in that first hour before I started restarting and moving cables. 4, 5, 7 times per hour later in the day, throughout the afternoon. issue #2. on_event gets triggered 5, 6 or up to 8 seconds after the event, thereby making useless the on_event trigger, as by the time motion notifies me of the event, it has already concluded. The trigger to start recording hi-res (outside motion) is thereby late, and now misses the event. ?The time stamp by motion is showing up to 8 seconds difference from the camera time stamp. The drift of the camera's clock is no more than 2 seconds, corrected by ntp. ?Note that 4.3.2 did not have this problem. In 4.3.2, all cameras are fast and at the head of the stream. Hi-res would start recording (on different computer so as to not slow down the pi) in 700ms. That is on_event_start sends UDP packet across network, different computer receives and starts ffmpeg, 700ms is good. ?in 4.3.1+git it becomes clear motion is not staying at the head of the stream, but is several seconds behind, is no longer live. ?If I Stop motion, move the camera, and Start motion, in 4.3.1+git it is out of sync, when I start motion, it is seeing the camera move, and so starts a false event. ?I've noted the Hisee and Hikvision cameras do not have this problem, but they do perform the same as 4.3.2. So the problem is only with Alptop cameras. Model# AP-IPC-500PE Feb25 repeat of Feb24, confirmed the cause is not bad cable or bad hub losing packets. 29 corrupted events in first hour after the 28 minute mark. I stopped motion, changed back to 4.3.2 and started service motion, and problem goes away. ?Now back to the occasional crash is better than the above trouble. Theory: Those occasional crashes in 4.3.2 are at the same time motion is trying to jump to the head of the live stream, and fails? ?That would happen at startup, thus the high rate of crash at start and restart. Does the developer have access to the Alptop camera for testing? or can I do anything for you here? log file, config files again: https://www.seahorsecorral.org/videos/tests/pi1_20210225_motion.zip Looking at motion.log when log_level=9 I see a few things of interest: 1. Lots of lines containing: ?mlp_setupmode: Raw changes: ? is at a rate of 45 times per second for 3 cameras. Really eats up flash storage fast. ? Changing to log level 1 does not stop the Raw lines. ? Is this part of debug symbols or part of the git package for testing? 2. The apparent trigger for a corrupted frame is captured in the logfile motion-20210225-1git9_line1775.log, see line 1775 where it says: ?? ffmpeg_avcodec_log: Could not find ref with POC 18 matches up with an event video, link in notes. One of the hundred event videos to demonstrate this problem: https://www.seahorsecorral.org/videos/tests/20210224-151253_DE20_corruptedPacket-rainbowSmear.mp4 Thank you, Stewart Admin Account wrote: > Try installing from github or the motion website. ------------------------------ Message: 2 Date: Thu, 25 Feb 2021 10:51:46 -0800 From: rmbusy <rmbusy+mot...@gmail.com> To: motion-user@lists.sourceforge.net Subject: Re: [Motion-user] motion crashing or stops working Message-ID: <c90e1653-1ab0-246f-9f9e-6a3ccbada...@gmail.com> Content-Type: text/plain; charset=utf-8; format=flowed Out of curiosity, what does "top" show for CPU usage with both versions?? Whenever I get grey screens, or corrupted video from streaming cameras, it has a direct correlation to CPU usage maxing out.? Typically I'll need to reboot the cameras once they get into that state. On my system (quad core, ARM based, 400% CPU max), top shows the CPU usage is around 175% to 200% w/o a browser connected.? If I connect to the built in web page, it maxes out around 375%.? If I open a second page from another browser, then cameras start failing. -- Rob. On 2/25/21 10:11 AM, S Andreason wrote: > Hi, > I have results to report from testing the git version. > Feb.24 booted running Motion 4.3.1+git20210125-34091a7 > > issue #1. After 28 minutes of running, it started having events with > corrupted frames. > ?73 from 206_E and 20 from 204_W were my total for yesterday. > ?I tried restarting motion, which worked for a while, then the problem > returned after 30 minutes. > ?I tried swapping hubs and cables, but no change. > > ?Only the Alptop cameras #204 and 206 are having this happen. The same > cameras involved with crashing running Motion 4.3.2. I think there is > a connection here. > ?The symptoms are either a grey frame (identical to when I press > forward x seconds in mpv), or a rainbow smear of corrupted changes, > P-frames or B-frames. Either way, something was lost. > ?On average this was happening once every 2 minutes, in that first > hour before I started restarting and moving cables. 4, 5, 7 times per > hour later in the day, throughout the afternoon. > > issue #2. on_event gets triggered 5, 6 or up to 8 seconds after the > event, thereby making useless the on_event trigger, as by the time > motion notifies me of the event, it has already concluded. The trigger > to start recording hi-res (outside motion) is thereby late, and now > misses the event. > ?The time stamp by motion is showing up to 8 seconds difference from > the camera time stamp. The drift of the camera's clock is no more than > 2 seconds, corrected by ntp. > > ?Note that 4.3.2 did not have this problem. In 4.3.2, all cameras are > fast and at the head of the stream. Hi-res would start recording (on > different computer so as to not slow down the pi) in 700ms. That is > on_event_start sends UDP packet across network, different computer > receives and starts ffmpeg, 700ms is good. > > ?in 4.3.1+git it becomes clear motion is not staying at the head of > the stream, but is several seconds behind, is no longer live. > ?If I Stop motion, move the camera, and Start motion, in 4.3.1+git it > is out of sync, when I start motion, it is seeing the camera move, and > so starts a false event. > ?I've noted the Hisee and Hikvision cameras do not have this problem, > but they do perform the same as 4.3.2. So the problem is only with > Alptop cameras. Model# AP-IPC-500PE > > Feb25 > repeat of Feb24, confirmed the cause is not bad cable or bad hub > losing packets. 29 corrupted events in first hour after the 28 minute > mark. > > I stopped motion, changed back to 4.3.2 and started service motion, > and problem goes away. > ?Now back to the occasional crash is better than the above trouble. > > Theory: Those occasional crashes in 4.3.2 are at the same time motion > is trying to jump to the head of the live stream, and fails? > ?That would happen at startup, thus the high rate of crash at start > and restart. > > Does the developer have access to the Alptop camera for testing? or > can I do anything for you here? > > log file, config files again: > https://www.seahorsecorral.org/videos/tests/pi1_20210225_motion.zip > > Looking at motion.log when log_level=9 I see a few things of interest: > 1. Lots of lines containing: > ?mlp_setupmode: Raw changes: > ? is at a rate of 45 times per second for 3 cameras. Really eats up > flash storage fast. > ? Changing to log level 1 does not stop the Raw lines. > ? Is this part of debug symbols or part of the git package for testing? > > 2. The apparent trigger for a corrupted frame is captured in the > logfile motion-20210225-1git9_line1775.log, see line 1775 where it says: > ?? ffmpeg_avcodec_log: Could not find ref with POC 18 > matches up with an event video, link in notes. > > One of the hundred event videos to demonstrate this problem: > https://www.seahorsecorral.org/videos/tests/20210224-151253_DE20_corruptedPacket-rainbowSmear.mp4 > > > > Thank you, > Stewart > > > Admin Account wrote: >> Try installing from github or the motion website. > > > > _______________________________________________ > Motion-user mailing list > Motion-user@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/motion-user > https://motion-project.github.io/ > > Unsubscribe: https://lists.sourceforge.net/lists/options/motion-user ------------------------------ Message: 3 Date: Thu, 25 Feb 2021 12:19:32 -0800 From: S Andreason <sandrea...@gmail.com> To: Motion-user <motion-user@lists.sourceforge.net> Subject: Re: [Motion-user] motion crashing or stops working Message-ID: <1395b362-5e52-36df-d684-f89ae557f...@gmail.com> Content-Type: text/plain; charset=UTF-8; format=flowed Good questions, I tried both versions, on both pi's, of course the one watching 3 cameras at 640x480 has higher cpu% than 1 camera at 640x480. Considering slightly different amount of movement outside, but very close to none, I don't really see a big change between versions (second and third columns of 6). max / min top% over a minute: pi (cams)??? 4.3.1git ? ? ?? 4.3.2 ?? ? ?? ? 4.3.2+1web 4.3.2+2web??? 1 event pi1 (3) 5??? 70.5 / 54.2??? 74.1 / 57.5??? 67.8 / 58.1?????? 67.9 / 57.3? ??? 137.5 / pi3 (1) 9??? - / -??? ??? ?????? 16.2 / 13.9??? 18.9 / 14.9 pi3 (1) 5??? 16.6 / 11.9 If I intentionally move 3 cameras at same time to max it out, pi1 (3) 5??? 3 events = 300.3% / __ So the above normal usage is well under 400% (4 cores) I thought I would see a big change with log_level at 9 but it did not do the 45 lines per second dump into motion.log this time. I don't understand what triggered that this morning... Since the majority of my grey frames (or rainbow smudges) were at low movement levels, and only one camera having an event, I can infer they had low cpu load. But yes I can see how high cpu load would would cause problems. Although when I was testing to determine how many cams a pi could manage, I found it drops frame rate first, so a 15 fps camera drops to 12 or 13 fps in the output video. But 640x480 it handles ok. High res 2560x1440 it can't handle even one stream fully at 10 fps. It can manage 6 fps. I opened 2 pages in 2 tabs, and then 2 different browsers, and did not see any change, I saw maybe 2% difference, but not consistently. However the reloading jpeg image for me does not load, so I think I'd need to figure out how to get past that block to make it work harder. More directly, If I load from my browser http://192.168.5.121:8081/stream I get error: The connection was refused, and Page load error. This has not bothered me since I use mpv to watch streams live. I don't have any firewall settings to block it, so maybe I should ask how to get that working across the local network? Can the stream be delivered to an address other than 127.0.0.1? rmbusy wrote: > Out of curiosity, what does "top" show for CPU usage with both > versions? Whenever I get grey screens, or corrupted video from > streaming cameras, it has a direct correlation to CPU usage maxing > out. Typically I'll need to reboot the cameras once they get into that > state. > > On my system (quad core, ARM based, 400% CPU max), top shows the CPU > usage is around 175% to 200% w/o a browser connected.? If I connect to > the built in web page, it maxes out around 375%.? If I open a second > page from another browser, then cameras start failing. > > ------------------------------ ------------------------------ Subject: Digest Footer _______________________________________________ Motion-user mailing list Motion-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/motion-user ------------------------------ End of Motion-user Digest, Vol 176, Issue 36 ********************************************