Re: [Freeswitch-users] Choppy sound with PCMU
Here is what I found... I tried high-priority scheduling as per your suggestion, reniced the program explicitly, rewrote timer thread to sleep on cond. variable and activate only when there are timers and only when the timer actually had to be clicked, turned off SQL thread and removed polling from sofia profile thread. That pretty much eliminated all idle 1ms sleepers that were there except for three in sofia itself (su_epoll_port). And when I was about to be happy, I found that two outgoing calls through my VOIP providers when bridged together showed terrible distortions. I undid all my changes, tried 1.0.4, trunk (noticed btw that when I bridge two calls via loopback in JS in the trunk I must keep JS running, or the calls get terminated - NOT the same as in 1.0.4 where exitting JS left calls running), got pretty much the same sad results. At the same time calls bridged by freeswitch between LAN and any of the VOIP providers behaved just fine. And calls bridged by Asterisk any way were fine too. So that pretty much looked like the end of the freeswitch trials for me. But then I timed your code, mine and found that all those 1ms sleeps that your timer thread was doing (and all those pollers were doing as well) were actually 4ms sleeps because you know what unless kernel is configured with HZ=1000, you can't sleep for less than 4ms (HZ=250) or perhaps even 10ms (HZ=100). Mine was 250. This actually meant that the original timer thread was firing once, sleeping for 4ms, firing 3 more times back-to-back, sleeping for 4ms more, firing 4 times back-to-back, etc. It was still firing 20ms timers on time, but 30ms ones of course were not, since 30ms doesn't divide by 4 evenly. Plus whoever relied on runtime.reference or switch_micro_time_now() were kind of screwed because both were running jumpy. Plus whoever assumed that apr_sleep(1000) or cond_yield() was sleeping for 1ms were also in for a surprise. It felt satisfying to find that, however it didn't explain why the same distortions were observed with rewritten timer thread and disabled RTP timers. Anyway, I sighed (pretty much like you) and recompiled the kernel with HZ=1000. Recompiling kernel on these ALIX boards is fun. If smth goes south, you need to hook up serial console and see what the heck went wrong. That eliminated distortions, ha! But made freeswitch more CPU hungry. Now the remaining 1ms threads sitting in sofia epoll were really polling for 1ms, not 4, and freeswitch was consistently sitting in the first line of the top chart showing 3% CPU utilization when idle. Don't know whether it's because of the remaining epolls in sofia or whether it's because there are still some threads left in freeswitch that I neglected to change because they were sleeping with 100ms interval, so I figured, who cares. Maybe when all things come together (sofia, 100ms*N) freeswitch ends up spending 3% of CPU while doing pretty much nothing. Btw, compared with Asterisk, the latter is not even visible on the first top's screen and spends 1% CPU when bridging two G711 calls and recording them to disk. So, at this time I have both original Asterisk and FS setups running. One is seemless but clumsy in configuration, the other one is neat and stylish but too preoccupied with smth... Should I look into sofia epollers? That's kind of deep in the code. Or should I just stick with Asterisk? Anthony Minessale-2 wrote: There is another user here with a 300mhz box. I am willing to investigate this improved performance for weak devices but I need to do it in a sane cross-platform way. On Fri, Dec 4, 2009 at 1:32 PM, Yossi Neiman freeswi...@cartissolutions.com wrote: A word to the wise to the general FreeSWITCH community: If Anthony Minessale suggests that you try to do any number of things, it's a very good idea to try all those ideas before continuing on. I've known him, MikeJ, and bkw for several years, and they almost always have very good ideas as to troubleshoot a problem in FreeSWITCH. It's extremely frustrating to try to help people out who won't try the provided suggestions first. And note directly to eaf - bogomips is quite possibly the least significant bit of data about a cpu that you will get out of /proc/cpuinfo... The name itself - bogo, means bogus. http://en.wikipedia.org/wiki/Bogomips -Yossi ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- Anthony Minessale II FreeSWITCH http://www.freeswitch.org/ ClueCon http://www.cluecon.com/ Twitter: http://twitter.com/FreeSWITCH_wire AIM: anthm MSN:anthony_miness...@hotmail.com msn%3aanthony_miness...@hotmail.com GTALK/JABBER/PAYPAL:anthony.miness...@gmail.compaypal%3aanthony.miness...@gmail.com IRC: irc.freenode.net
Re: [Freeswitch-users] Choppy sound with PCMU
What do you want me to check while running these tests? Sound quality (it's good now even with original 1.0.4). Or CPU utilization? It's Debian 4. Anthony Minessale-2 wrote: Did you do each thing alone too to tell the difference? -hp alone, disable monotonic alone (i did not see you mention the disable monotonic) as for your 4ms thing, yes we require high resolution timing, if we ask to sleep 1000 microseconds that is what we need it to sleep for or at least as close as possible, and the main reason that thread is never sleeping is because you can't actually count on it to run every 1ms but you mostly can. Hence the whole philosophy on only making 1 thread run hot all the time to ensure that the rest don't have to repeat the same algorithm. We focus on high end performance this was the point of your experimentation because we will need to use a compile time defines and other logic to make it more efficient on your platform, a platform which we are not using. I am curious what would happen if you install Kristian's astlinux on one of your devices, i think you should also compare the kernel versions. What OS are you running anyway? Here are some more things to try (running plain trunk with no mods) do these systematically each alone and all together with/without -hp or disable monotonic etc to see what different combos create comment out this line (line 10) #define DISABLE_1MS_COND rebuild, this tells it to run a conditional at 1ms in the same timer thread which will make all the switch_cond_next share a 1ms conditional instead of doing microsleeps next some kernels/devices work better using select(0) for sleep where others work better using usleep. comment out line 109 apr_sleep(t); and try usleep(t) also mac works better using nanosleep so you could try changing it so it uses the code starting at 101 instead. also your claim about JS should be investigated because I do not think it should be the case. but you may want to move this to a jira http://jira.freeswitch.org As for the asterisk comparison, not sure how to answer you, that's your decision. On Mon, Dec 7, 2009 at 9:28 AM, eaf erandr-j...@usa.net wrote: Here is what I found... I tried high-priority scheduling as per your suggestion, reniced the program explicitly, rewrote timer thread to sleep on cond. variable and activate only when there are timers and only when the timer actually had to be clicked, turned off SQL thread and removed polling from sofia profile thread. That pretty much eliminated all idle 1ms sleepers that were there except for three in sofia itself (su_epoll_port). And when I was about to be happy, I found that two outgoing calls through my VOIP providers when bridged together showed terrible distortions. I undid all my changes, tried 1.0.4, trunk (noticed btw that when I bridge two calls via loopback in JS in the trunk I must keep JS running, or the calls get terminated - NOT the same as in 1.0.4 where exitting JS left calls running), got pretty much the same sad results. At the same time calls bridged by freeswitch between LAN and any of the VOIP providers behaved just fine. And calls bridged by Asterisk any way were fine too. So that pretty much looked like the end of the freeswitch trials for me. But then I timed your code, mine and found that all those 1ms sleeps that your timer thread was doing (and all those pollers were doing as well) were actually 4ms sleeps because you know what unless kernel is configured with HZ=1000, you can't sleep for less than 4ms (HZ=250) or perhaps even 10ms (HZ=100). Mine was 250. This actually meant that the original timer thread was firing once, sleeping for 4ms, firing 3 more times back-to-back, sleeping for 4ms more, firing 4 times back-to-back, etc. It was still firing 20ms timers on time, but 30ms ones of course were not, since 30ms doesn't divide by 4 evenly. Plus whoever relied on runtime.reference or switch_micro_time_now() were kind of screwed because both were running jumpy. Plus whoever assumed that apr_sleep(1000) or cond_yield() was sleeping for 1ms were also in for a surprise. It felt satisfying to find that, however it didn't explain why the same distortions were observed with rewritten timer thread and disabled RTP timers. Anyway, I sighed (pretty much like you) and recompiled the kernel with HZ=1000. Recompiling kernel on these ALIX boards is fun. If smth goes south, you need to hook up serial console and see what the heck went wrong. That eliminated distortions, ha! But made freeswitch more CPU hungry. Now the remaining 1ms threads sitting in sofia epoll were really polling for 1ms, not 4, and freeswitch was consistently sitting in the first line of the top chart showing 3% CPU utilization when idle. Don't know whether it's because of the remaining epolls in sofia or whether it's because there are still some threads left
Re: [Freeswitch-users] Choppy sound with PCMU
Oh, it's not just one timer thread... Why, why is sql_thread keeps on checking for messages every millisecond? Couldn't there be some signalling implemented that will make the thread suspend on condition variable or a socket/pipe in between? #0 do_sleep (t=1000) at src/switch_time.c:109 #1 0xb7e5e215 in switch_core_sql_thread (thread=0xb7586ae8, obj=0x0) at src/switch_core_sqldb.c:783 Why does this sofia_profile_worker_thread keeps on looping checking for the queue? Have a semaphore! #0 do_sleep (t=1000) at src/switch_time.c:109 #1 0xb73a4701 in sofia_profile_worker_thread_run (thread=0x80f3a30, obj=0x80f2490) at sofia.c:978 Nothing's happening on the box, but there are three threads that pretend to be actively busy with smth. Others at least sleep for hundreds of milliseconds, not for one. And there is even infrastructure present to do blocking pops: i.e. why couldn't sqldb thread do queue_pop() instead of queue_trypop() intermixed with 1ms sleeps? This looping is such a waste... eaf wrote: As I see it, switch_cond_next() currently is just a do_sleep(1000). Yes, it could be mapped to a 1ms timer, but #define DISABLE_1MS_COND overrides that. Yeah, there is a global timestamp... It's easy to workaround that for RTP who calls switch_micro_time_now()... But if somebody accesses runtime.timestamp directly, it's gonna be tough to grep for that. If only this was C++... I'll play around. Never liked polling too much. Never could've guessed that polling could be so useful for scalability ;) My naive implementation would've pulled timestamp via system calls and would've done sleeping by passing exact interval to select() instead of syncing with a pacing thread. Which would be dead-quiet at idle time, but, of course, would stop scaling at some point due to excessive number of system calls. Thanks. Michael Jerris wrote: In short. No, you can not for many reasons. The milisecond tic is used throughout the code even when there is not any calls up. You can grep for switch_cond_next if you would like to see where but it is required to keep our global timestamp and for pacing the scheduler among other services that run all the time. Mike On Dec 2, 2009, at 7:31 PM, eaf erandr-j...@usa.net wrote: Can I reduce resolution of that timer thread 10 times? I mean, I glanced through the code, and see that among others (are there others?) RTP and IVR set up their timers that are subsequently managed by this thread. RTP timers should be eliminated by that setting you've suggested. IVR timers are set at 20ms... So, if the thread is set to wake up every 10ms instead of 1ms it should be able to wake up those IVR timers just fine. Right? That's a cool design to have one dedicated thread that maintains accurate timing and then broadcasts via condition variables to hundreds of other threads events that they can register for. I'm sure it's one of the reasons why FS scales so much better than Asterisk. But for poor low-end setups that sit in the closet, eat only 6W of power and hardly ever run more than two calls at the same time, can I hack it somehow to be more UNIX- friendly? I.e. make it stuck in select() or recv() when there is nothing to do, call clock_gettime() right from the thread that wants and when it wants to know current time? Say, what if that thread is made to suspend on a condition variable in case if there are no timers registered in TIMER_MATRIX? Then, if some other thread comes up and adds its timer into the matrix, it could wake up the timer thread and enjoy accurate timing as needed, on demand? And in- between the calls, when there is no RTP or IVR, it will all go silent? I mean, sitting on a wait queue in the kernel is way better than go back and forth incrementing counters that nobody even needs at the moment? Anthony Minessale-2 wrote: idle is a 4 letter word to a realtime application. The core keeps a single high-priority thread to keep 1ms timing and expands that broadcasting to hundreds or thousand of threads who need accurate timing. Your choppy audio is caused by linksys lying about the packet len that it's using and we set our timer to the wrong speed. -- View this message in context: http://old.nabble.com/Choppy-sound-with-PCMU-tp26594250p26619085.html Sent from the Freeswitch-users mailing list archive at Nabble.com. ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch- users http://www.freeswitch.org ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch
Re: [Freeswitch-users] Choppy sound with PCMU
Btw, I have these popping up in my logs from time to time: 2009-12-03 09:42:06.035294 [DEBUG] switch_core_state_machine.c:314 (sofia/external/xx...@4.68.250.148) Running State Change CS_HANGUP 2009-12-03 09:42:06.035294 [CRIT] switch_time.c:473 Virtual Migration Detected! Syncing Clock In this case an incoming call rang to both FS and Asterisk, Asterisk picked up, but the surge of activity made FS timer thread miss a beat or two. eaf wrote: Oh, it's not just one timer thread... Why, why is sql_thread keeps on checking for messages every millisecond? Couldn't there be some signalling implemented that will make the thread suspend on condition variable or a socket/pipe in between? #0 do_sleep (t=1000) at src/switch_time.c:109 #1 0xb7e5e215 in switch_core_sql_thread (thread=0xb7586ae8, obj=0x0) at src/switch_core_sqldb.c:783 Why does this sofia_profile_worker_thread keeps on looping checking for the queue? Have a semaphore! #0 do_sleep (t=1000) at src/switch_time.c:109 #1 0xb73a4701 in sofia_profile_worker_thread_run (thread=0x80f3a30, obj=0x80f2490) at sofia.c:978 Nothing's happening on the box, but there are three threads that pretend to be actively busy with smth. Others at least sleep for hundreds of milliseconds, not for one. And there is even infrastructure present to do blocking pops: i.e. why couldn't sqldb thread do queue_pop() instead of queue_trypop() intermixed with 1ms sleeps? This looping is such a waste... eaf wrote: As I see it, switch_cond_next() currently is just a do_sleep(1000). Yes, it could be mapped to a 1ms timer, but #define DISABLE_1MS_COND overrides that. Yeah, there is a global timestamp... It's easy to workaround that for RTP who calls switch_micro_time_now()... But if somebody accesses runtime.timestamp directly, it's gonna be tough to grep for that. If only this was C++... I'll play around. Never liked polling too much. Never could've guessed that polling could be so useful for scalability ;) My naive implementation would've pulled timestamp via system calls and would've done sleeping by passing exact interval to select() instead of syncing with a pacing thread. Which would be dead-quiet at idle time, but, of course, would stop scaling at some point due to excessive number of system calls. Thanks. Michael Jerris wrote: In short. No, you can not for many reasons. The milisecond tic is used throughout the code even when there is not any calls up. You can grep for switch_cond_next if you would like to see where but it is required to keep our global timestamp and for pacing the scheduler among other services that run all the time. Mike On Dec 2, 2009, at 7:31 PM, eaf erandr-j...@usa.net wrote: Can I reduce resolution of that timer thread 10 times? I mean, I glanced through the code, and see that among others (are there others?) RTP and IVR set up their timers that are subsequently managed by this thread. RTP timers should be eliminated by that setting you've suggested. IVR timers are set at 20ms... So, if the thread is set to wake up every 10ms instead of 1ms it should be able to wake up those IVR timers just fine. Right? That's a cool design to have one dedicated thread that maintains accurate timing and then broadcasts via condition variables to hundreds of other threads events that they can register for. I'm sure it's one of the reasons why FS scales so much better than Asterisk. But for poor low-end setups that sit in the closet, eat only 6W of power and hardly ever run more than two calls at the same time, can I hack it somehow to be more UNIX- friendly? I.e. make it stuck in select() or recv() when there is nothing to do, call clock_gettime() right from the thread that wants and when it wants to know current time? Say, what if that thread is made to suspend on a condition variable in case if there are no timers registered in TIMER_MATRIX? Then, if some other thread comes up and adds its timer into the matrix, it could wake up the timer thread and enjoy accurate timing as needed, on demand? And in- between the calls, when there is no RTP or IVR, it will all go silent? I mean, sitting on a wait queue in the kernel is way better than go back and forth incrementing counters that nobody even needs at the moment? Anthony Minessale-2 wrote: idle is a 4 letter word to a realtime application. The core keeps a single high-priority thread to keep 1ms timing and expands that broadcasting to hundreds or thousand of threads who need accurate timing. Your choppy audio is caused by linksys lying about the packet len that it's using and we set our timer to the wrong speed. -- View this message in context: http://old.nabble.com/Choppy-sound-with-PCMU-tp26594250p26619085.html Sent from the Freeswitch-users mailing list archive at Nabble.com. ___ FreeSWITCH
Re: [Freeswitch-users] Choppy sound with PCMU
I'm sorry if I sounded that way. Did mean to. :) Yes, it's an embedded platform. It's an ALIX board with AMD Geode LX800 chip and 256MB of RAM. http://www.pcengines.ch/alix2d3.htm Line offset difference is due to some minor logging changes I made to see who's allocating timers and how often. This way I found MOH streaming and that RTP still allocates timers even when it's set to none in the profile. I feel that this platform turned out to be underpowered for FS because it cannot meet its scheduling expectations. I guess, some degree of kernel tweaking or setting priorities will fix that. Meanwhile I just got rid of the SQLDB 1ms thread via -nosql command line option, split sofia worker 1ms thread in two (one blocked and waiting for new commands in the SQL queue, the other one checking registrations and gateways with 1sec interval), and don't know yet what to do about the timer thread. Again, I apologize for stupid or accusing questions, I'm just trying to see how FS can be made friendlier to this board. Or the board be made friendlier to FS ;) Anthony Minessale-2 wrote: If you see that message then your machine/os/combo is having some problems keeping up. It's not the timer missing anything its the monotonic clock detecting a 1 second or more differential from what its next prediction for the time should be. The best way to trigger this would be to suspend FS with control-z or attach to it with gdb blocking the entire process, that 1ms thread would have to miss 1000 iterations to trigger that warning. Btw, that error message is at line 471 not 473 so you are using modified code. Its possible your box has a bad monotonic timer, you can set under settings in switch.conf.xml We are now starting to guess you are using some small embedded type platform perhaps? I've run FS even on a nokia n810 and never caused that message to fire. if 1 call can interrupt the cpu enough to cause noticeable issues you might want to consider running the process at a greater priority by using the -hp command line arg or at least nice it Why don't you tell us the whole story about what OS/platform you are using here rather that form conjectures about what is wrong with our code that thousands of people are happy with. On Thu, Dec 3, 2009 at 8:55 AM, eaf erandr-j...@usa.net wrote: Btw, I have these popping up in my logs from time to time: 2009-12-03 09:42:06.035294 [DEBUG] switch_core_state_machine.c:314 (sofia/external/xx...@4.68.250.148) Running State Change CS_HANGUP 2009-12-03 09:42:06.035294 [CRIT] switch_time.c:473 Virtual Migration Detected! Syncing Clock In this case an incoming call rang to both FS and Asterisk, Asterisk picked up, but the surge of activity made FS timer thread miss a beat or two. eaf wrote: Oh, it's not just one timer thread... Why, why is sql_thread keeps on checking for messages every millisecond? Couldn't there be some signalling implemented that will make the thread suspend on condition variable or a socket/pipe in between? #0 do_sleep (t=1000) at src/switch_time.c:109 #1 0xb7e5e215 in switch_core_sql_thread (thread=0xb7586ae8, obj=0x0) at src/switch_core_sqldb.c:783 Why does this sofia_profile_worker_thread keeps on looping checking for the queue? Have a semaphore! #0 do_sleep (t=1000) at src/switch_time.c:109 #1 0xb73a4701 in sofia_profile_worker_thread_run (thread=0x80f3a30, obj=0x80f2490) at sofia.c:978 Nothing's happening on the box, but there are three threads that pretend to be actively busy with smth. Others at least sleep for hundreds of milliseconds, not for one. And there is even infrastructure present to do blocking pops: i.e. why couldn't sqldb thread do queue_pop() instead of queue_trypop() intermixed with 1ms sleeps? This looping is such a waste... eaf wrote: As I see it, switch_cond_next() currently is just a do_sleep(1000). Yes, it could be mapped to a 1ms timer, but #define DISABLE_1MS_COND overrides that. Yeah, there is a global timestamp... It's easy to workaround that for RTP who calls switch_micro_time_now()... But if somebody accesses runtime.timestamp directly, it's gonna be tough to grep for that. If only this was C++... I'll play around. Never liked polling too much. Never could've guessed that polling could be so useful for scalability ;) My naive implementation would've pulled timestamp via system calls and would've done sleeping by passing exact interval to select() instead of syncing with a pacing thread. Which would be dead-quiet at idle time, but, of course, would stop scaling at some point due to excessive number of system calls. Thanks. Michael Jerris wrote: In short. No, you can not for many reasons. The milisecond tic is used throughout the code even when there is not any calls up. You can grep for switch_cond_next if you would like to see where but it is required
Re: [Freeswitch-users] Choppy sound with PCMU
You mean, upgrading to the trunk and disabling RTP timers? Yes, I did. I thought I responded back. Perhaps it didn't make through though, as I just emailed back to the list instead of using nabble.com... Anyway, upgrading to the trunk didn't change much, forcing SPA to 30ms went w/o any effect either, but disabling RTP timers did the trick. I don't have the original choppy sound with PCMU problem any more, thanks a lot for the quick turnaround on that question. But your suggestions made me look, into logs, strace, code, etc, so now I'm just checking on how to quiet down those busy loops a little and how to get rid of periodic CRIT messages about Virtual Machine Migration. Anthony Minessale-2 wrote: What about the things I spent time suggesting in my last email? Did you try them because I was actually curious if they made any impact. On Thu, Dec 3, 2009 at 11:29 AM, eaf erandr-j...@usa.net wrote: I'm sorry if I sounded that way. Did mean to. :) Yes, it's an embedded platform. It's an ALIX board with AMD Geode LX800 chip and 256MB of RAM. http://www.pcengines.ch/alix2d3.htm Line offset difference is due to some minor logging changes I made to see who's allocating timers and how often. This way I found MOH streaming and that RTP still allocates timers even when it's set to none in the profile. I feel that this platform turned out to be underpowered for FS because it cannot meet its scheduling expectations. I guess, some degree of kernel tweaking or setting priorities will fix that. Meanwhile I just got rid of the SQLDB 1ms thread via -nosql command line option, split sofia worker 1ms thread in two (one blocked and waiting for new commands in the SQL queue, the other one checking registrations and gateways with 1sec interval), and don't know yet what to do about the timer thread. Again, I apologize for stupid or accusing questions, I'm just trying to see how FS can be made friendlier to this board. Or the board be made friendlier to FS ;) Anthony Minessale-2 wrote: If you see that message then your machine/os/combo is having some problems keeping up. It's not the timer missing anything its the monotonic clock detecting a 1 second or more differential from what its next prediction for the time should be. The best way to trigger this would be to suspend FS with control-z or attach to it with gdb blocking the entire process, that 1ms thread would have to miss 1000 iterations to trigger that warning. Btw, that error message is at line 471 not 473 so you are using modified code. Its possible your box has a bad monotonic timer, you can set under settings in switch.conf.xml We are now starting to guess you are using some small embedded type platform perhaps? I've run FS even on a nokia n810 and never caused that message to fire. if 1 call can interrupt the cpu enough to cause noticeable issues you might want to consider running the process at a greater priority by using the -hp command line arg or at least nice it Why don't you tell us the whole story about what OS/platform you are using here rather that form conjectures about what is wrong with our code that thousands of people are happy with. On Thu, Dec 3, 2009 at 8:55 AM, eaf erandr-j...@usa.net wrote: Btw, I have these popping up in my logs from time to time: 2009-12-03 09:42:06.035294 [DEBUG] switch_core_state_machine.c:314 (sofia/external/xx...@4.68.250.148) Running State Change CS_HANGUP 2009-12-03 09:42:06.035294 [CRIT] switch_time.c:473 Virtual Migration Detected! Syncing Clock In this case an incoming call rang to both FS and Asterisk, Asterisk picked up, but the surge of activity made FS timer thread miss a beat or two. eaf wrote: Oh, it's not just one timer thread... Why, why is sql_thread keeps on checking for messages every millisecond? Couldn't there be some signalling implemented that will make the thread suspend on condition variable or a socket/pipe in between? #0 do_sleep (t=1000) at src/switch_time.c:109 #1 0xb7e5e215 in switch_core_sql_thread (thread=0xb7586ae8, obj=0x0) at src/switch_core_sqldb.c:783 Why does this sofia_profile_worker_thread keeps on looping checking for the queue? Have a semaphore! #0 do_sleep (t=1000) at src/switch_time.c:109 #1 0xb73a4701 in sofia_profile_worker_thread_run (thread=0x80f3a30, obj=0x80f2490) at sofia.c:978 Nothing's happening on the box, but there are three threads that pretend to be actively busy with smth. Others at least sleep for hundreds of milliseconds, not for one. And there is even infrastructure present to do blocking pops: i.e. why couldn't sqldb thread do queue_pop() instead of queue_trypop() intermixed with 1ms sleeps? This looping is such a waste... eaf wrote: As I see it, switch_cond_next() currently is just a do_sleep(1000). Yes
Re: [Freeswitch-users] Choppy sound with PCMU
Oh, you mean giving FS higher priority? Yeah, as a last resort I'll do that. At the moment, I hope it won't be necessary as I can make those hyper threads behave, and will see how that goes first. I see where your implementation could be coming from. There is a queue of SQL queries in sofia.c processed by the worker thread. There are only two pop functions available in APR: queue_pop() and queue_trypop(), so alas no option with a timeout here. You don't want to block the thread in pop() indefinitely because you chose that same worker needs to do ireg and gw processing once in a while (separated by tens or hundreds of seconds, btw). You also want to be able to detect shutdown condition so that the worker doesn't hold up profile thread. So you chose to poll for events every millisecond instead of just creating an apr_thread_cond_t for resource friendly signalling. I agree that the timer thread philosophy is great and was the right choice for scaling, but I just don't comprehend responses to things like these other SQL or sofia worker threads. Did somebody even remotely acknowledge that busy loops at least in those areas that I showed may probably be a bad idea and could've been eliminated? I've heard suggestions to bump up priority, I've heard that the code was perfect already, that it's the result of 4-year effort, that I am arrogant, don't listen and don't understand squat. I'm sorry if I gave you impression that I was looking for the bad parts in the software. I apologized for that already. All I wanted was to have constructive conversation, perhaps I'm not too good at it. Code is already perfect according to you? Fine with me. Anthony Minessale-2 wrote: no, I mean the one after that that you must have completely skipped with a command line option to try and a param to set in the config. It somewhat annoys me for taking the time to compose it now. I wrote all of the code you are talking about myself and I was trying to give you some suggestions Well, actually, you did answer my question about the platform so you must have seen it. The loops are not the cause of that migration message, something wrong with the hardware or the kernel is. Another guy just told you he does not see that problem on the same exact hardware. Even if you have a point about the sql threads, you could make a patch to slow them down but you cant slow down too much or you will not be able to handle 400 cps all asking to send updates to transactions in batches of thousands of sql stmts. Every line of that code is carefully designed so I don't know what else to tell you but to stop being so arrogant and re-read this thread for all the advice you have totally ignored. I started out trying to help you but I have a lot of work to do. I thoroughly explained it to you and you are choosing to ignore me so I guess I'm done. You can do whatever you want with your working copy, i'll see you in 3 or 4 years when you get up to speed with the rest of us -- View this message in context: http://old.nabble.com/Choppy-sound-with-PCMU-tp26594250p26633739.html Sent from the Freeswitch-users mailing list archive at Nabble.com. ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
[Freeswitch-users] Best way to run originate calls through dial plan
What would be the best way of making originate() run call through a dial plan
(compared to directly going to a specified VOIP gateway). Would it be
loopbacks, i.e. smth like this?
/opt/freeswitch/bin/fs_cli -x originate
{ignore_early_media=true,origination_caller_id_number=xx}loopback/yy/default/XML
'javascript(/opt/freeswitch/conf/dialplan/public/webcall.js zz)'
The idea of this is that originate() sets up the first call, then webcall.js
plays back a WAV, and bridges the first call with the second one (also set
up via loopback).
Thanks!
--
View this message in context:
http://old.nabble.com/Best-way-to-run-originate-calls-through-dial-plan-tp26610094p26610094.html
Sent from the Freeswitch-users mailing list archive at Nabble.com.
___
FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org
Re: [Freeswitch-users] Best way to run originate calls through dial plan
I need a way to start a call from the PHP script to the originating number,
tell the party on that number to hold on, start another call to destination
number, and bridge everything together. On both legs I need to pass custom
caller ID. I can of course open direct connections to VOIP gateways right
from PHP, but I want to reuse existing routing rules in the dial plan, hence
I want to know what's the best way of making originate go through a specific
context of the dial plan.
As for the number of calls per second, it's going to be only occasionally
used.
mercutioviz wrote:
On Wed, Dec 2, 2009 at 6:47 AM, eaf erandr-j...@usa.net wrote:
What would be the best way of making originate() run call through a dial
plan
(compared to directly going to a specified VOIP gateway). Would it be
loopbacks, i.e. smth like this?
/opt/freeswitch/bin/fs_cli -x originate
{ignore_early_media=true,origination_caller_id_number=xx}loopback/yy/default/XML
'javascript(/opt/freeswitch/conf/dialplan/public/webcall.js
zz)'
The idea of this is that originate() sets up the first call, then
webcall.js
plays back a WAV, and bridges the first call with the second one (also
set
up via loopback).
Could you describe the problem that you're trying to solve? That would
make
it easier to know if what you've come up with is the best solution. How
many
calls per second were you wanting to generate with this setup?
-MC
___
FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org
--
View this message in context:
http://old.nabble.com/Best-way-to-run-originate-calls-through-dial-plan-tp26610094p26613841.html
Sent from the Freeswitch-users mailing list archive at Nabble.com.
___
FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org
Re: [Freeswitch-users] Choppy sound with PCMU
Can I reduce resolution of that timer thread 10 times? I mean, I glanced through the code, and see that among others (are there others?) RTP and IVR set up their timers that are subsequently managed by this thread. RTP timers should be eliminated by that setting you've suggested. IVR timers are set at 20ms... So, if the thread is set to wake up every 10ms instead of 1ms it should be able to wake up those IVR timers just fine. Right? That's a cool design to have one dedicated thread that maintains accurate timing and then broadcasts via condition variables to hundreds of other threads events that they can register for. I'm sure it's one of the reasons why FS scales so much better than Asterisk. But for poor low-end setups that sit in the closet, eat only 6W of power and hardly ever run more than two calls at the same time, can I hack it somehow to be more UNIX-friendly? I.e. make it stuck in select() or recv() when there is nothing to do, call clock_gettime() right from the thread that wants and when it wants to know current time? Say, what if that thread is made to suspend on a condition variable in case if there are no timers registered in TIMER_MATRIX? Then, if some other thread comes up and adds its timer into the matrix, it could wake up the timer thread and enjoy accurate timing as needed, on demand? And in-between the calls, when there is no RTP or IVR, it will all go silent? I mean, sitting on a wait queue in the kernel is way better than go back and forth incrementing counters that nobody even needs at the moment? Anthony Minessale-2 wrote: idle is a 4 letter word to a realtime application. The core keeps a single high-priority thread to keep 1ms timing and expands that broadcasting to hundreds or thousand of threads who need accurate timing. Your choppy audio is caused by linksys lying about the packet len that it's using and we set our timer to the wrong speed. -- View this message in context: http://old.nabble.com/Choppy-sound-with-PCMU-tp26594250p26619085.html Sent from the Freeswitch-users mailing list archive at Nabble.com. ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] Choppy sound with PCMU
Oh, looks like the timers are also used for streaming local data in read_stream_thread(). Due to this there is always one timer active with 20ms interval. But wait a sec, why is freeswitch periodically trying to stream /opt/freeswitch/sounds/music/8000/ponce-preludio-in-e-major.wav somewhere? Every minute or so? Did I misconfigure it? eaf wrote: Say, what if that thread is made to suspend on a condition variable in case if there are no timers registered in TIMER_MATRIX? Then, if some other thread comes up and adds its timer into the matrix, it could wake up the timer thread and enjoy accurate timing as needed, on demand? And in-between the calls, when there is no RTP or IVR, it will all go silent? I mean, sitting on a wait queue in the kernel is way better than go back and forth incrementing counters that nobody even needs at the moment? Anthony Minessale-2 wrote: idle is a 4 letter word to a realtime application. The core keeps a single high-priority thread to keep 1ms timing and expands that broadcasting to hundreds or thousand of threads who need accurate timing. Your choppy audio is caused by linksys lying about the packet len that it's using and we set our timer to the wrong speed. -- View this message in context: http://old.nabble.com/Choppy-sound-with-PCMU-tp26594250p26620518.html Sent from the Freeswitch-users mailing list archive at Nabble.com. ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] Choppy sound with PCMU
OK, I'm slow. It's music-on-hold, and it's playing non-stop like that timer thread. Even when there are no calls. Why? eaf wrote: Oh, looks like the timers are also used for streaming local data in read_stream_thread(). Due to this there is always one timer active with 20ms interval. But wait a sec, why is freeswitch periodically trying to stream /opt/freeswitch/sounds/music/8000/ponce-preludio-in-e-major.wav somewhere? Every minute or so? Did I misconfigure it? eaf wrote: Say, what if that thread is made to suspend on a condition variable in case if there are no timers registered in TIMER_MATRIX? Then, if some other thread comes up and adds its timer into the matrix, it could wake up the timer thread and enjoy accurate timing as needed, on demand? And in-between the calls, when there is no RTP or IVR, it will all go silent? I mean, sitting on a wait queue in the kernel is way better than go back and forth incrementing counters that nobody even needs at the moment? Anthony Minessale-2 wrote: idle is a 4 letter word to a realtime application. The core keeps a single high-priority thread to keep 1ms timing and expands that broadcasting to hundreds or thousand of threads who need accurate timing. Your choppy audio is caused by linksys lying about the packet len that it's using and we set our timer to the wrong speed. -- View this message in context: http://old.nabble.com/Choppy-sound-with-PCMU-tp26594250p26620610.html Sent from the Freeswitch-users mailing list archive at Nabble.com. ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] Choppy sound with PCMU
As I see it, switch_cond_next() currently is just a do_sleep(1000). Yes, it could be mapped to a 1ms timer, but #define DISABLE_1MS_COND overrides that. Yeah, there is a global timestamp... It's easy to workaround that for RTP who calls switch_micro_time_now()... But if somebody accesses runtime.timestamp directly, it's gonna be tough to grep for that. If only this was C++... I'll play around. Never liked polling too much. Never could've guessed that polling could be so useful for scalability ;) My naive implementation would've pulled timestamp via system calls and would've done sleeping by passing exact interval to select() instead of syncing with a pacing thread. Which would be dead-quiet at idle time, but, of course, would stop scaling at some point due to excessive number of system calls. Thanks. Michael Jerris wrote: In short. No, you can not for many reasons. The milisecond tic is used throughout the code even when there is not any calls up. You can grep for switch_cond_next if you would like to see where but it is required to keep our global timestamp and for pacing the scheduler among other services that run all the time. Mike On Dec 2, 2009, at 7:31 PM, eaf erandr-j...@usa.net wrote: Can I reduce resolution of that timer thread 10 times? I mean, I glanced through the code, and see that among others (are there others?) RTP and IVR set up their timers that are subsequently managed by this thread. RTP timers should be eliminated by that setting you've suggested. IVR timers are set at 20ms... So, if the thread is set to wake up every 10ms instead of 1ms it should be able to wake up those IVR timers just fine. Right? That's a cool design to have one dedicated thread that maintains accurate timing and then broadcasts via condition variables to hundreds of other threads events that they can register for. I'm sure it's one of the reasons why FS scales so much better than Asterisk. But for poor low-end setups that sit in the closet, eat only 6W of power and hardly ever run more than two calls at the same time, can I hack it somehow to be more UNIX- friendly? I.e. make it stuck in select() or recv() when there is nothing to do, call clock_gettime() right from the thread that wants and when it wants to know current time? Say, what if that thread is made to suspend on a condition variable in case if there are no timers registered in TIMER_MATRIX? Then, if some other thread comes up and adds its timer into the matrix, it could wake up the timer thread and enjoy accurate timing as needed, on demand? And in- between the calls, when there is no RTP or IVR, it will all go silent? I mean, sitting on a wait queue in the kernel is way better than go back and forth incrementing counters that nobody even needs at the moment? Anthony Minessale-2 wrote: idle is a 4 letter word to a realtime application. The core keeps a single high-priority thread to keep 1ms timing and expands that broadcasting to hundreds or thousand of threads who need accurate timing. Your choppy audio is caused by linksys lying about the packet len that it's using and we set our timer to the wrong speed. -- View this message in context: http://old.nabble.com/Choppy-sound-with-PCMU-tp26594250p26619085.html Sent from the Freeswitch-users mailing list archive at Nabble.com. ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch- users http://www.freeswitch.org ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org -- View this message in context: http://old.nabble.com/Choppy-sound-with-PCMU-tp26594250p26621005.html Sent from the Freeswitch-users mailing list archive at Nabble.com. ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
[Freeswitch-users] Choppy sound with PCMU
Hi, I'm trying to migrate from Asterisk to FreeSWITCH (really like the way how it can be programmed), but ran into one issue with sound quality that I just cannot workaround by myself. I would describe the sound problem as being choppy. From time to time small portions of the other party's voice are dropped, so the voice kind of stutters. This is not too bad, but is really noticeable, happens in every call and I don't experience the same with Asterisk running on the same box. I attached two files: freeswitch.wav and asterisk.mp3 to illustrate my point. Issue completely goes away, if I set inbound-proxy-media to true. The way how I test is to connect SPA-2000 via 10mbps LAN to the box directly exposed to internet, and then dial a toll-free via FutureNine (a SIP provider). The codec in use is PCMU. Can't really try PCMA or anything else with this provider. Only PCMU. Tried to match ptime of provider (30) with ptime of the SPA, didn't get any improvement. Tried turning off recording, no change either. What puzzles me is that even with greedy codec negotiations and with PCMU on both sides of FreeSWITCH, it's still saying that TRANSCODING_NECESSARY. I'm attaching relevant portion of freeswitch.log to illustrate. The box isn't particularly fast: Linux (Debian 4), CPU - AMD Geode LX800 with 997 bogomips. 256MB RAM. Only one call in progress, so I hope that it's not a performance issue. http://old.nabble.com/file/p26594250/freeswitch.wav freeswitch.wav http://old.nabble.com/file/p26594250/asterisk.mp3 asterisk.mp3 http://old.nabble.com/file/p26594250/freeswitch.log freeswitch.log Tried both 1.0.4 and 1.0.5pre5. Same results. What should I do next? Calls are consistently bad with FreeSWITCH, and consistently show no glitches with Asterisk. -- View this message in context: http://old.nabble.com/Choppy-sound-with-PCMU-tp26594250p26594250.html Sent from the Freeswitch-users mailing list archive at Nabble.com. ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
Re: [Freeswitch-users] Choppy sound with PCMU
I should also add, after browsing through some topics here, that my SIP provider sends 172-byte RTP frames, which is in accordance with ptime:20 that it gives to FreeSWITCH. eaf wrote: Hi, I'm trying to migrate from Asterisk to FreeSWITCH (really like the way how it can be programmed), but ran into one issue with sound quality that I just cannot workaround by myself. I would describe the sound problem as being choppy. From time to time small portions of the other party's voice are dropped, so the voice kind of stutters. This is not too bad, but is really noticeable, happens in every call and I don't experience the same with Asterisk running on the same box. I attached two files: freeswitch.wav and asterisk.mp3 to illustrate my point. Issue completely goes away, if I set inbound-proxy-media to true. The way how I test is to connect SPA-2000 via 10mbps LAN to the box directly exposed to internet, and then dial a toll-free via FutureNine (a SIP provider). The codec in use is PCMU. Can't really try PCMA or anything else with this provider. Only PCMU. Tried to match ptime of provider (30) with ptime of the SPA, didn't get any improvement. Tried turning off recording, no change either. What puzzles me is that even with greedy codec negotiations and with PCMU on both sides of FreeSWITCH, it's still saying that TRANSCODING_NECESSARY. I'm attaching relevant portion of freeswitch.log to illustrate. The box isn't particularly fast: Linux (Debian 4), CPU - AMD Geode LX800 with 997 bogomips. 256MB RAM. Only one call in progress, so I hope that it's not a performance issue. http://old.nabble.com/file/p26594250/freeswitch.wav freeswitch.wav http://old.nabble.com/file/p26594250/asterisk.mp3 asterisk.mp3 http://old.nabble.com/file/p26594250/freeswitch.log freeswitch.log Tried both 1.0.4 and 1.0.5pre5. Same results. What should I do next? Calls are consistently bad with FreeSWITCH, and consistently show no glitches with Asterisk. -- View this message in context: http://old.nabble.com/Choppy-sound-with-PCMU-tp26594250p26599565.html Sent from the Freeswitch-users mailing list archive at Nabble.com. ___ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org
