Re: [asterisk-dev] Memory leak since Asterisk 16.5.x / pjsip
On Mon, Sep 16, 2019 at 12:45 PM Corey Farrell wrote: > Updating pjproject does take less time/effort than maintaining a fork > for the many years of an LTS release. That reduced effort isn't just > free time to developers, the time is instead spent working on > enhancements and bug fixes. Maintaining a fork would prevent users from > getting important upstream bug fixes and would introduce risk of > regressions caused by the fork itself. So my vote is that pjsip version > bumps should not be held back. I'm also not in favor of creating > separate releases for pjsip version bumps, this takes time that I feel > can/should be spent on other tasks. > > One thing I think Asterisk can improve is to always make sure that any > pjsip version bump is prominently mentioned in release notes, probably a > note in the CHANGES and/or UPGRADE files. This would let people who use > bundled pjproject of potentially major changes. It would also tell > users of non-bundled pjproject that they should upgrade their own > libraries as the older version is no longer tested. > I think we normally do that unless we missed one by accident. I also do a blog post with a title like 'PJproject x.x qualified for use by Asterisk" > > On 9/16/19 1:06 PM, Michael Maier wrote: > > On 15.09.19 at 21:19 Joshua C. Colp wrote: > >> On Sun, Sep 15, 2019, at 2:21 AM, Michael Maier wrote: > >>> BTW: I'm not really happy with the fact, that an existing LTS / stable > version gets a new pjsip version "on the fly". From my point of view, this > should have been > >>> done during a normal development cycle and not during a stable phase. > >> Since support for bundled PJSIP we've actively tried to keep up to > date, so that we don't end up managing a fork and backporting a lot of > patches. This has worked well > >> for us and we haven't seen any problems - in fact we've gained some > stability at times. > > Chance - there's always a first time :-) > > BTW: I like the bundling of pjsip! > > > >> If this is a problem in PJSIP this would be the first time we've > encountered a > >> regression. If people feel that we should instead lock versions then > this is certainly something we can discuss. What do others think? > > From a developers perspective, it's for sure better to do it as you do > it like now. From a users / customers perspective, it's most probably the > other way round. I don't > > want to have any deep changes during a LTS version (that's exactly why > I'm using LTS versions). The new pjsip release should have been put to a > new asterisk release, too. > > Asterisk 16.x was thoroughly tested and released on base of pjsip 4.8. > Anybody who wants new pjsip 4.9 should consider using new Asterisk version, > too. > > > > At least, I would expect a severe distinction by using a dedicated minor > version (without any own asterisk changes) to detect more easily potential > pjsip regressions. > > > > > > Thanks > > Michael > > > > -- > _ > -- Bandwidth and Colocation Provided by http://www.api-digital.com -- > > asterisk-dev mailing list > To UNSUBSCRIBE or update options visit: >http://lists.digium.com/mailman/listinfo/asterisk-dev -- *George Joseph* Digium - A Sangoma Company | Software Developer | Software Engineering 445 Jan Davis Drive NW - Huntsville, AL 35806 - US direct/fax: +1 256 428 6012 Check us out at: https://digium.com · https://sangoma.com -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-dev mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-dev
Re: [asterisk-dev] Memory leak since Asterisk 16.5.x / pjsip
Am 16.09.19 um 20:19 schrieb Kevin Harwell: Actually can you create a new issue for this on the issue tracker. As well included the following: Has anything else changed? For instance was openssl upgraded between then too? > Speaking of which, what version of openssl is installed? As well what OS, and version? CentOS 7, openssl didn't change. It's openssl-1.0.2k-16.el7_6.1.x86_64 I assume before downgrading you were using the bundled version of Asterisk? I'm normally using *always* the bundled version - I'm working on base of the rpm source package from Schmooze Com, Inc. I've just changed the third party directory in 16.5.1 from pjsip 4.9 to 4.8 including all patches. I can provide you with the complete source package if you like. When you downgraded to pjproject 2.8 did you also install the patches that had been included with Asterisk for 2.8 that were under the "third-party/pjproject/patches/" directory? Yes Lastly, can you attach the config_site.h you used and the ./configure command you used when configuring/building pjproject 2.8. It should be the same as in the source file of Schmooze Com, Inc. I can provide the complete rpm source file if you want to have it. Both "versions", the one using 4.8 and the one using 4.9. Regards, Michael -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-dev mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-dev
Re: [asterisk-dev] Memory leak since Asterisk 16.5.x / pjsip
Updating pjproject does take less time/effort than maintaining a fork for the many years of an LTS release. That reduced effort isn't just free time to developers, the time is instead spent working on enhancements and bug fixes. Maintaining a fork would prevent users from getting important upstream bug fixes and would introduce risk of regressions caused by the fork itself. So my vote is that pjsip version bumps should not be held back. I'm also not in favor of creating separate releases for pjsip version bumps, this takes time that I feel can/should be spent on other tasks. One thing I think Asterisk can improve is to always make sure that any pjsip version bump is prominently mentioned in release notes, probably a note in the CHANGES and/or UPGRADE files. This would let people who use bundled pjproject of potentially major changes. It would also tell users of non-bundled pjproject that they should upgrade their own libraries as the older version is no longer tested. On 9/16/19 1:06 PM, Michael Maier wrote: On 15.09.19 at 21:19 Joshua C. Colp wrote: On Sun, Sep 15, 2019, at 2:21 AM, Michael Maier wrote: BTW: I'm not really happy with the fact, that an existing LTS / stable version gets a new pjsip version "on the fly". From my point of view, this should have been done during a normal development cycle and not during a stable phase. Since support for bundled PJSIP we've actively tried to keep up to date, so that we don't end up managing a fork and backporting a lot of patches. This has worked well for us and we haven't seen any problems - in fact we've gained some stability at times. Chance - there's always a first time :-) BTW: I like the bundling of pjsip! If this is a problem in PJSIP this would be the first time we've encountered a regression. If people feel that we should instead lock versions then this is certainly something we can discuss. What do others think? From a developers perspective, it's for sure better to do it as you do it like now. From a users / customers perspective, it's most probably the other way round. I don't want to have any deep changes during a LTS version (that's exactly why I'm using LTS versions). The new pjsip release should have been put to a new asterisk release, too. Asterisk 16.x was thoroughly tested and released on base of pjsip 4.8. Anybody who wants new pjsip 4.9 should consider using new Asterisk version, too. At least, I would expect a severe distinction by using a dedicated minor version (without any own asterisk changes) to detect more easily potential pjsip regressions. Thanks Michael -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-dev mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-dev
Re: [asterisk-dev] Memory leak since Asterisk 16.5.x / pjsip
Actually can you create a new issue for this on the issue tracker. As well included the following: Has anything else changed? For instance was openssl upgraded between then too? Speaking of which, what version of openssl is installed? As well what OS, and version? I assume before downgrading you were using the bundled version of Asterisk? When you downgraded to pjproject 2.8 did you also install the patches that had been included with Asterisk for 2.8 that were under the "third-party/pjproject/patches/" directory? Lastly, can you attach the config_site.h you used and the ./configure command you used when configuring/building pjproject 2.8. Thanks! On Sun, Sep 15, 2019 at 12:23 AM Michael Maier wrote: > Hello! > > Since Asterisk 16.5.x (without any additional patch at all), I' m seeing > continuously rising memory consumption each hour even if there where no > calls at all. It's about > 792 kbytes (RSS) each hour (3 SIPS / TLS 1.2 trunks, 10 min > ReRegistration, 240 s OPTIONS, 1 SIPS trunk, 60 s ReRegistration, 2 SIP > extensions) > > Therefore I tested different scenarios: > > Asterisk 16.4.1 -> no memory leak > Asterisk 16.5.1 -> memory leak > > Asterisk 16.4.1 with pjsip 4.9 -> memory leak > Asterisk 16.5.1 with pjsip 4.8 -> no memory leak > > > Therefore, I can say, that there is a problem related to pjsip 4.9 (maybe > it's a pjsip problem itself or Asterisk should have been changed to adopt > new behavior of pjsip 4.9). > > Anyway: Asterisk is quite unusable since 16.5.x. 2 times I already > encountered the OOM killer reaping an asterisk 16.5.x process consuming far > too much memory. > > > BTW: I'm not really happy with the fact, that an existing LTS / stable > version gets a new pjsip version "on the fly". From my point of view, this > should have been done > during a normal development cycle and not during a stable phase. > > > > Thanks > Michael > > -- > _ > -- Bandwidth and Colocation Provided by http://www.api-digital.com -- > > asterisk-dev mailing list > To UNSUBSCRIBE or update options visit: >http://lists.digium.com/mailman/listinfo/asterisk-dev -- Kevin Harwell Digium - A Sangoma Company | Senior Software Developer 445 Jan Davis Drive NW - Huntsville, AL 35806 - US Check us out at: https://digium.com & https://asterisk.org -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-dev mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-dev
Re: [asterisk-dev] Memory leak since Asterisk 16.5.x / pjsip
Le lun. 16 sept. 2019 13 h 07, Michael Maier a écrit : > On 15.09.19 at 21:19 Joshua C. Colp wrote: > > On Sun, Sep 15, 2019, at 2:21 AM, Michael Maier wrote: > > >> BTW: I'm not really happy with the fact, that an existing LTS / stable > version gets a new pjsip version "on the fly". From my point of view, this > should have been > >> done during a normal development cycle and not during a stable phase. > > > > Since support for bundled PJSIP we've actively tried to keep up to date, > so that we don't end up managing a fork and backporting a lot of patches. > This has worked well > > for us and we haven't seen any problems - in fact we've gained some > stability at times. > > Chance - there's always a first time :-) > BTW: I like the bundling of pjsip! > > > If this is a problem in PJSIP this would be the first time we've > encountered a > > regression. If people feel that we should instead lock versions then > this is certainly something we can discuss. What do others think? > > From a developers perspective, it's for sure better to do it as you do it > like now. From a users / customers perspective, it's most probably the > other way round. I don't > want to have any deep changes during a LTS version (that's exactly why I'm > using LTS versions). The new pjsip release should have been put to a new > asterisk release, too. > Asterisk 16.x was thoroughly tested and released on base of pjsip 4.8. > Anybody who wants new pjsip 4.9 should consider using new Asterisk version, > too. > > At least, I would expect a severe distinction by using a dedicated minor > version (without any own asterisk changes) to detect more easily potential > pjsip regressions. > > > > Thanks > Michael > > -- > _ > -- Bandwidth and Colocation Provided by http://www.api-digital.com -- > > asterisk-dev mailing list > To UNSUBSCRIBE or update options visit: >http://lists.digium.com/mailman/listinfo/asterisk-dev Hello, Probably it would be nice to choose the bundle version of pjsip at the compilation? Sylvain > -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-dev mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-dev
Re: [asterisk-dev] Memory leak since Asterisk 16.5.x / pjsip
On 15.09.19 at 21:19 Joshua C. Colp wrote: > On Sun, Sep 15, 2019, at 2:21 AM, Michael Maier wrote: >> BTW: I'm not really happy with the fact, that an existing LTS / stable >> version gets a new pjsip version "on the fly". From my point of view, this >> should have been >> done during a normal development cycle and not during a stable phase. > > Since support for bundled PJSIP we've actively tried to keep up to date, so > that we don't end up managing a fork and backporting a lot of patches. This > has worked well > for us and we haven't seen any problems - in fact we've gained some stability > at times. Chance - there's always a first time :-) BTW: I like the bundling of pjsip! > If this is a problem in PJSIP this would be the first time we've encountered a > regression. If people feel that we should instead lock versions then this is > certainly something we can discuss. What do others think? From a developers perspective, it's for sure better to do it as you do it like now. From a users / customers perspective, it's most probably the other way round. I don't want to have any deep changes during a LTS version (that's exactly why I'm using LTS versions). The new pjsip release should have been put to a new asterisk release, too. Asterisk 16.x was thoroughly tested and released on base of pjsip 4.8. Anybody who wants new pjsip 4.9 should consider using new Asterisk version, too. At least, I would expect a severe distinction by using a dedicated minor version (without any own asterisk changes) to detect more easily potential pjsip regressions. Thanks Michael -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-dev mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-dev
Re: [asterisk-dev] Memory leak since Asterisk 16.5.x / pjsip
On Mon, Sep 16, 2019 at 12:39 PM Andreas Wehrmann wrote: > > > >> BTW: I'm not really happy with the fact, that an existing LTS / stable > >> version gets a new pjsip version "on the fly". From my point of view, > >> this should have been done > >> during a normal development cycle and not during a stable phase. > > Since support for bundled PJSIP we've actively tried to keep up to date, so > > that we don't end up managing a fork and backporting a lot of patches. This > > has worked well for us and we haven't seen any problems - in fact we've > > gained some stability at times. If this is a problem in PJSIP this would be > > the first time we've encountered a regression. If people feel that we > > should instead lock versions then this is certainly something we can > > discuss. What do others think? > > > Hi, > > I think staying up to date with the latest version is better than > locking in on some version mainly for the reason of manageability and risk. > Upgrading from an 'older' release (skipping a few releases) involves a > greater risk than updating to the next minor immediately > because less changes need to be checked and integrated. The risk of > breaking something "not obvious" is simply smaller. Yeah, we've tried things like that both ways. If you stay on old versions of PJSIP, you end up with old bugs that never get fixed (including security vulnerabilities in the SIP stack). I think we've leaned towards keeping it up to date being better (as you get bug fixes in the SIP stack) than trying to own an old version of the code that very infrequently gets updated. We have no interest as a development team to maintain a long running fork of the PJSIP stack - we tried playing that game for a long time and it was problematic. There are positives and negatives to both approaches, but the last few years we've been taking the "keep it up to date approach" instead. Matthew Fredrickson > > Best regards, > Andreas > > > -- > _ > -- Bandwidth and Colocation Provided by http://www.api-digital.com -- > > asterisk-dev mailing list > To UNSUBSCRIBE or update options visit: >http://lists.digium.com/mailman/listinfo/asterisk-dev -- Matthew Fredrickson Digium - A Sangoma Company | Director of Open Source Software Development 445 Jan Davis Drive NW - Huntsville, AL 35806 - USA -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-dev mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-dev
Re: [asterisk-dev] Memory leak since Asterisk 16.5.x / pjsip
BTW: I'm not really happy with the fact, that an existing LTS / stable version gets a new pjsip version "on the fly". From my point of view, this should have been done during a normal development cycle and not during a stable phase. Since support for bundled PJSIP we've actively tried to keep up to date, so that we don't end up managing a fork and backporting a lot of patches. This has worked well for us and we haven't seen any problems - in fact we've gained some stability at times. If this is a problem in PJSIP this would be the first time we've encountered a regression. If people feel that we should instead lock versions then this is certainly something we can discuss. What do others think? Hi, I think staying up to date with the latest version is better than locking in on some version mainly for the reason of manageability and risk. Upgrading from an 'older' release (skipping a few releases) involves a greater risk than updating to the next minor immediately because less changes need to be checked and integrated. The risk of breaking something "not obvious" is simply smaller. Best regards, Andreas -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-dev mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-dev
Re: [asterisk-dev] Memory leak since Asterisk 16.5.x / pjsip
On Sun, Sep 15, 2019, at 2:21 AM, Michael Maier wrote: > Hello! Kia ora, > Since Asterisk 16.5.x (without any additional patch at all), I' m > seeing continuously rising memory consumption each hour even if there > where no calls at all. It's about > 792 kbytes (RSS) each hour (3 SIPS / TLS 1.2 trunks, 10 min > ReRegistration, 240 s OPTIONS, 1 SIPS trunk, 60 s ReRegistration, 2 SIP > extensions) > > Therefore I tested different scenarios: > > Asterisk 16.4.1 -> no memory leak > Asterisk 16.5.1 -> memory leak > > Asterisk 16.4.1 with pjsip 4.9 -> memory leak > Asterisk 16.5.1 with pjsip 4.8 -> no memory leak > > > Therefore, I can say, that there is a problem related to pjsip 4.9 > (maybe it's a pjsip problem itself or Asterisk should have been changed > to adopt new behavior of pjsip 4.9). > > Anyway: Asterisk is quite unusable since 16.5.x. 2 times I already > encountered the OOM killer reaping an asterisk 16.5.x process consuming > far too much memory. Recently two issues were created[1][2] dealing with memory leaks and Kevin Harwell is actively trying to narrow down what is happening. If you believe one is related then any additional information or details you can provide on it would be appreciated, such as your PJSIP version finding. > > > BTW: I'm not really happy with the fact, that an existing LTS / stable > version gets a new pjsip version "on the fly". From my point of view, > this should have been done > during a normal development cycle and not during a stable phase. Since support for bundled PJSIP we've actively tried to keep up to date, so that we don't end up managing a fork and backporting a lot of patches. This has worked well for us and we haven't seen any problems - in fact we've gained some stability at times. If this is a problem in PJSIP this would be the first time we've encountered a regression. If people feel that we should instead lock versions then this is certainly something we can discuss. What do others think? I'd also urge everyone reading this to consider testing the release candidates we release in different environments so we can uncover things quicker. The more testing of those we can get the better actual releases will be and it's a great way to help the project. [1] https://issues.asterisk.org/jira/browse/ASTERISK-28521 [2] https://issues.asterisk.org/jira/browse/ASTERISK-28523 -- Joshua C. Colp Digium - A Sangoma Company | Senior Software Developer 445 Jan Davis Drive NW - Huntsville, AL 35806 - US Check us out at: www.digium.com & www.asterisk.org -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-dev mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-dev
[asterisk-dev] Memory leak since Asterisk 16.5.x / pjsip
Hello! Since Asterisk 16.5.x (without any additional patch at all), I' m seeing continuously rising memory consumption each hour even if there where no calls at all. It's about 792 kbytes (RSS) each hour (3 SIPS / TLS 1.2 trunks, 10 min ReRegistration, 240 s OPTIONS, 1 SIPS trunk, 60 s ReRegistration, 2 SIP extensions) Therefore I tested different scenarios: Asterisk 16.4.1 -> no memory leak Asterisk 16.5.1 -> memory leak Asterisk 16.4.1 with pjsip 4.9 -> memory leak Asterisk 16.5.1 with pjsip 4.8 -> no memory leak Therefore, I can say, that there is a problem related to pjsip 4.9 (maybe it's a pjsip problem itself or Asterisk should have been changed to adopt new behavior of pjsip 4.9). Anyway: Asterisk is quite unusable since 16.5.x. 2 times I already encountered the OOM killer reaping an asterisk 16.5.x process consuming far too much memory. BTW: I'm not really happy with the fact, that an existing LTS / stable version gets a new pjsip version "on the fly". From my point of view, this should have been done during a normal development cycle and not during a stable phase. Thanks Michael -- _ -- Bandwidth and Colocation Provided by http://www.api-digital.com -- asterisk-dev mailing list To UNSUBSCRIBE or update options visit: http://lists.digium.com/mailman/listinfo/asterisk-dev
