Re: Out of memory crash of 3000 routes in BGP

[Madhu]

Is this issue going to be fixed ?

Please let me know.

Thanks,
Madhu

Sent from my iPhone

> On Oct 17, 2016, at 10:33 PM, Madhu  wrote:
> 
> Any update on this ?
> Madhu
> 
> Sent from my iPhone
> 
>> On Oct 14, 2016, at 11:40 AM, Madhu  wrote:
>> 
>> I still see the crash with the changes. Below is the trace.
>> 
>> [ 1037.412668] Call Trace:
>> [ 1037.415402]  [] dump_stack+0x64/0x82
>> [ 1037.421146]  [] dump_header+0x7f/0x1f1
>> [ 1037.427083]  [] ? put_online_cpus+0x56/0x80
>> [ 1037.433507]  [] ? rcu_oom_notify+0xcc/0xf0
>> [ 1037.439834]  [] oom_kill_process+0x205/0x360
>> [ 1037.446356]  [] ? security_capable_noaudit+0x15/0x20
>> [ 1037.453655]  [] out_of_memory+0x492/0x4d0
>> [ 1037.459883]  [] __alloc_pages_nodemask+0xa00/0xb60
>> [ 1037.466988]  [] alloc_pages_vma+0x9a/0x160
>> [ 1037.473314]  [] handle_mm_fault+0xd38/0x1080
>> [ 1037.479835]  [] ? change_protection+0x594/0x850
>> [ 1037.486647]  [] __do_page_fault+0x19e/0x560
>> [ 1037.493070]  [] ? mprotect_fixup+0x151/0x290
>> [ 1037.499588]  [] do_page_fault+0x31/0x70
>> [ 1037.505623]  [] page_fault+0x28/0x30
>> 
>> 
>>> On Thu, Oct 13, 2016 at 1:52 AM, Justin Cattle  wrote:
>>> 
>>> I would be:
>>> 
>>> bird_1.6.2-1~bpo8+1madhupatch1_amd64.deb
>>> 
>>> 
>>> Apologies, but I just realised I forgot the step to actually apply the 
>>> patch to the file! :)
>>> You may have realised that, and done it - but if not, just repeat the steps 
>>> again, but change the "apply patch" section to this:
>>> 
>>> # Apply patch #
>>> 
>>> cd bird-1.6.2
>>> 
>>> quilt new 1000-fixed-memory-bloating-on-kernel-merge-paths-together.patch
>>> quilt add nest/rt-table.c
>>> patch -p1 < 
>>> ../0001-Fixed-memory-bloating-on-kernel-merge-paths-together.patc
>>> quilt refresh
>>> quilt header -e # enter a description of the patch
>>> quilt push
>>> quilt refresh
>>> 
>>> 
>>> If you do have to do this process again, it's best to delete the directory 
>>> you did the work in, and create it again, or just do it again in another 
>>> directory starting from "# get source #"
>>> That should prevent any associated build issues.
>>> 
>>> 
>>> 
>>> 
>>> Cheers,
>>> Just
>>> 
 On 13 October 2016 at 01:14, Madhu  wrote:
 Thanks Just. I followed your steps. I got these files. Which one I have to 
 install?
 
 ls
 bird-1.6.2
 bird-bgp_1.6.2-1~bpo8+1madhupatch1_all.deb
 bird-doc_1.6.2-1~bpo8+1madhupatch1_all.deb
 bird_1.6.2-1~bpo8+1.debian.tar.xz
 bird_1.6.2-1~bpo8+1.dsc
 bird_1.6.2-1~bpo8+1madhupatch1.debian.tar.xz
 bird_1.6.2-1~bpo8+1madhupatch1.dsc
 bird_1.6.2-1~bpo8+1madhupatch1_amd64.changes
 bird_1.6.2-1~bpo8+1madhupatch1_amd64.deb
 bird_1.6.2.orig.tar.gz
 
 
 Please let me know.
 
 Thanks,
 Madhu
 
> On Wed, Oct 12, 2016 at 12:16 PM, Justin Cattle  wrote:
> I'd like to, but sorry I can't really.
> 
> I can give you a lot of pointers though - and it's good to learn 
> something about the package management :)
> 
> 
> First, get yourself a host that is the same debian or ubuntu version that 
> you want to deploy the package on, and use it to build the package. Then 
> create a directory where you want to do the build and cd in, preferably 
> not as root.
> 
> # setup #
> 
> sudo apt-get install devscripts build-essential dpkg-dev quilt
> 
> cat << END > ~/.quiltrc
> > export QUILT_PATCHES=debian/patches
> > export QUILT_NO_DIFF_INDEX=1
> > export QUILT_NO_DIFF_TIMESTAMPS=1
> > export QUILT_REFRESH_ARGS="-p ab"
> > export QUILT_DIFF_ARGS="--color=auto"
> > END
> 
> sudo apt-get build-dep bird
> 
> # get source #
> 
> apt-get source bird
> 
> # Apply patch #
> 
> cd bird-1.6.2
> 
> quilt new 1000-fixed-memory-bloating-on-kernel-merge-paths-together.patch
> quilt add nest/rt-table.c
> quilt refresh
> quilt header -e # enter a description of the patch
> quilt push
> quilt refresh
> 
> # Bump version with local name #
> 
> dch -l companyname # or some other name - enter into as appropriate and 
> save the file
> 
> # Build the package #
> 
> dpkg-buildpackage -tc
> 
> # That's it! #
> cd ..
> 
> ls -al  # you should see some packages if it worked.
> 
> 
> Hope that helps..
> 
> 
> 
> 
> Cheers,
> Just
> 
>> On 12 October 2016 at 18:21, Madhu  wrote:
>> Hi just,
>> 
>>Is there any way you can give me your built debian package, So that I 
>> can install and check :) ?
>> 
>> Madhu
>> 
>>> On Wed, Oct 12, 2016 at 8:37 AM, Justin Cattle  wrote:
>>> Personally - I get the debian source package, then apply the patch to 
>>> that using quilt, bump the version 

Re: Out of memory crash of 3000 routes in BGP

[Madhu]

Any update on this ?
Madhu

Sent from my iPhone

> On Oct 14, 2016, at 11:40 AM, Madhu  wrote:
> 
> I still see the crash with the changes. Below is the trace.
> 
> [ 1037.412668] Call Trace:
> [ 1037.415402]  [] dump_stack+0x64/0x82
> [ 1037.421146]  [] dump_header+0x7f/0x1f1
> [ 1037.427083]  [] ? put_online_cpus+0x56/0x80
> [ 1037.433507]  [] ? rcu_oom_notify+0xcc/0xf0
> [ 1037.439834]  [] oom_kill_process+0x205/0x360
> [ 1037.446356]  [] ? security_capable_noaudit+0x15/0x20
> [ 1037.453655]  [] out_of_memory+0x492/0x4d0
> [ 1037.459883]  [] __alloc_pages_nodemask+0xa00/0xb60
> [ 1037.466988]  [] alloc_pages_vma+0x9a/0x160
> [ 1037.473314]  [] handle_mm_fault+0xd38/0x1080
> [ 1037.479835]  [] ? change_protection+0x594/0x850
> [ 1037.486647]  [] __do_page_fault+0x19e/0x560
> [ 1037.493070]  [] ? mprotect_fixup+0x151/0x290
> [ 1037.499588]  [] do_page_fault+0x31/0x70
> [ 1037.505623]  [] page_fault+0x28/0x30
> 
> 
>> On Thu, Oct 13, 2016 at 1:52 AM, Justin Cattle  wrote:
>> 
>> I would be:
>> 
>> bird_1.6.2-1~bpo8+1madhupatch1_amd64.deb
>> 
>> 
>> Apologies, but I just realised I forgot the step to actually apply the patch 
>> to the file! :)
>> You may have realised that, and done it - but if not, just repeat the steps 
>> again, but change the "apply patch" section to this:
>> 
>> # Apply patch #
>> 
>> cd bird-1.6.2
>> 
>> quilt new 1000-fixed-memory-bloating-on-kernel-merge-paths-together.patch
>> quilt add nest/rt-table.c
>> patch -p1 < ../0001-Fixed-memory-bloating-on-kernel-merge-paths-together.patc
>> quilt refresh
>> quilt header -e # enter a description of the patch
>> quilt push
>> quilt refresh
>> 
>> 
>> If you do have to do this process again, it's best to delete the directory 
>> you did the work in, and create it again, or just do it again in another 
>> directory starting from "# get source #"
>> That should prevent any associated build issues.
>> 
>> 
>> 
>> 
>> Cheers,
>> Just
>> 
>>> On 13 October 2016 at 01:14, Madhu  wrote:
>>> Thanks Just. I followed your steps. I got these files. Which one I have to 
>>> install?
>>> 
>>> ls
>>> bird-1.6.2
>>> bird-bgp_1.6.2-1~bpo8+1madhupatch1_all.deb
>>> bird-doc_1.6.2-1~bpo8+1madhupatch1_all.deb
>>> bird_1.6.2-1~bpo8+1.debian.tar.xz
>>> bird_1.6.2-1~bpo8+1.dsc
>>> bird_1.6.2-1~bpo8+1madhupatch1.debian.tar.xz
>>> bird_1.6.2-1~bpo8+1madhupatch1.dsc
>>> bird_1.6.2-1~bpo8+1madhupatch1_amd64.changes
>>> bird_1.6.2-1~bpo8+1madhupatch1_amd64.deb
>>> bird_1.6.2.orig.tar.gz
>>> 
>>> 
>>> Please let me know.
>>> 
>>> Thanks,
>>> Madhu
>>> 
 On Wed, Oct 12, 2016 at 12:16 PM, Justin Cattle  wrote:
 I'd like to, but sorry I can't really.
 
 I can give you a lot of pointers though - and it's good to learn something 
 about the package management :)
 
 
 First, get yourself a host that is the same debian or ubuntu version that 
 you want to deploy the package on, and use it to build the package. Then 
 create a directory where you want to do the build and cd in, preferably 
 not as root.
 
 # setup #
 
 sudo apt-get install devscripts build-essential dpkg-dev quilt
 
 cat << END > ~/.quiltrc
 > export QUILT_PATCHES=debian/patches
 > export QUILT_NO_DIFF_INDEX=1
 > export QUILT_NO_DIFF_TIMESTAMPS=1
 > export QUILT_REFRESH_ARGS="-p ab"
 > export QUILT_DIFF_ARGS="--color=auto"
 > END
 
 sudo apt-get build-dep bird
 
 # get source #
 
 apt-get source bird
 
 # Apply patch #
 
 cd bird-1.6.2
 
 quilt new 1000-fixed-memory-bloating-on-kernel-merge-paths-together.patch
 quilt add nest/rt-table.c
 quilt refresh
 quilt header -e # enter a description of the patch
 quilt push
 quilt refresh
 
 # Bump version with local name #
 
 dch -l companyname # or some other name - enter into as appropriate and 
 save the file
 
 # Build the package #
 
 dpkg-buildpackage -tc
 
 # That's it! #
 cd ..
 
 ls -al  # you should see some packages if it worked.
 
 
 Hope that helps..
 
 
 
 
 Cheers,
 Just
 
> On 12 October 2016 at 18:21, Madhu  wrote:
> Hi just,
> 
>Is there any way you can give me your built debian package, So that I 
> can install and check :) ?
> 
> Madhu
> 
>> On Wed, Oct 12, 2016 at 8:37 AM, Justin Cattle  wrote:
>> Personally - I get the debian source package, then apply the patch to 
>> that using quilt, bump the version with my own local identifier, and 
>> build the new package.
>> 
>> 
>> 
>> Cheers,
>> Just
>> 
>>> On 12 October 2016 at 16:19, Madhu  wrote:
>>> How to install this patch?
>>> 
>>> Madhu
>>> 
>>> Sent from my iPhone
>>> 
>>> > On Oct 12, 2016, at 

Re: Out of memory crash of 3000 routes in BGP

[Madhu]

I still see the crash with the changes. Below is the trace.

[ 1037.412668] Call Trace:
[ 1037.415402]  [] dump_stack+0x64/0x82
[ 1037.421146]  [] dump_header+0x7f/0x1f1
[ 1037.427083]  [] ? put_online_cpus+0x56/0x80
[ 1037.433507]  [] ? rcu_oom_notify+0xcc/0xf0
[ 1037.439834]  [] oom_kill_process+0x205/0x360
[ 1037.446356]  [] ? security_capable_noaudit+0x15/0x20
[ 1037.453655]  [] out_of_memory+0x492/0x4d0
[ 1037.459883]  [] __alloc_pages_nodemask+0xa00/0xb60
[ 1037.466988]  [] alloc_pages_vma+0x9a/0x160
[ 1037.473314]  [] handle_mm_fault+0xd38/0x1080
[ 1037.479835]  [] ? change_protection+0x594/0x850
[ 1037.486647]  [] __do_page_fault+0x19e/0x560
[ 1037.493070]  [] ? mprotect_fixup+0x151/0x290
[ 1037.499588]  [] do_page_fault+0x31/0x70
[ 1037.505623]  [] page_fault+0x28/0x30


On Thu, Oct 13, 2016 at 1:52 AM, Justin Cattle  wrote:

>
> I would be:
>
> bird_1.6.2-1~bpo8+1madhupatch1_amd64.deb
>
>
> Apologies, but I just realised I forgot the step to actually apply the
> patch to the file! :)
> You may have realised that, and done it - but if not, just repeat the
> steps again, but change the "apply patch" section to this:
>
> # Apply patch #
>
> cd bird-1.6.2
>
> quilt new 1000-fixed-memory-bloating-on-kernel-merge-paths-together.patch
> quilt add nest/rt-table.c
> *patch -p1 <
> ../0001-Fixed-memory-bloating-on-kernel-merge-paths-together.patc*
> quilt refresh
> quilt header -e # enter a description of the patch
> quilt push
> quilt refresh
>
>
> If you do have to do this process again, it's best to delete the directory
> you did the work in, and create it again, or just do it again in another
> directory starting from "# get source #"
> That should prevent any associated build issues.
>
>
>
>
> Cheers,
> Just
>
> On 13 October 2016 at 01:14, Madhu  wrote:
>
>> Thanks Just. I followed your steps. I got these files. Which one I have
>> to install?
>>
>> ls
>> bird-1.6.2
>> bird-bgp_1.6.2-1~bpo8+1madhupatch1_all.deb
>> bird-doc_1.6.2-1~bpo8+1madhupatch1_all.deb
>> bird_1.6.2-1~bpo8+1.debian.tar.xz
>> bird_1.6.2-1~bpo8+1.dsc
>> bird_1.6.2-1~bpo8+1madhupatch1.debian.tar.xz
>> bird_1.6.2-1~bpo8+1madhupatch1.dsc
>> bird_1.6.2-1~bpo8+1madhupatch1_amd64.changes
>> bird_1.6.2-1~bpo8+1madhupatch1_amd64.deb
>> bird_1.6.2.orig.tar.gz
>>
>>
>> Please let me know.
>>
>> Thanks,
>> Madhu
>>
>> On Wed, Oct 12, 2016 at 12:16 PM, Justin Cattle  wrote:
>>
>>> I'd like to, but sorry I can't really.
>>>
>>> I can give you a lot of pointers though - and it's good to learn
>>> something about the package management :)
>>>
>>>
>>> First, get yourself a host that is the same debian or ubuntu version
>>> that you want to deploy the package on, and use it to build the package.
>>> Then create a directory where you want to do the build and cd in,
>>> preferably not as root.
>>>
>>> # setup #
>>>
>>> sudo apt-get install devscripts build-essential dpkg-dev quilt
>>>
>>> cat << END > ~/.quiltrc
>>> > export QUILT_PATCHES=debian/patches
>>> > export QUILT_NO_DIFF_INDEX=1
>>> > export QUILT_NO_DIFF_TIMESTAMPS=1
>>> > export QUILT_REFRESH_ARGS="-p ab"
>>> > export QUILT_DIFF_ARGS="--color=auto"
>>> > END
>>>
>>> sudo apt-get build-dep bird
>>>
>>> # get source #
>>>
>>> apt-get source bird
>>>
>>> # Apply patch #
>>>
>>> cd bird-1.6.2
>>>
>>> quilt new 1000-fixed-memory-bloating-on-kernel-merge-paths-togethe
>>> r.patch
>>> quilt add nest/rt-table.c
>>> quilt refresh
>>> quilt header -e # enter a description of the patch
>>> quilt push
>>> quilt refresh
>>>
>>> # Bump version with local name #
>>>
>>> dch -l companyname # or some other name - enter into as appropriate and
>>> save the file
>>>
>>> # Build the package #
>>>
>>> dpkg-buildpackage -tc
>>>
>>> # That's it! #
>>> cd ..
>>>
>>> ls -al  # you should see some packages if it worked.
>>>
>>>
>>> Hope that helps..
>>>
>>>
>>>
>>>
>>> Cheers,
>>> Just
>>>
>>> On 12 October 2016 at 18:21, Madhu  wrote:
>>>
 Hi just,

Is there any way you can give me your built debian package, So that
 I can install and check :) ?

 Madhu

 On Wed, Oct 12, 2016 at 8:37 AM, Justin Cattle  wrote:

> Personally - I get the debian source package, then apply the patch to
> that using quilt, bump the version with my own local identifier, and build
> the new package.
>
>
>
> Cheers,
> Just
>
> On 12 October 2016 at 16:19, Madhu  wrote:
>
>> How to install this patch?
>>
>> Madhu
>>
>> Sent from my iPhone
>>
>> > On Oct 12, 2016, at 5:27 AM, Jan Matejka 
>> wrote:
>> >
>> > Hi!
>> >
>> > Please try the last commit in GIT, branch master. Commit ID
>> 2e7fb11a6e31324151c6db98df2fe26d2d6cffab.
>> > Attaching the patch as well.
>> >
>> > Thank you both for reporting this issue.
>> >
>> > Jan
>> >
>> >> On 

Re: Out of memory crash of 3000 routes in BGP

[Justin Cattle]

I would be:

bird_1.6.2-1~bpo8+1madhupatch1_amd64.deb


Apologies, but I just realised I forgot the step to actually apply the
patch to the file! :)
You may have realised that, and done it - but if not, just repeat the steps
again, but change the "apply patch" section to this:

# Apply patch #

cd bird-1.6.2

quilt new 1000-fixed-memory-bloating-on-kernel-merge-paths-together.patch
quilt add nest/rt-table.c
*patch -p1 <
../0001-Fixed-memory-bloating-on-kernel-merge-paths-together.patc*
quilt refresh
quilt header -e # enter a description of the patch
quilt push
quilt refresh


If you do have to do this process again, it's best to delete the directory
you did the work in, and create it again, or just do it again in another
directory starting from "# get source #"
That should prevent any associated build issues.




Cheers,
Just

On 13 October 2016 at 01:14, Madhu  wrote:

> Thanks Just. I followed your steps. I got these files. Which one I have to
> install?
>
> ls
> bird-1.6.2
> bird-bgp_1.6.2-1~bpo8+1madhupatch1_all.deb
> bird-doc_1.6.2-1~bpo8+1madhupatch1_all.deb
> bird_1.6.2-1~bpo8+1.debian.tar.xz
> bird_1.6.2-1~bpo8+1.dsc
> bird_1.6.2-1~bpo8+1madhupatch1.debian.tar.xz
> bird_1.6.2-1~bpo8+1madhupatch1.dsc
> bird_1.6.2-1~bpo8+1madhupatch1_amd64.changes
> bird_1.6.2-1~bpo8+1madhupatch1_amd64.deb
> bird_1.6.2.orig.tar.gz
>
>
> Please let me know.
>
> Thanks,
> Madhu
>
> On Wed, Oct 12, 2016 at 12:16 PM, Justin Cattle  wrote:
>
>> I'd like to, but sorry I can't really.
>>
>> I can give you a lot of pointers though - and it's good to learn
>> something about the package management :)
>>
>>
>> First, get yourself a host that is the same debian or ubuntu version that
>> you want to deploy the package on, and use it to build the package. Then
>> create a directory where you want to do the build and cd in, preferably not
>> as root.
>>
>> # setup #
>>
>> sudo apt-get install devscripts build-essential dpkg-dev quilt
>>
>> cat << END > ~/.quiltrc
>> > export QUILT_PATCHES=debian/patches
>> > export QUILT_NO_DIFF_INDEX=1
>> > export QUILT_NO_DIFF_TIMESTAMPS=1
>> > export QUILT_REFRESH_ARGS="-p ab"
>> > export QUILT_DIFF_ARGS="--color=auto"
>> > END
>>
>> sudo apt-get build-dep bird
>>
>> # get source #
>>
>> apt-get source bird
>>
>> # Apply patch #
>>
>> cd bird-1.6.2
>>
>> quilt new 1000-fixed-memory-bloating-on-kernel-merge-paths-together.patch
>> quilt add nest/rt-table.c
>> quilt refresh
>> quilt header -e # enter a description of the patch
>> quilt push
>> quilt refresh
>>
>> # Bump version with local name #
>>
>> dch -l companyname # or some other name - enter into as appropriate and
>> save the file
>>
>> # Build the package #
>>
>> dpkg-buildpackage -tc
>>
>> # That's it! #
>> cd ..
>>
>> ls -al  # you should see some packages if it worked.
>>
>>
>> Hope that helps..
>>
>>
>>
>>
>> Cheers,
>> Just
>>
>> On 12 October 2016 at 18:21, Madhu  wrote:
>>
>>> Hi just,
>>>
>>>Is there any way you can give me your built debian package, So that I
>>> can install and check :) ?
>>>
>>> Madhu
>>>
>>> On Wed, Oct 12, 2016 at 8:37 AM, Justin Cattle  wrote:
>>>
 Personally - I get the debian source package, then apply the patch to
 that using quilt, bump the version with my own local identifier, and build
 the new package.



 Cheers,
 Just

 On 12 October 2016 at 16:19, Madhu  wrote:

> How to install this patch?
>
> Madhu
>
> Sent from my iPhone
>
> > On Oct 12, 2016, at 5:27 AM, Jan Matejka  wrote:
> >
> > Hi!
> >
> > Please try the last commit in GIT, branch master. Commit ID
> 2e7fb11a6e31324151c6db98df2fe26d2d6cffab.
> > Attaching the patch as well.
> >
> > Thank you both for reporting this issue.
> >
> > Jan
> >
> >> On 10/12/2016 02:10 PM, Justin Cattle wrote:
> >> Good stuff - once again, please let me know if you want to me test
> any patching at this end :)
> >>
> >>
> >>
> >> Cheers,
> >> Just
> >>
> >> On 12 October 2016 at 13:02, Ondrej Zajicek  > wrote:
> >>
> >>On Wed, Oct 12, 2016 at 12:09:17PM +0100, Justin Cattle wrote:
> >>> Are there any thoughts as to why I still see quite large memory
> usage, and
> >>> only on some [ seemingly random ] hosts ?
> >>
> >>Hi
> >>
> >>We are doing some testing and trying to identify the cause of
> the problem.
> >>We found some problems and strange behavior. One of them is
> excessive
> >>memory consumption when kernel export filters are defined.
> >>
> >>--
> >>Elen sila lumenn' omentielvo
> >>
> >>Ondrej 'Santiago' Zajicek (email: santi...@crfreenet.org
> )
> >>OpenPGP encrypted e-mails 

Re: Out of memory crash of 3000 routes in BGP

[Madhu]

Thanks Just. I followed your steps. I got these files. Which one I have to
install?

ls
bird-1.6.2
bird-bgp_1.6.2-1~bpo8+1madhupatch1_all.deb
bird-doc_1.6.2-1~bpo8+1madhupatch1_all.deb
bird_1.6.2-1~bpo8+1.debian.tar.xz
bird_1.6.2-1~bpo8+1.dsc
bird_1.6.2-1~bpo8+1madhupatch1.debian.tar.xz
bird_1.6.2-1~bpo8+1madhupatch1.dsc
bird_1.6.2-1~bpo8+1madhupatch1_amd64.changes
bird_1.6.2-1~bpo8+1madhupatch1_amd64.deb
bird_1.6.2.orig.tar.gz


Please let me know.

Thanks,
Madhu

On Wed, Oct 12, 2016 at 12:16 PM, Justin Cattle  wrote:

> I'd like to, but sorry I can't really.
>
> I can give you a lot of pointers though - and it's good to learn something
> about the package management :)
>
>
> First, get yourself a host that is the same debian or ubuntu version that
> you want to deploy the package on, and use it to build the package. Then
> create a directory where you want to do the build and cd in, preferably not
> as root.
>
> # setup #
>
> sudo apt-get install devscripts build-essential dpkg-dev quilt
>
> cat << END > ~/.quiltrc
> > export QUILT_PATCHES=debian/patches
> > export QUILT_NO_DIFF_INDEX=1
> > export QUILT_NO_DIFF_TIMESTAMPS=1
> > export QUILT_REFRESH_ARGS="-p ab"
> > export QUILT_DIFF_ARGS="--color=auto"
> > END
>
> sudo apt-get build-dep bird
>
> # get source #
>
> apt-get source bird
>
> # Apply patch #
>
> cd bird-1.6.2
>
> quilt new 1000-fixed-memory-bloating-on-kernel-merge-paths-together.patch
> quilt add nest/rt-table.c
> quilt refresh
> quilt header -e # enter a description of the patch
> quilt push
> quilt refresh
>
> # Bump version with local name #
>
> dch -l companyname # or some other name - enter into as appropriate and
> save the file
>
> # Build the package #
>
> dpkg-buildpackage -tc
>
> # That's it! #
> cd ..
>
> ls -al  # you should see some packages if it worked.
>
>
> Hope that helps..
>
>
>
>
> Cheers,
> Just
>
> On 12 October 2016 at 18:21, Madhu  wrote:
>
>> Hi just,
>>
>>Is there any way you can give me your built debian package, So that I
>> can install and check :) ?
>>
>> Madhu
>>
>> On Wed, Oct 12, 2016 at 8:37 AM, Justin Cattle  wrote:
>>
>>> Personally - I get the debian source package, then apply the patch to
>>> that using quilt, bump the version with my own local identifier, and build
>>> the new package.
>>>
>>>
>>>
>>> Cheers,
>>> Just
>>>
>>> On 12 October 2016 at 16:19, Madhu  wrote:
>>>
 How to install this patch?

 Madhu

 Sent from my iPhone

 > On Oct 12, 2016, at 5:27 AM, Jan Matejka  wrote:
 >
 > Hi!
 >
 > Please try the last commit in GIT, branch master. Commit ID
 2e7fb11a6e31324151c6db98df2fe26d2d6cffab.
 > Attaching the patch as well.
 >
 > Thank you both for reporting this issue.
 >
 > Jan
 >
 >> On 10/12/2016 02:10 PM, Justin Cattle wrote:
 >> Good stuff - once again, please let me know if you want to me test
 any patching at this end :)
 >>
 >>
 >>
 >> Cheers,
 >> Just
 >>
 >> On 12 October 2016 at 13:02, Ondrej Zajicek > wrote:
 >>
 >>On Wed, Oct 12, 2016 at 12:09:17PM +0100, Justin Cattle wrote:
 >>> Are there any thoughts as to why I still see quite large memory
 usage, and
 >>> only on some [ seemingly random ] hosts ?
 >>
 >>Hi
 >>
 >>We are doing some testing and trying to identify the cause of the
 problem.
 >>We found some problems and strange behavior. One of them is
 excessive
 >>memory consumption when kernel export filters are defined.
 >>
 >>--
 >>Elen sila lumenn' omentielvo
 >>
 >>Ondrej 'Santiago' Zajicek (email: santi...@crfreenet.org >>> santi...@crfreenet.org>)
 >>OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3,
 wwwkeys.pgp.net )
 >>"To err is human -- to blame it on a computer is even more so."
 >>
 >>
 >>
 >> Notice:  This email is confidential and may contain copyright
 material of members of the Ocado Group. Opinions and views expressed in
 this message may not necessarily reflect the opinions and views of the
 members of the Ocado Group.
 >>
 >>
 >>
 >> If you are not the intended recipient, please notify us immediately
 and delete all copies of this message. Please note that it is your
 responsibility to scan this message for viruses.
 >>
 >>
 >>
 >> Fetch and Sizzle are trading names of Speciality Stores Limited and
 Fabled is a trading name of Marie Claire Beauty Limited, both members of
 the Ocado Group.
 >>
 >>
 >>
 >> References to the “Ocado Group” are to Ocado Group plc (registered
 in England and Wales with number 7098618) and its subsidiary undertakings
 (as that expression is defined in the Companies 

Re: Out of memory crash of 3000 routes in BGP

[Madhu]

Hi just,

   Is there any way you can give me your built debian package, So that I
can install and check :) ?

Madhu

On Wed, Oct 12, 2016 at 8:37 AM, Justin Cattle  wrote:

> Personally - I get the debian source package, then apply the patch to that
> using quilt, bump the version with my own local identifier, and build the
> new package.
>
>
>
> Cheers,
> Just
>
> On 12 October 2016 at 16:19, Madhu  wrote:
>
>> How to install this patch?
>>
>> Madhu
>>
>> Sent from my iPhone
>>
>> > On Oct 12, 2016, at 5:27 AM, Jan Matejka  wrote:
>> >
>> > Hi!
>> >
>> > Please try the last commit in GIT, branch master. Commit ID
>> 2e7fb11a6e31324151c6db98df2fe26d2d6cffab.
>> > Attaching the patch as well.
>> >
>> > Thank you both for reporting this issue.
>> >
>> > Jan
>> >
>> >> On 10/12/2016 02:10 PM, Justin Cattle wrote:
>> >> Good stuff - once again, please let me know if you want to me test any
>> patching at this end :)
>> >>
>> >>
>> >>
>> >> Cheers,
>> >> Just
>> >>
>> >> On 12 October 2016 at 13:02, Ondrej Zajicek > > wrote:
>> >>
>> >>On Wed, Oct 12, 2016 at 12:09:17PM +0100, Justin Cattle wrote:
>> >>> Are there any thoughts as to why I still see quite large memory
>> usage, and
>> >>> only on some [ seemingly random ] hosts ?
>> >>
>> >>Hi
>> >>
>> >>We are doing some testing and trying to identify the cause of the
>> problem.
>> >>We found some problems and strange behavior. One of them is
>> excessive
>> >>memory consumption when kernel export filters are defined.
>> >>
>> >>--
>> >>Elen sila lumenn' omentielvo
>> >>
>> >>Ondrej 'Santiago' Zajicek (email: santi...@crfreenet.org > santi...@crfreenet.org>)
>> >>OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3,
>> wwwkeys.pgp.net )
>> >>"To err is human -- to blame it on a computer is even more so."
>> >>
>> >>
>> >>
>> >> Notice:  This email is confidential and may contain copyright material
>> of members of the Ocado Group. Opinions and views expressed in this message
>> may not necessarily reflect the opinions and views of the members of the
>> Ocado Group.
>> >>
>> >>
>> >>
>> >> If you are not the intended recipient, please notify us immediately
>> and delete all copies of this message. Please note that it is your
>> responsibility to scan this message for viruses.
>> >>
>> >>
>> >>
>> >> Fetch and Sizzle are trading names of Speciality Stores Limited and
>> Fabled is a trading name of Marie Claire Beauty Limited, both members of
>> the Ocado Group.
>> >>
>> >>
>> >>
>> >> References to the “Ocado Group” are to Ocado Group plc (registered in
>> England and Wales with number 7098618) and its subsidiary undertakings (as
>> that expression is defined in the Companies Act 2006) from time to time.
>> The registered office of Ocado Group plc is Titan Court, 3 Bishops Square,
>> Hatfield Business Park, Hatfield, Herts. AL10 9NE.
>> >>
>> >
>> > <0001-Fixed-memory-bloating-on-kernel-merge-paths-together.patch>
>>
>>
>
> Notice:  This email is confidential and may contain copyright material of
> members of the Ocado Group. Opinions and views expressed in this message
> may not necessarily reflect the opinions and views of the members of the
> Ocado Group.
>
>
>
> If you are not the intended recipient, please notify us immediately and
> delete all copies of this message. Please note that it is your
> responsibility to scan this message for viruses.
>
>
>
> Fetch and Sizzle are trading names of Speciality Stores Limited and Fabled
> is a trading name of Marie Claire Beauty Limited, both members of the Ocado
> Group.
>
>
>
> References to the “Ocado Group” are to Ocado Group plc (registered in
> England and Wales with number 7098618) and its subsidiary undertakings (as
> that expression is defined in the Companies Act 2006) from time to time.
> The registered office of Ocado Group plc is Titan Court, 3 Bishops Square,
> Hatfield Business Park, Hatfield, Herts. AL10 9NE.
>

Re: Out of memory crash of 3000 routes in BGP

[Justin Cattle]

Personally - I get the debian source package, then apply the patch to that
using quilt, bump the version with my own local identifier, and build the
new package.



Cheers,
Just

On 12 October 2016 at 16:19, Madhu  wrote:

> How to install this patch?
>
> Madhu
>
> Sent from my iPhone
>
> > On Oct 12, 2016, at 5:27 AM, Jan Matejka  wrote:
> >
> > Hi!
> >
> > Please try the last commit in GIT, branch master. Commit ID
> 2e7fb11a6e31324151c6db98df2fe26d2d6cffab.
> > Attaching the patch as well.
> >
> > Thank you both for reporting this issue.
> >
> > Jan
> >
> >> On 10/12/2016 02:10 PM, Justin Cattle wrote:
> >> Good stuff - once again, please let me know if you want to me test any
> patching at this end :)
> >>
> >>
> >>
> >> Cheers,
> >> Just
> >>
> >> On 12 October 2016 at 13:02, Ondrej Zajicek  > wrote:
> >>
> >>On Wed, Oct 12, 2016 at 12:09:17PM +0100, Justin Cattle wrote:
> >>> Are there any thoughts as to why I still see quite large memory usage,
> and
> >>> only on some [ seemingly random ] hosts ?
> >>
> >>Hi
> >>
> >>We are doing some testing and trying to identify the cause of the
> problem.
> >>We found some problems and strange behavior. One of them is excessive
> >>memory consumption when kernel export filters are defined.
> >>
> >>--
> >>Elen sila lumenn' omentielvo
> >>
> >>Ondrej 'Santiago' Zajicek (email: santi...@crfreenet.org  santi...@crfreenet.org>)
> >>OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3,
> wwwkeys.pgp.net )
> >>"To err is human -- to blame it on a computer is even more so."
> >>
> >>
> >>
> >> Notice:  This email is confidential and may contain copyright material
> of members of the Ocado Group. Opinions and views expressed in this message
> may not necessarily reflect the opinions and views of the members of the
> Ocado Group.
> >>
> >>
> >>
> >> If you are not the intended recipient, please notify us immediately and
> delete all copies of this message. Please note that it is your
> responsibility to scan this message for viruses.
> >>
> >>
> >>
> >> Fetch and Sizzle are trading names of Speciality Stores Limited and
> Fabled is a trading name of Marie Claire Beauty Limited, both members of
> the Ocado Group.
> >>
> >>
> >>
> >> References to the “Ocado Group” are to Ocado Group plc (registered in
> England and Wales with number 7098618) and its subsidiary undertakings (as
> that expression is defined in the Companies Act 2006) from time to time.
> The registered office of Ocado Group plc is Titan Court, 3 Bishops Square,
> Hatfield Business Park, Hatfield, Herts. AL10 9NE.
> >>
> >
> > <0001-Fixed-memory-bloating-on-kernel-merge-paths-together.patch>
>
>

-- 


Notice:  This email is confidential and may contain copyright material of 
members of the Ocado Group. Opinions and views expressed in this message 
may not necessarily reflect the opinions and views of the members of the 
Ocado Group. 

 

If you are not the intended recipient, please notify us immediately and 
delete all copies of this message. Please note that it is your 
responsibility to scan this message for viruses. 

 

Fetch and Sizzle are trading names of Speciality Stores Limited and Fabled 
is a trading name of Marie Claire Beauty Limited, both members of the Ocado 
Group.

 

References to the “Ocado Group” are to Ocado Group plc (registered in 
England and Wales with number 7098618) and its subsidiary undertakings (as 
that expression is defined in the Companies Act 2006) from time to time. 
 The registered office of Ocado Group plc is Titan Court, 3 Bishops Square, 
Hatfield Business Park, Hatfield, Herts. AL10 9NE.

Re: Out of memory crash of 3000 routes in BGP

[Madhu]

How to install this patch?

Madhu

Sent from my iPhone

> On Oct 12, 2016, at 5:27 AM, Jan Matejka  wrote:
> 
> Hi!
> 
> Please try the last commit in GIT, branch master. Commit ID 
> 2e7fb11a6e31324151c6db98df2fe26d2d6cffab.
> Attaching the patch as well.
> 
> Thank you both for reporting this issue.
> 
> Jan
> 
>> On 10/12/2016 02:10 PM, Justin Cattle wrote:
>> Good stuff - once again, please let me know if you want to me test any 
>> patching at this end :)
>> 
>> 
>> 
>> Cheers,
>> Just
>> 
>> On 12 October 2016 at 13:02, Ondrej Zajicek > > wrote:
>> 
>>On Wed, Oct 12, 2016 at 12:09:17PM +0100, Justin Cattle wrote:
>>> Are there any thoughts as to why I still see quite large memory usage, and
>>> only on some [ seemingly random ] hosts ?
>> 
>>Hi
>> 
>>We are doing some testing and trying to identify the cause of the problem.
>>We found some problems and strange behavior. One of them is excessive
>>memory consumption when kernel export filters are defined.
>> 
>>--
>>Elen sila lumenn' omentielvo
>> 
>>Ondrej 'Santiago' Zajicek (email: santi...@crfreenet.org 
>> )
>>OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net 
>> )
>>"To err is human -- to blame it on a computer is even more so."
>> 
>> 
>> 
>> Notice:  This email is confidential and may contain copyright material of 
>> members of the Ocado Group. Opinions and views expressed in this message may 
>> not necessarily reflect the opinions and views of the members of the Ocado 
>> Group. 
>> 
>> 
>> 
>> If you are not the intended recipient, please notify us immediately and 
>> delete all copies of this message. Please note that it is your 
>> responsibility to scan this message for viruses. 
>> 
>> 
>> 
>> Fetch and Sizzle are trading names of Speciality Stores Limited and Fabled 
>> is a trading name of Marie Claire Beauty Limited, both members of the Ocado 
>> Group.
>> 
>> 
>> 
>> References to the “Ocado Group” are to Ocado Group plc (registered in 
>> England and Wales with number 7098618) and its subsidiary undertakings (as 
>> that expression is defined in the Companies Act 2006) from time to time.  
>> The registered office of Ocado Group plc is Titan Court, 3 Bishops Square, 
>> Hatfield Business Park, Hatfield, Herts. AL10 9NE.
>> 
> 
> <0001-Fixed-memory-bloating-on-kernel-merge-paths-together.patch>

Re: Out of memory crash of 3000 routes in BGP

[Jan Matejka]

Hi!

Please try the last commit in GIT, branch master. Commit ID 
2e7fb11a6e31324151c6db98df2fe26d2d6cffab.
Attaching the patch as well.

Thank you both for reporting this issue.

Jan

On 10/12/2016 02:10 PM, Justin Cattle wrote:
> Good stuff - once again, please let me know if you want to me test any 
> patching at this end :)
> 
> 
> 
> Cheers,
> Just
> 
> On 12 October 2016 at 13:02, Ondrej Zajicek  > wrote:
> 
> On Wed, Oct 12, 2016 at 12:09:17PM +0100, Justin Cattle wrote:
> > Are there any thoughts as to why I still see quite large memory usage, 
> and
> > only on some [ seemingly random ] hosts ?
> 
> Hi
> 
> We are doing some testing and trying to identify the cause of the problem.
> We found some problems and strange behavior. One of them is excessive
> memory consumption when kernel export filters are defined.
> 
> --
> Elen sila lumenn' omentielvo
> 
> Ondrej 'Santiago' Zajicek (email: santi...@crfreenet.org 
> )
> OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net 
> )
> "To err is human -- to blame it on a computer is even more so."
> 
> 
> 
> Notice:  This email is confidential and may contain copyright material of 
> members of the Ocado Group. Opinions and views expressed in this message may 
> not necessarily reflect the opinions and views of the members of the Ocado 
> Group. 
> 
>  
> 
> If you are not the intended recipient, please notify us immediately and 
> delete all copies of this message. Please note that it is your responsibility 
> to scan this message for viruses. 
> 
>  
> 
> Fetch and Sizzle are trading names of Speciality Stores Limited and Fabled is 
> a trading name of Marie Claire Beauty Limited, both members of the Ocado 
> Group.
> 
>  
> 
> References to the “Ocado Group” are to Ocado Group plc (registered in England 
> and Wales with number 7098618) and its subsidiary undertakings (as that 
> expression is defined in the Companies Act 2006) from time to time.  The 
> registered office of Ocado Group plc is Titan Court, 3 Bishops Square, 
> Hatfield Business Park, Hatfield, Herts. AL10 9NE.
> 

>From 2e7fb11a6e31324151c6db98df2fe26d2d6cffab Mon Sep 17 00:00:00 2001
From: Jan Moskyto Matejka 
Date: Wed, 12 Oct 2016 14:16:34 +0200
Subject: [PATCH] Fixed memory bloating on kernel merge paths together with
 export filter.

Some memory was being allocated from bad linpool, not from the given one
as they should.

Thanks to Madhu and Justin Cattle for reporting this.
---
 nest/rt-table.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/nest/rt-table.c b/nest/rt-table.c
index d3aba08..0047606 100644
--- a/nest/rt-table.c
+++ b/nest/rt-table.c
@@ -65,7 +65,7 @@ make_tmp_attrs(struct rte *rt, struct linpool *pool)
 {
   struct ea_list *(*mta)(struct rte *rt, struct linpool *pool);
   mta = rt->attrs->src->proto->make_tmp_attrs;
-  return mta ? mta(rt, rte_update_pool) : NULL;
+  return mta ? mta(rt, pool) : NULL;
 }
 
 /* Like fib_route(), but skips empty net entries */
@@ -596,7 +596,7 @@ mpnh_merge_rta(struct mpnh *nhs, rta *a, linpool *pool, int max)
 {
   struct mpnh nh = { .gw = a->gw, .iface = a->iface };
   struct mpnh *nh2 = (a->dest == RTD_MULTIPATH) ? a->nexthops : 
-  return mpnh_merge(nhs, nh2, 1, 0, max, rte_update_pool);
+  return mpnh_merge(nhs, nh2, 1, 0, max, pool);
 }
 
 rte *
-- 
2.9.3

Re: Out of memory crash of 3000 routes in BGP

[Justin Cattle]

Good stuff - once again, please let me know if you want to me test any
patching at this end :)



Cheers,
Just

On 12 October 2016 at 13:02, Ondrej Zajicek  wrote:

> On Wed, Oct 12, 2016 at 12:09:17PM +0100, Justin Cattle wrote:
> > Are there any thoughts as to why I still see quite large memory usage,
> and
> > only on some [ seemingly random ] hosts ?
>
> Hi
>
> We are doing some testing and trying to identify the cause of the problem.
> We found some problems and strange behavior. One of them is excessive
> memory consumption when kernel export filters are defined.
>
> --
> Elen sila lumenn' omentielvo
>
> Ondrej 'Santiago' Zajicek (email: santi...@crfreenet.org)
> OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net)
> "To err is human -- to blame it on a computer is even more so."
>

-- 


Notice:  This email is confidential and may contain copyright material of 
members of the Ocado Group. Opinions and views expressed in this message 
may not necessarily reflect the opinions and views of the members of the 
Ocado Group. 

 

If you are not the intended recipient, please notify us immediately and 
delete all copies of this message. Please note that it is your 
responsibility to scan this message for viruses. 

 

Fetch and Sizzle are trading names of Speciality Stores Limited and Fabled 
is a trading name of Marie Claire Beauty Limited, both members of the Ocado 
Group.

 

References to the “Ocado Group” are to Ocado Group plc (registered in 
England and Wales with number 7098618) and its subsidiary undertakings (as 
that expression is defined in the Companies Act 2006) from time to time. 
 The registered office of Ocado Group plc is Titan Court, 3 Bishops Square, 
Hatfield Business Park, Hatfield, Herts. AL10 9NE.

Re: Out of memory crash of 3000 routes in BGP

[Ondrej Zajicek]

On Wed, Oct 12, 2016 at 12:09:17PM +0100, Justin Cattle wrote:
> Are there any thoughts as to why I still see quite large memory usage, and
> only on some [ seemingly random ] hosts ?

Hi

We are doing some testing and trying to identify the cause of the problem.
We found some problems and strange behavior. One of them is excessive
memory consumption when kernel export filters are defined.

-- 
Elen sila lumenn' omentielvo

Ondrej 'Santiago' Zajicek (email: santi...@crfreenet.org)
OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net)
"To err is human -- to blame it on a computer is even more so."

Re: Out of memory crash of 3000 routes in BGP

[Justin Cattle]

Are there any thoughts as to why I still see quite large memory usage, and
only on some [ seemingly random ] hosts ?



Cheers,
Just

On 11 October 2016 at 19:05, Madhu  wrote:

> Hi everyone,
>
>   Now i tested with 16 path and 32 path, I don't see the crash. Below are
> the memory usage for each.
>
> *16 path*
>
> bird> show memory
> BIRD memory usage
> Routing tables:   5109 kB
> Route attributes:   13 kB
> ROA tables:192  B
> Protocols: 163 kB
> Total:5384 kB
>
>
> *32 path*
>
> bird> show memory
> BIRD memory usage
> Routing tables: 10 MB
> Route attributes:   16 kB
> ROA tables:192  B
> Protocols: 239 kB
> Total:  10 MB
>
>
> Now I increased to* 62 path*, I see that* CPU is going high to 100%*, and
> the memory usage is 0.1%. birdc socket got struck. I was not able to access
> the socket.
>
> bird> show memory
> BIRD memory usage
> Routing tables:   5127 kB
> Route attributes:   20 kB
> ROA tables:192  B
> Protocols: 382 kB
> Total:5641 kB
> bird> show memory >>. After this it got struck
>
> # top
> top - 17:47:50 up  1:02,  2 users,  load average: 1.35, 1.42, 1.10
> Tasks: 163 total,   2 running, 161 sleeping,   0 stopped,   0 zombie
> %Cpu(s): 29.7 us,  0.5 sy,  0.2 ni, 69.4 id,  0.0 wa,  0.0 hi,  0.2 si,
> 0.0 st
> KiB Mem:   8127384 total,  1279304 used,  6848080 free,36040 buffers
> KiB Swap:   812736 total,0 used,   812736 free.   456220 cached Mem
>
>   PID USER  PR  NIVIRTRESSHR S  %CPU %MEM TIME+ COMMAND
>  8768 bird  20   0   21728   8808   2352 R* 100.0*  0.1   8:12.97
> bird
>
> Attached the bird config file.
>
> Madhu
>
>
>
>
>
>
> On Tue, Oct 11, 2016 at 5:56 AM, Justin Cattle  wrote:
>
>> It looks like oom-kill from the call trace I think.
>>
>>
>>
>> Cheers,
>> Just
>>
>> On 11 October 2016 at 13:07, Ondrej Zajicek 
>> wrote:
>>
>>> On Mon, Oct 10, 2016 at 06:21:44PM -0700, Madhu wrote:
>>> > I have 3000 routes with 64 path ecmp. It is crashing . I don't see the
>>> coredump . Is there anyway to fix using trace ?
>>>
>>> Hi
>>>
>>> I would like to confirm, it is BIRD that was killed by out of memory
>>> killer and could be just restarted? Or Linux kernel crashed? (You wrote
>>> 'the kernel crashed with out of memory issue' in one prior mail.)
>>>
>>> In the first case, how much memory BIRD takes after start and before it
>>> crashes?
>>> How long it takes to crash to crash due to OOM?
>>>
>>>
>>> > On Oct 10, 2016, at 6:08 PM, Jonathan Stewart <
>>> jonathan.stew...@gmail.com> wrote:
>>> > It must be something specific to your environment.
>>>
>>> It is likely related to 64 path BGP multipath exported to kernel.
>>>
>>> --
>>> Elen sila lumenn' omentielvo
>>>
>>> Ondrej 'Santiago' Zajicek (email: santi...@crfreenet.org)
>>> OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net)
>>> "To err is human -- to blame it on a computer is even more so."
>>>
>>>
>>
>> Notice:  This email is confidential and may contain copyright material of
>> members of the Ocado Group. Opinions and views expressed in this message
>> may not necessarily reflect the opinions and views of the members of the
>> Ocado Group.
>>
>>
>>
>> If you are not the intended recipient, please notify us immediately and
>> delete all copies of this message. Please note that it is your
>> responsibility to scan this message for viruses.
>>
>>
>>
>> Fetch and Sizzle are trading names of Speciality Stores Limited and
>> Fabled is a trading name of Marie Claire Beauty Limited, both members of
>> the Ocado Group.
>>
>>
>>
>> References to the “Ocado Group” are to Ocado Group plc (registered in
>> England and Wales with number 7098618) and its subsidiary undertakings (as
>> that expression is defined in the Companies Act 2006) from time to time.
>> The registered office of Ocado Group plc is Titan Court, 3 Bishops Square,
>> Hatfield Business Park, Hatfield, Herts. AL10 9NE.
>>
>
>

-- 


Notice:  This email is confidential and may contain copyright material of 
members of the Ocado Group. Opinions and views expressed in this message 
may not necessarily reflect the opinions and views of the members of the 
Ocado Group. 

 

If you are not the intended recipient, please notify us immediately and 
delete all copies of this message. Please note that it is your 
responsibility to scan this message for viruses. 

 

Fetch and Sizzle are trading names of Speciality Stores Limited and Fabled 
is a trading name of Marie Claire Beauty Limited, both members of the Ocado 
Group.

 

References to the “Ocado Group” are to Ocado Group plc (registered in 
England and Wales with number 7098618) and its subsidiary undertakings (as 
that expression is defined in the Companies Act 2006) from time to time. 
 The registered office of Ocado Group plc is Titan Court, 3 Bishops Square, 
Hatfield Business Park, Hatfield, Herts. AL10 9NE.

Re: Out of memory crash of 3000 routes in BGP

[Madhu]

Hi everyone,

  Now i tested with 16 path and 32 path, I don't see the crash. Below are
the memory usage for each.

*16 path*

bird> show memory
BIRD memory usage
Routing tables:   5109 kB
Route attributes:   13 kB
ROA tables:192  B
Protocols: 163 kB
Total:5384 kB


*32 path*

bird> show memory
BIRD memory usage
Routing tables: 10 MB
Route attributes:   16 kB
ROA tables:192  B
Protocols: 239 kB
Total:  10 MB


Now I increased to* 62 path*, I see that* CPU is going high to 100%*, and
the memory usage is 0.1%. birdc socket got struck. I was not able to access
the socket.

bird> show memory
BIRD memory usage
Routing tables:   5127 kB
Route attributes:   20 kB
ROA tables:192  B
Protocols: 382 kB
Total:5641 kB
bird> show memory >>. After this it got struck

# top
top - 17:47:50 up  1:02,  2 users,  load average: 1.35, 1.42, 1.10
Tasks: 163 total,   2 running, 161 sleeping,   0 stopped,   0 zombie
%Cpu(s): 29.7 us,  0.5 sy,  0.2 ni, 69.4 id,  0.0 wa,  0.0 hi,  0.2 si,
0.0 st
KiB Mem:   8127384 total,  1279304 used,  6848080 free,36040 buffers
KiB Swap:   812736 total,0 used,   812736 free.   456220 cached Mem

  PID USER  PR  NIVIRTRESSHR S  %CPU %MEM TIME+ COMMAND
 8768 bird  20   0   21728   8808   2352 R* 100.0*  0.1   8:12.97
bird

Attached the bird config file.

Madhu






On Tue, Oct 11, 2016 at 5:56 AM, Justin Cattle  wrote:

> It looks like oom-kill from the call trace I think.
>
>
>
> Cheers,
> Just
>
> On 11 October 2016 at 13:07, Ondrej Zajicek 
> wrote:
>
>> On Mon, Oct 10, 2016 at 06:21:44PM -0700, Madhu wrote:
>> > I have 3000 routes with 64 path ecmp. It is crashing . I don't see the
>> coredump . Is there anyway to fix using trace ?
>>
>> Hi
>>
>> I would like to confirm, it is BIRD that was killed by out of memory
>> killer and could be just restarted? Or Linux kernel crashed? (You wrote
>> 'the kernel crashed with out of memory issue' in one prior mail.)
>>
>> In the first case, how much memory BIRD takes after start and before it
>> crashes?
>> How long it takes to crash to crash due to OOM?
>>
>>
>> > On Oct 10, 2016, at 6:08 PM, Jonathan Stewart <
>> jonathan.stew...@gmail.com> wrote:
>> > It must be something specific to your environment.
>>
>> It is likely related to 64 path BGP multipath exported to kernel.
>>
>> --
>> Elen sila lumenn' omentielvo
>>
>> Ondrej 'Santiago' Zajicek (email: santi...@crfreenet.org)
>> OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net)
>> "To err is human -- to blame it on a computer is even more so."
>>
>>
>
> Notice:  This email is confidential and may contain copyright material of
> members of the Ocado Group. Opinions and views expressed in this message
> may not necessarily reflect the opinions and views of the members of the
> Ocado Group.
>
>
>
> If you are not the intended recipient, please notify us immediately and
> delete all copies of this message. Please note that it is your
> responsibility to scan this message for viruses.
>
>
>
> Fetch and Sizzle are trading names of Speciality Stores Limited and Fabled
> is a trading name of Marie Claire Beauty Limited, both members of the Ocado
> Group.
>
>
>
> References to the “Ocado Group” are to Ocado Group plc (registered in
> England and Wales with number 7098618) and its subsidiary undertakings (as
> that expression is defined in the Companies Act 2006) from time to time.
> The registered office of Ocado Group plc is Titan Court, 3 Bishops Square,
> Hatfield Business Park, Hatfield, Herts. AL10 9NE.
>
router id 100.0.2.1;

log syslog all;

protocol kernel {
scan time 60;
import all;
merge paths yes limit 255;
export all;   # Actually insert routes into the kernel routing table
}

protocol device {
scan time 60;
}

filter bgp_import {
if net ~ [ 120.100.0.0/16+] then accept;
reject;
}

filter kernel_export {
if net ~ [ 120.100.0.0/16+] then accept;
reject;
}

filter bgp_export {
reject;
}



protocol bgp 'AG1-TR1' {
   import all;
#   export where proto = "static_bgp";
   local as 64801;
   neighbor 100.0.3.1 as 64801;
   source address 100.0.2.1;
   rr client;
   next hop self;
   keepalive time 1;
   hold time 3;
}

protocol bgp 'AG1-TR2' {
   local as 64801;
   neighbor 100.0.3.2 as 64801;
   source address 100.0.2.1;
   rr client;
   next hop self;
   keepalive time 1;
   hold time 3;
}
protocol bgp 'AG1-CR1-link0' {
   local as 64801;
   neighbor 100.1.2.0 as 64901;
   source address 100.1.2.1;
#   import filter bgp_import;
#   export filter bgp_export;
}



protocol bgp 'AG1-CR1-link1' {
   local as 64801;
   neighbor 100.1.3.0 as 64901;
   source address 100.1.3.1;
#   import filter bgp_import;
#   export filter bgp_export;
}

protocol bgp 

Re: Out of memory crash of 3000 routes in BGP

[Justin Cattle]

It looks like oom-kill from the call trace I think.



Cheers,
Just

On 11 October 2016 at 13:07, Ondrej Zajicek  wrote:

> On Mon, Oct 10, 2016 at 06:21:44PM -0700, Madhu wrote:
> > I have 3000 routes with 64 path ecmp. It is crashing . I don't see the
> coredump . Is there anyway to fix using trace ?
>
> Hi
>
> I would like to confirm, it is BIRD that was killed by out of memory
> killer and could be just restarted? Or Linux kernel crashed? (You wrote
> 'the kernel crashed with out of memory issue' in one prior mail.)
>
> In the first case, how much memory BIRD takes after start and before it
> crashes?
> How long it takes to crash to crash due to OOM?
>
>
> > On Oct 10, 2016, at 6:08 PM, Jonathan Stewart <
> jonathan.stew...@gmail.com> wrote:
> > It must be something specific to your environment.
>
> It is likely related to 64 path BGP multipath exported to kernel.
>
> --
> Elen sila lumenn' omentielvo
>
> Ondrej 'Santiago' Zajicek (email: santi...@crfreenet.org)
> OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net)
> "To err is human -- to blame it on a computer is even more so."
>
>

-- 


Notice:  This email is confidential and may contain copyright material of 
members of the Ocado Group. Opinions and views expressed in this message 
may not necessarily reflect the opinions and views of the members of the 
Ocado Group. 

 

If you are not the intended recipient, please notify us immediately and 
delete all copies of this message. Please note that it is your 
responsibility to scan this message for viruses. 

 

Fetch and Sizzle are trading names of Speciality Stores Limited and Fabled 
is a trading name of Marie Claire Beauty Limited, both members of the Ocado 
Group.

 

References to the “Ocado Group” are to Ocado Group plc (registered in 
England and Wales with number 7098618) and its subsidiary undertakings (as 
that expression is defined in the Companies Act 2006) from time to time. 
 The registered office of Ocado Group plc is Titan Court, 3 Bishops Square, 
Hatfield Business Park, Hatfield, Herts. AL10 9NE.

Re: Out of memory crash of 3000 routes in BGP

[Ondrej Zajicek]

On Mon, Oct 10, 2016 at 06:21:44PM -0700, Madhu wrote:
> I have 3000 routes with 64 path ecmp. It is crashing . I don't see the 
> coredump . Is there anyway to fix using trace ?

Hi

I would like to confirm, it is BIRD that was killed by out of memory
killer and could be just restarted? Or Linux kernel crashed? (You wrote
'the kernel crashed with out of memory issue' in one prior mail.)

In the first case, how much memory BIRD takes after start and before it crashes?
How long it takes to crash to crash due to OOM?


> On Oct 10, 2016, at 6:08 PM, Jonathan Stewart  
> wrote:
> It must be something specific to your environment.

It is likely related to 64 path BGP multipath exported to kernel.

-- 
Elen sila lumenn' omentielvo

Ondrej 'Santiago' Zajicek (email: santi...@crfreenet.org)
OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net)
"To err is human -- to blame it on a computer is even more so."

Re: Out of memory crash of 3000 routes in BGP

[Justin Cattle]

I must admit, after deploying 1.6.2 to production, I am still seeing more
memory usage than expected.
I was going to take some time to gather some more stats - but seeing this
made me jump early :)

Memory usage is still better than it was originally, but I see 2G in some
cases, which seems pretty excessive for 2500 routes.

The other strange this is, I see a lot of variance across hosts with the
same route count, config, and similar process run times.
Some node processes are using 12Mg - 16Mg, while some are using up
to 2228Mg after just one day.

The bird.conf has changed a little, so I'll attach it again here.
Madhu - perhaps you should provide your bird.conf as well ?


Also - some basic info:


# birdc
BIRD 1.6.2 ready.
bird>
bird> show route count
2484 of 2484 routes for 1245 networks
bird> show mem
BIRD memory usage
Routing tables:   2235 MB
Route attributes:   87 kB
ROA tables:192  B
Protocols:  41 kB
Total:2235 MB



If you need any further info - please let me know.





Cheers,
Just

On 11 October 2016 at 02:21, Madhu  wrote:

> I have 3000 routes with 64 path ecmp. It is crashing . I don't see the
> coredump . Is there anyway to fix using trace ?
>
> Madhu
>
> Sent from my iPhone
>
> On Oct 10, 2016, at 6:08 PM, Jonathan Stewart 
> wrote:
>
> It is not a general issue, many here run hundreds of thousands of routes.
>
> It must be something specific to your environment.
>
> Jonathan
>
> On 10 Oct 2016 4:40 p.m., "Madhu"  wrote:
>
>> Looks like the problem is still there. I used 1.6.0 and also 1.6.2.
>>
>>
>>
>> On Mon, Oct 10, 2016 at 10:56 AM, james machado 
>> wrote:
>>
>>> Madhu check out the following thread.  http://bird.network.cz/piperm
>>> ail/bird-users/2016-September/010578.html
>>>
>>>
>>>
>>> On Mon, Oct 10, 2016 at 10:02 AM, Madhu  wrote:
>>>
 Hi,

   I am seeing a out of memory crash with 3000 routes in BGP. Is this a
 known issue? I m seeing this issue in 1.6.0 and 1.6.2 .


 [ 1098.912753] Call Trace:
 [ 1098.915481]  [] dump_stack+0x64/0x82
 [ 1098.921225]  [] dump_header+0x7f/0x1f1
 [ 1098.927163]  [] ? put_online_cpus+0x56/0x80
 [ 1098.933587]  [] ? rcu_oom_notify+0xcc/0xf0
 [ 1098.939914]  [] oom_kill_process+0x205/0x360
 [ 1098.946435]  [] ? security_capable_noaudit+0x15/
 0x20
 [ 1098.953733]  [] out_of_memory+0x492/0x4d0
 [ 1098.959962]  [] __alloc_pages_nodemask+0xa00/0xb60
 [ 1098.967067]  [] alloc_pages_vma+0x9a/0x160
 [ 1098.973393]  [] handle_mm_fault+0xd38/0x1080
 [ 1098.979914]  [] ? change_protection+0x594/0x850
 [ 1098.986726]  [] __do_page_fault+0x19e/0x560
 [ 1098.993149]  [] ? mprotect_fixup+0x151/0x290
 [ 1098.999669]  [] do_page_fault+0x31/0x70
 [ 1099.005704]  [] page_fault+0x28/0x30

 Madhu

>>>
>>>
>>

-- 


Notice:  This email is confidential and may contain copyright material of 
members of the Ocado Group. Opinions and views expressed in this message 
may not necessarily reflect the opinions and views of the members of the 
Ocado Group. 

 

If you are not the intended recipient, please notify us immediately and 
delete all copies of this message. Please note that it is your 
responsibility to scan this message for viruses. 

 

Fetch and Sizzle are trading names of Speciality Stores Limited and Fabled 
is a trading name of Marie Claire Beauty Limited, both members of the Ocado 
Group.

 

References to the “Ocado Group” are to Ocado Group plc (registered in 
England and Wales with number 7098618) and its subsidiary undertakings (as 
that expression is defined in the Companies Act 2006) from time to time. 
 The registered office of Ocado Group plc is Titan Court, 3 Bishops Square, 
Hatfield Business Park, Hatfield, Herts. AL10 9NE.


bird.conf
Description: Binary data

Re: Out of memory crash of 3000 routes in BGP

[Madhu]

I have 3000 routes with 64 path ecmp. It is crashing . I don't see the coredump 
. Is there anyway to fix using trace ?

Madhu

Sent from my iPhone

> On Oct 10, 2016, at 6:08 PM, Jonathan Stewart  
> wrote:
> 
> It is not a general issue, many here run hundreds of thousands of routes.
> 
> It must be something specific to your environment.
> 
> Jonathan
> 
> 
>> On 10 Oct 2016 4:40 p.m., "Madhu"  wrote:
>> Looks like the problem is still there. I used 1.6.0 and also 1.6.2.
>> 
>> 
>> 
>>> On Mon, Oct 10, 2016 at 10:56 AM, james machado  
>>> wrote:
>>> Madhu check out the following thread.  
>>> http://bird.network.cz/pipermail/bird-users/2016-September/010578.html
>>> 
>>> 
>>> 
 On Mon, Oct 10, 2016 at 10:02 AM, Madhu  wrote:
 Hi,
 
   I am seeing a out of memory crash with 3000 routes in BGP. Is this a 
 known issue? I m seeing this issue in 1.6.0 and 1.6.2 .
 
 
 [ 1098.912753] Call Trace:
 [ 1098.915481]  [] dump_stack+0x64/0x82
 [ 1098.921225]  [] dump_header+0x7f/0x1f1
 [ 1098.927163]  [] ? put_online_cpus+0x56/0x80
 [ 1098.933587]  [] ? rcu_oom_notify+0xcc/0xf0
 [ 1098.939914]  [] oom_kill_process+0x205/0x360
 [ 1098.946435]  [] ? security_capable_noaudit+0x15/0x20
 [ 1098.953733]  [] out_of_memory+0x492/0x4d0
 [ 1098.959962]  [] __alloc_pages_nodemask+0xa00/0xb60
 [ 1098.967067]  [] alloc_pages_vma+0x9a/0x160
 [ 1098.973393]  [] handle_mm_fault+0xd38/0x1080
 [ 1098.979914]  [] ? change_protection+0x594/0x850
 [ 1098.986726]  [] __do_page_fault+0x19e/0x560
 [ 1098.993149]  [] ? mprotect_fixup+0x151/0x290
 [ 1098.999669]  [] do_page_fault+0x31/0x70
 [ 1099.005704]  [] page_fault+0x28/0x30
 
 Madhu
>>> 
>> 

Re: Out of memory crash of 3000 routes in BGP

[Jonathan Stewart]

It is not a general issue, many here run hundreds of thousands of routes.

It must be something specific to your environment.

Jonathan

On 10 Oct 2016 4:40 p.m., "Madhu"  wrote:

> Looks like the problem is still there. I used 1.6.0 and also 1.6.2.
>
>
>
> On Mon, Oct 10, 2016 at 10:56 AM, james machado 
> wrote:
>
>> Madhu check out the following thread.  http://bird.network.cz/piperm
>> ail/bird-users/2016-September/010578.html
>>
>>
>>
>> On Mon, Oct 10, 2016 at 10:02 AM, Madhu  wrote:
>>
>>> Hi,
>>>
>>>   I am seeing a out of memory crash with 3000 routes in BGP. Is this a
>>> known issue? I m seeing this issue in 1.6.0 and 1.6.2 .
>>>
>>>
>>> [ 1098.912753] Call Trace:
>>> [ 1098.915481]  [] dump_stack+0x64/0x82
>>> [ 1098.921225]  [] dump_header+0x7f/0x1f1
>>> [ 1098.927163]  [] ? put_online_cpus+0x56/0x80
>>> [ 1098.933587]  [] ? rcu_oom_notify+0xcc/0xf0
>>> [ 1098.939914]  [] oom_kill_process+0x205/0x360
>>> [ 1098.946435]  [] ? security_capable_noaudit+0x15/
>>> 0x20
>>> [ 1098.953733]  [] out_of_memory+0x492/0x4d0
>>> [ 1098.959962]  [] __alloc_pages_nodemask+0xa00/0xb60
>>> [ 1098.967067]  [] alloc_pages_vma+0x9a/0x160
>>> [ 1098.973393]  [] handle_mm_fault+0xd38/0x1080
>>> [ 1098.979914]  [] ? change_protection+0x594/0x850
>>> [ 1098.986726]  [] __do_page_fault+0x19e/0x560
>>> [ 1098.993149]  [] ? mprotect_fixup+0x151/0x290
>>> [ 1098.999669]  [] do_page_fault+0x31/0x70
>>> [ 1099.005704]  [] page_fault+0x28/0x30
>>>
>>> Madhu
>>>
>>
>>
>