[lng-odp] api-next branch force updated

[Maxim Uvarov]

Hello,

api-next branch is forced update to current master. Please audjust that in
your PR. Backup is here:
https://github.com/muvarov/odp/tree/api-next_1.18_backup

Thank you,
Maxim.

Re: [lng-odp] API-next branch

[Dmitry Eremin-Solenikov]

On 26/10/17 09:57, Savolainen, Petri (Nokia - FI/Espoo) wrote:
> We need one branch that holds all new API changes before those are released 
> (> 4 months currently): either it's master or api-next. If we change API 
> development to master, then applications would need follow the latest release 
> branch instead of master. At least previous there was resistance to change 
> master into API development branch.

Yes, I understand that. That is why I proposed to have api-next being
regularly rebased on top of master. Doing that monthly (or after
release) should be enough. This way we will see all pending patches easily.

> Separate topic branches for API changes won't scale ... we would not have a 
> common view (e.g. for 4 months) what the API is before all those are merged 
> together. Implementation changes are easier in topic branches as there's less 
> dependency to other files and applications (other files and apps already 
> agree what the API is).

Topic branches would would allow easier tracking of patches pending into
master. So we will know that ``these branches were merged into api-next
during this development cycle but were not merged into master, so we
should consider them''.

> Api-next does not cause the problem (big delta), it's caused by the lack of 
> steady release cycle. The big  delta won't go away before we have a short 
> enough release cycle (merge often => small delta).
> 
> -Petri
> 
> 
> 
>> -Original Message-
>> From: lng-odp [mailto:lng-odp-boun...@lists.linaro.org] On Behalf Of Bill
>> Fischofer
>> Sent: Wednesday, October 25, 2017 6:53 PM
>> To: Dmitry Eremin-Solenikov 
>> Cc: lng-odp-forward 
>> Subject: Re: [lng-odp] API-next branch
>>
>> I'm all for using topic branches, especially since we've switched to
>> GitHub
>> and most contributors are now familiar with it and using pull requests
>> rather than raw patches sent to the mailing list. The whole reason for
>> api-next was to separate in-progress API changes from regular maintenance
>> patches since they were all mixed together on the mailing list. The PR
>> structure is much cleaner in that regard.
>>
>> IPsec in particular clearly could have been a separate branch. We've
>> talked
>> about doing this in the context of 2.0 as well since that's also going to
>> involve some major subsystems that could benefit from collaborative
>> development before being merged back into the 2.0 main development branch.
>>
>>
>> On Wed, Oct 25, 2017 at 10:39 AM, Dmitry Eremin-Solenikov <
>> dmitry.ereminsoleni...@linaro.org> wrote:
>>
>>> Hello,
>>>
>>> I tried to actually check, which patches are sitting in the api-next.
>>> And actually I failed
>>> to do that in a timely manner. git cherry produces a list of patches,
>>> that contains a lot of patches, which already landed to the master.
>>>
>>> Quick proposal would be to stop using api-next as a long-lived branch
>>> which received updates from master and rather use it as a branch being
>>> regularly rebased on top of current master.
>>>
>>> Another possiblity would be to abandon api-next completely, develop
>>> features on topic branches, which get merged to master, rather than to
>>> api-next. At least this would save us from situations, when there is
>>> API definition (or change), but no actual implementation behind.
>>>
>>> --
>>> With best wishes
>>> Dmitry
>>>


-- 
With best wishes
Dmitry

Re: [lng-odp] API-next branch

[Maxim Uvarov]

On 10/26/17 12:32, Savolainen, Petri (Nokia - FI/Espoo) wrote:
> Hi,
> 
> It's not so trivial to draw a line what is a small change. E.g. one word 
> change in documentation may be a big change for an application: "ODP 
> maintains packet order during operation X" vs "... does not maintain order 
> ..."
> 
> So, it's much simpler if all API changes go through the same branch, and that 
> branch is released often.
> 
> One of the largest motivation to change to github was to enable easy (and 
> fast) release cycle, since all commits are always check automatically in 
> Travis. So, there should be now less work to make a release. 
> 
> Tomorrow is the last Friday of the month and a perfect day for making a 
> release (1.16) of everything that is ready in api-next... Maybe it's only 
> couple of new APIs and some typo corrections, but it still makes the delta 
> that much smaller.
> 

I pushed master to next. So we can stage ready patches there. If
somebody wants to help, please send pool requests.

Maxim.


> -Petri
> 
> 
> From: Maxim Uvarov [mailto:maxim.uva...@linaro.org] 
> Sent: Thursday, October 26, 2017 10:30 AM
> To: Savolainen, Petri (Nokia - FI/Espoo) 
> Cc: Bill Fischofer ; Dmitry Eremin-Solenikov 
> ; lng-odp-forward 
> 
> Subject: Re: [lng-odp] API-next branch
> 
> 1. we have api-next branch to collect all api changes with implementation in 
> tests in one places. A lot of people said that it's useful.
> 2. Yes it's hard to find patches which are in api-next and not in master.
> 3.  New API acceptance period is not very clean.  If we can improve that than 
> probably we will resolve merge issue.
> 
> I.e. my proposal is  - if feature is complete (api, impl, tests) and  
> accepted (members can implement it)  all patches for this feature should be 
> moved to master branch. No need to wait for release date.
> From other point there is no need for small api changes (like debug prints, 
> pktio capabilities, small improvements) stage in api-next. They can go 
> directly to master.
> The other deal is complex things which need several months for review.  This 
> should go to staging branch api-next .
> 
> To help with back port we can use github issues. Where one entry will be one 
> feature. And place in the comments all commit id's related to this feature. 
> It's also possible to do some automation for that.
> 
> How about that?
> 
> Maxim.
> 
> 
> On 26 October 2017 at 09:57, Savolainen, Petri (Nokia - FI/Espoo) 
> <mailto:petri.savolai...@nokia.com> wrote:
> We need one branch that holds all new API changes before those are released 
> (> 4 months currently): either it's master or api-next. If we change API 
> development to master, then applications would need follow the latest release 
> branch instead of master. At least previous there was resistance to change 
> master into API development branch.
> 
> Separate topic branches for API changes won't scale ... we would not have a 
> common view (e.g. for 4 months) what the API is before all those are merged 
> together. Implementation changes are easier in topic branches as there's less 
> dependency to other files and applications (other files and apps already 
> agree what the API is).
> 
> Api-next does not cause the problem (big delta), it's caused by the lack of 
> steady release cycle. The big  delta won't go away before we have a short 
> enough release cycle (merge often => small delta).
> 
> -Petri
> 
> 
> 
>> -----Original Message-
>> From: lng-odp [mailto:mailto:lng-odp-boun...@lists.linaro.org] On Behalf Of 
>> Bill
>> Fischofer
>> Sent: Wednesday, October 25, 2017 6:53 PM
>> To: Dmitry Eremin-Solenikov <mailto:dmitry.ereminsoleni...@linaro.org>
>> Cc: lng-odp-forward <mailto:lng-odp@lists.linaro.org>
>> Subject: Re: [lng-odp] API-next branch
>>
>> I'm all for using topic branches, especially since we've switched to
>> GitHub
>> and most contributors are now familiar with it and using pull requests
>> rather than raw patches sent to the mailing list. The whole reason for
>> api-next was to separate in-progress API changes from regular maintenance
>> patches since they were all mixed together on the mailing list. The PR
>> structure is much cleaner in that regard.
>>
>> IPsec in particular clearly could have been a separate branch. We've
>> talked
>> about doing this in the context of 2.0 as well since that's also going to
>> involve some major subsystems that could benefit from collaborative
>> development before being merged back into the 2.0 mai

Re: [lng-odp] API-next branch

[Savolainen, Petri (Nokia - FI/Espoo)]

Hi,

It's not so trivial to draw a line what is a small change. E.g. one word change 
in documentation may be a big change for an application: "ODP maintains packet 
order during operation X" vs "... does not maintain order ..."

So, it's much simpler if all API changes go through the same branch, and that 
branch is released often.

One of the largest motivation to change to github was to enable easy (and fast) 
release cycle, since all commits are always check automatically in Travis. So, 
there should be now less work to make a release. 

Tomorrow is the last Friday of the month and a perfect day for making a release 
(1.16) of everything that is ready in api-next... Maybe it's only couple of new 
APIs and some typo corrections, but it still makes the delta that much smaller.

-Petri


From: Maxim Uvarov [mailto:maxim.uva...@linaro.org] 
Sent: Thursday, October 26, 2017 10:30 AM
To: Savolainen, Petri (Nokia - FI/Espoo) 
Cc: Bill Fischofer ; Dmitry Eremin-Solenikov 
; lng-odp-forward 
Subject: Re: [lng-odp] API-next branch

1. we have api-next branch to collect all api changes with implementation in 
tests in one places. A lot of people said that it's useful.
2. Yes it's hard to find patches which are in api-next and not in master.
3.  New API acceptance period is not very clean.  If we can improve that than 
probably we will resolve merge issue.

I.e. my proposal is  - if feature is complete (api, impl, tests) and  accepted 
(members can implement it)  all patches for this feature should be moved to 
master branch. No need to wait for release date.
From other point there is no need for small api changes (like debug prints, 
pktio capabilities, small improvements) stage in api-next. They can go directly 
to master.
The other deal is complex things which need several months for review.  This 
should go to staging branch api-next .

To help with back port we can use github issues. Where one entry will be one 
feature. And place in the comments all commit id's related to this feature. 
It's also possible to do some automation for that.

How about that?

Maxim.


On 26 October 2017 at 09:57, Savolainen, Petri (Nokia - FI/Espoo) 
<mailto:petri.savolai...@nokia.com> wrote:
We need one branch that holds all new API changes before those are released (> 
4 months currently): either it's master or api-next. If we change API 
development to master, then applications would need follow the latest release 
branch instead of master. At least previous there was resistance to change 
master into API development branch.

Separate topic branches for API changes won't scale ... we would not have a 
common view (e.g. for 4 months) what the API is before all those are merged 
together. Implementation changes are easier in topic branches as there's less 
dependency to other files and applications (other files and apps already agree 
what the API is).

Api-next does not cause the problem (big delta), it's caused by the lack of 
steady release cycle. The big  delta won't go away before we have a short 
enough release cycle (merge often => small delta).

-Petri



> -Original Message-
> From: lng-odp [mailto:mailto:lng-odp-boun...@lists.linaro.org] On Behalf Of 
> Bill
> Fischofer
> Sent: Wednesday, October 25, 2017 6:53 PM
> To: Dmitry Eremin-Solenikov <mailto:dmitry.ereminsoleni...@linaro.org>
> Cc: lng-odp-forward <mailto:lng-odp@lists.linaro.org>
> Subject: Re: [lng-odp] API-next branch
>
> I'm all for using topic branches, especially since we've switched to
> GitHub
> and most contributors are now familiar with it and using pull requests
> rather than raw patches sent to the mailing list. The whole reason for
> api-next was to separate in-progress API changes from regular maintenance
> patches since they were all mixed together on the mailing list. The PR
> structure is much cleaner in that regard.
>
> IPsec in particular clearly could have been a separate branch. We've
> talked
> about doing this in the context of 2.0 as well since that's also going to
> involve some major subsystems that could benefit from collaborative
> development before being merged back into the 2.0 main development branch.
>
>
> On Wed, Oct 25, 2017 at 10:39 AM, Dmitry Eremin-Solenikov <
> mailto:dmitry.ereminsoleni...@linaro.org> wrote:
>
> > Hello,
> >
> > I tried to actually check, which patches are sitting in the api-next.
> > And actually I failed
> > to do that in a timely manner. git cherry produces a list of patches,
> > that contains a lot of patches, which already landed to the master.
> >
> > Quick proposal would be to stop using api-next as a long-lived branch
> > which received updates from master and rather use it as a branch being
> > regularly rebased on top o

Re: [lng-odp] API-next branch

[Maxim Uvarov]

1. we have api-next branch to collect all api changes with implementation
in tests in one places. A lot of people said that it's useful.

2. Yes it's hard to find patches which are in api-next and not in master.

3.  New API acceptance period is not very clean.  If we can improve that
than probably we will resolve merge issue.

I.e. my proposal is  - if feature is complete (api, impl, tests) and
accepted (members can implement it)  all patches for this feature should be
moved to master branch. No need to wait for release date.

>From other point there is no need for small api changes (like debug prints,
pktio capabilities, small improvements) stage in api-next. They can go
directly to master.

The other deal is complex things which need several months for review.
This should go to staging branch api-next .

To help with back port we can use github issues. Where one entry will be
one feature. And place in the comments all commit id's related to this
feature. It's also possible to do some automation for that.

How about that?

Maxim.


On 26 October 2017 at 09:57, Savolainen, Petri (Nokia - FI/Espoo) <
petri.savolai...@nokia.com> wrote:

> We need one branch that holds all new API changes before those are
> released (> 4 months currently): either it's master or api-next. If we
> change API development to master, then applications would need follow the
> latest release branch instead of master. At least previous there was
> resistance to change master into API development branch.
>
> Separate topic branches for API changes won't scale ... we would not have
> a common view (e.g. for 4 months) what the API is before all those are
> merged together. Implementation changes are easier in topic branches as
> there's less dependency to other files and applications (other files and
> apps already agree what the API is).
>
> Api-next does not cause the problem (big delta), it's caused by the lack
> of steady release cycle. The big  delta won't go away before we have a
> short enough release cycle (merge often => small delta).
>
> -Petri
>
>
>
> > -Original Message-
> > From: lng-odp [mailto:lng-odp-boun...@lists.linaro.org] On Behalf Of
> Bill
> > Fischofer
> > Sent: Wednesday, October 25, 2017 6:53 PM
> > To: Dmitry Eremin-Solenikov 
> > Cc: lng-odp-forward 
> > Subject: Re: [lng-odp] API-next branch
> >
> > I'm all for using topic branches, especially since we've switched to
> > GitHub
> > and most contributors are now familiar with it and using pull requests
> > rather than raw patches sent to the mailing list. The whole reason for
> > api-next was to separate in-progress API changes from regular maintenance
> > patches since they were all mixed together on the mailing list. The PR
> > structure is much cleaner in that regard.
> >
> > IPsec in particular clearly could have been a separate branch. We've
> > talked
> > about doing this in the context of 2.0 as well since that's also going to
> > involve some major subsystems that could benefit from collaborative
> > development before being merged back into the 2.0 main development
> branch.
> >
> >
> > On Wed, Oct 25, 2017 at 10:39 AM, Dmitry Eremin-Solenikov <
> > dmitry.ereminsoleni...@linaro.org> wrote:
> >
> > > Hello,
> > >
> > > I tried to actually check, which patches are sitting in the api-next.
> > > And actually I failed
> > > to do that in a timely manner. git cherry produces a list of patches,
> > > that contains a lot of patches, which already landed to the master.
> > >
> > > Quick proposal would be to stop using api-next as a long-lived branch
> > > which received updates from master and rather use it as a branch being
> > > regularly rebased on top of current master.
> > >
> > > Another possiblity would be to abandon api-next completely, develop
> > > features on topic branches, which get merged to master, rather than to
> > > api-next. At least this would save us from situations, when there is
> > > API definition (or change), but no actual implementation behind.
> > >
> > > --
> > > With best wishes
> > > Dmitry
> > >
>

Re: [lng-odp] API-next branch

[Savolainen, Petri (Nokia - FI/Espoo)]

We need one branch that holds all new API changes before those are released (> 
4 months currently): either it's master or api-next. If we change API 
development to master, then applications would need follow the latest release 
branch instead of master. At least previous there was resistance to change 
master into API development branch.

Separate topic branches for API changes won't scale ... we would not have a 
common view (e.g. for 4 months) what the API is before all those are merged 
together. Implementation changes are easier in topic branches as there's less 
dependency to other files and applications (other files and apps already agree 
what the API is).

Api-next does not cause the problem (big delta), it's caused by the lack of 
steady release cycle. The big  delta won't go away before we have a short 
enough release cycle (merge often => small delta).

-Petri



> -Original Message-
> From: lng-odp [mailto:lng-odp-boun...@lists.linaro.org] On Behalf Of Bill
> Fischofer
> Sent: Wednesday, October 25, 2017 6:53 PM
> To: Dmitry Eremin-Solenikov 
> Cc: lng-odp-forward 
> Subject: Re: [lng-odp] API-next branch
> 
> I'm all for using topic branches, especially since we've switched to
> GitHub
> and most contributors are now familiar with it and using pull requests
> rather than raw patches sent to the mailing list. The whole reason for
> api-next was to separate in-progress API changes from regular maintenance
> patches since they were all mixed together on the mailing list. The PR
> structure is much cleaner in that regard.
> 
> IPsec in particular clearly could have been a separate branch. We've
> talked
> about doing this in the context of 2.0 as well since that's also going to
> involve some major subsystems that could benefit from collaborative
> development before being merged back into the 2.0 main development branch.
> 
> 
> On Wed, Oct 25, 2017 at 10:39 AM, Dmitry Eremin-Solenikov <
> dmitry.ereminsoleni...@linaro.org> wrote:
> 
> > Hello,
> >
> > I tried to actually check, which patches are sitting in the api-next.
> > And actually I failed
> > to do that in a timely manner. git cherry produces a list of patches,
> > that contains a lot of patches, which already landed to the master.
> >
> > Quick proposal would be to stop using api-next as a long-lived branch
> > which received updates from master and rather use it as a branch being
> > regularly rebased on top of current master.
> >
> > Another possiblity would be to abandon api-next completely, develop
> > features on topic branches, which get merged to master, rather than to
> > api-next. At least this would save us from situations, when there is
> > API definition (or change), but no actual implementation behind.
> >
> > --
> > With best wishes
> > Dmitry
> >

Re: [lng-odp] API-next branch

[Bill Fischofer]

I'm all for using topic branches, especially since we've switched to GitHub
and most contributors are now familiar with it and using pull requests
rather than raw patches sent to the mailing list. The whole reason for
api-next was to separate in-progress API changes from regular maintenance
patches since they were all mixed together on the mailing list. The PR
structure is much cleaner in that regard.

IPsec in particular clearly could have been a separate branch. We've talked
about doing this in the context of 2.0 as well since that's also going to
involve some major subsystems that could benefit from collaborative
development before being merged back into the 2.0 main development branch.


On Wed, Oct 25, 2017 at 10:39 AM, Dmitry Eremin-Solenikov <
dmitry.ereminsoleni...@linaro.org> wrote:

> Hello,
>
> I tried to actually check, which patches are sitting in the api-next.
> And actually I failed
> to do that in a timely manner. git cherry produces a list of patches,
> that contains a lot of patches, which already landed to the master.
>
> Quick proposal would be to stop using api-next as a long-lived branch
> which received updates from master and rather use it as a branch being
> regularly rebased on top of current master.
>
> Another possiblity would be to abandon api-next completely, develop
> features on topic branches, which get merged to master, rather than to
> api-next. At least this would save us from situations, when there is
> API definition (or change), but no actual implementation behind.
>
> --
> With best wishes
> Dmitry
>

[lng-odp] API-next branch

[Dmitry Eremin-Solenikov]

Hello,

I tried to actually check, which patches are sitting in the api-next.
And actually I failed
to do that in a timely manner. git cherry produces a list of patches,
that contains a lot of patches, which already landed to the master.

Quick proposal would be to stop using api-next as a long-lived branch
which received updates from master and rather use it as a branch being
regularly rebased on top of current master.

Another possiblity would be to abandon api-next completely, develop
features on topic branches, which get merged to master, rather than to
api-next. At least this would save us from situations, when there is
API definition (or change), but no actual implementation behind.

-- 
With best wishes
Dmitry

Re: [lng-odp] api-next branch broken?

[Maxim Uvarov]

On 05/20/17 21:59, Bill Fischofer wrote:
> 
> On Sat, May 20, 2017 at 1:38 PM, Maxim Uvarov  > wrote:
> 
> On 05/20/17 01:07, Bill Fischofer wrote:
> > Something is clearly going on as I'm seeing the pktio/pktio_ipc_run.sh
> > script suddenly start failing for completely unrelated changes.
> >
> 
> 
> Can you reproduce it manually? I can not reproduce it locally
> and here looks like it passed for the latest code (if not take in
> account that check patch failed):
> 
> https://github.com/Linaro/odp/commits/api-next
> 
> 
> 
> Yes, I can. I've been rebasing my packet reference patch which I'm
> trying to put into a pull request. You can see the failure in my local
> GitHub repo
> at https://travis-ci.org/Bill-Fischofer-Linaro/odp/jobs/234169324
> 
> I can reproduce the failure on my local Linux system as well, but I
> don't understand the output. This is the output I get after I apply part
> 9 of my patch (things work normally for the patches before that):
> 
>  run pktio_ipc1 then pktio_ipc2 
> _ishm.c:866:_odp_ishm_reserve():No huge pages, fall back to normal
> pages. check: /proc/sys/vm/nr_hugepages.
> _ishm.c:866:_odp_ishm_reserve():No huge pages, fall back to normal
> pages. check: /proc/sys/vm/nr_hugepages.
>  PKTIO: initialized loop interface.
>  PKTIO: initialized pcap interface.
>  PKTIO: initialized ipc interface.
>  PKTIO: initialized socket mmap, use export
> ODP_PKTIO_DISABLE_SOCKET_MMAP=1 to disable.
>  PKTIO: initialized socket mmsg,use export
> ODP_PKTIO_DISABLE_SOCKET_MMSG=1 to disable.
> 
> ODP system info
> ---
> ODP API version: 1.14.0
> CPU model:   Intel(R) Core(TM) i7-4790K CPU
> 
> Running ODP appl: "pktio_ipc1"
> -
> Using IF:(null)
> 
> 
> pktio_ipc1.c:188:pktio_run_loop():head.seq 0 - cnt_recv 1 =
> 18446744073709551615
> pktio_ipc2.c:107:ipc_second_process():exit after 5 seconds
>  PKTIO: initialized loop interface.
>  PKTIO: initialized pcap interface.
>  PKTIO: initialized ipc interface.
>  PKTIO: initialized socket mmap, use export
> ODP_PKTIO_DISABLE_SOCKET_MMAP=1 to disable.
>  PKTIO: initialized socket mmsg,use export
> ODP_PKTIO_DISABLE_SOCKET_MMSG=1 to disable.
> pid: 13402, create IPC pktio ipc:13401:ipktio
> normal exit of 2 application

that means that second application was exited normally with time-out
value set from command line.

> srwxr-xr-x 1 bill bill0 May 19 20:00 /tmp/odp-13401-fdserver
> -rw-r--r-- 1 bill bill 4096 May 19 20:00
> /tmp/odp-13401-ishm-ipc:ipktio_info
> -rw-r--r-- 1 bill bill36864 May 19 20:00
> /tmp/odp-13401-ishm-ipc:ipktio_m_cons
> -rw-r--r-- 1 bill bill36864 May 19 20:00
> /tmp/odp-13401-ishm-ipc:ipktio_m_prod
> -rw-r--r-- 1 bill bill36864 May 19 20:00
> /tmp/odp-13401-ishm-ipc:ipktio_s_cons
> -rw-r--r-- 1 bill bill36864 May 19 20:00
> /tmp/odp-13401-ishm-ipc:ipktio_s_prod
> -rw-r--r-- 1 bill bill 71827456 May 19 20:00
> /tmp/odp-13401-ishm-ipc_packet_pool
> -rw-r--r-- 1 bill bill  168 May 19 20:00
> /tmp/odp-13401-shm-ipc:ipktio_info
> -rw-r--r-- 1 bill bill  176 May 19 20:00
> /tmp/odp-13401-shm-ipc:ipktio_m_cons
> -rw-r--r-- 1 bill bill  176 May 19 20:00
> /tmp/odp-13401-shm-ipc:ipktio_m_prod
> -rw-r--r-- 1 bill bill  176 May 19 20:00
> /tmp/odp-13401-shm-ipc:ipktio_s_cons
> -rw-r--r-- 1 bill bill  176 May 19 20:00
> /tmp/odp-13401-shm-ipc:ipktio_s_prod
> -rw-r--r-- 1 bill bill  177 May 19 20:00
> /tmp/odp-13401-shm-ipc_packet_pool
> !!!First stage  FAILED 255!!!

That means that first application does not did termination functions
without errors. I.e. pool destroy (odp-13401-shm-ipc_packet_pool)  and
shm destroy (odp-13401-shm-...) did not destroy resources. More likely
pool still has packet which was not freed.

>From email log I do not see that counters for packet exchange between
apps is increased. That is strange.

> 
> Any suggestions for how to debug this? The patch that causes the failure
> is some restructuring of the odp_packet_free() and
> odp_packet_free_multi() routines to deal with packet references, so I
> don't see why pktio_ipc should be affected by that (or by the changes
> you posted either) but obviously there is some sensitivity that's going on.
>  

Run manually the same thing as shell scripts does from 2 consoles. Turn
on debug prints:

./platform/linux-generic/pktio/ipc.c:#define IPC_ODP_DEBUG_PRINT 1

Check why pool was not destroyed correctly.

In multi process scenario if one pool is used by 2 applications you
never know when second application is still alive and can process
packets from that pool. If you play with references it might be a case
where first application waits packets to be freed by second application
but because it quit it never happens. So pool destroy might fail due to
increasing packet reference or something like that.

Maxim.

> 
> 
> 
> 
> > On Fri, May 19, 2017 at 5:02 PM, Dm

Re: [lng-odp] api-next branch broken?

[Bill Fischofer]

On Sat, May 20, 2017 at 1:38 PM, Maxim Uvarov 
wrote:

> On 05/20/17 01:07, Bill Fischofer wrote:
> > Something is clearly going on as I'm seeing the pktio/pktio_ipc_run.sh
> > script suddenly start failing for completely unrelated changes.
> >
>
>
> Can you reproduce it manually? I can not reproduce it locally
> and here looks like it passed for the latest code (if not take in
> account that check patch failed):
>
> https://github.com/Linaro/odp/commits/api-next


Yes, I can. I've been rebasing my packet reference patch which I'm trying
to put into a pull request. You can see the failure in my local GitHub repo
at https://travis-ci.org/Bill-Fischofer-Linaro/odp/jobs/234169324

I can reproduce the failure on my local Linux system as well, but I don't
understand the output. This is the output I get after I apply part 9 of my
patch (things work normally for the patches before that):

 run pktio_ipc1 then pktio_ipc2 
_ishm.c:866:_odp_ishm_reserve():No huge pages, fall back to normal pages.
check: /proc/sys/vm/nr_hugepages.
_ishm.c:866:_odp_ishm_reserve():No huge pages, fall back to normal pages.
check: /proc/sys/vm/nr_hugepages.
 PKTIO: initialized loop interface.
 PKTIO: initialized pcap interface.
 PKTIO: initialized ipc interface.
 PKTIO: initialized socket mmap, use export ODP_PKTIO_DISABLE_SOCKET_MMAP=1
to disable.
 PKTIO: initialized socket mmsg,use export ODP_PKTIO_DISABLE_SOCKET_MMSG=1
to disable.

ODP system info
---
ODP API version: 1.14.0
CPU model:   Intel(R) Core(TM) i7-4790K CPU

Running ODP appl: "pktio_ipc1"
-
Using IF:(null)


pktio_ipc1.c:188:pktio_run_loop():head.seq 0 - cnt_recv 1 =
18446744073709551615
pktio_ipc2.c:107:ipc_second_process():exit after 5 seconds
 PKTIO: initialized loop interface.
 PKTIO: initialized pcap interface.
 PKTIO: initialized ipc interface.
 PKTIO: initialized socket mmap, use export ODP_PKTIO_DISABLE_SOCKET_MMAP=1
to disable.
 PKTIO: initialized socket mmsg,use export ODP_PKTIO_DISABLE_SOCKET_MMSG=1
to disable.
pid: 13402, create IPC pktio ipc:13401:ipktio
normal exit of 2 application
srwxr-xr-x 1 bill bill0 May 19 20:00 /tmp/odp-13401-fdserver
-rw-r--r-- 1 bill bill 4096 May 19 20:00
/tmp/odp-13401-ishm-ipc:ipktio_info
-rw-r--r-- 1 bill bill36864 May 19 20:00
/tmp/odp-13401-ishm-ipc:ipktio_m_cons
-rw-r--r-- 1 bill bill36864 May 19 20:00
/tmp/odp-13401-ishm-ipc:ipktio_m_prod
-rw-r--r-- 1 bill bill36864 May 19 20:00
/tmp/odp-13401-ishm-ipc:ipktio_s_cons
-rw-r--r-- 1 bill bill36864 May 19 20:00
/tmp/odp-13401-ishm-ipc:ipktio_s_prod
-rw-r--r-- 1 bill bill 71827456 May 19 20:00
/tmp/odp-13401-ishm-ipc_packet_pool
-rw-r--r-- 1 bill bill  168 May 19 20:00
/tmp/odp-13401-shm-ipc:ipktio_info
-rw-r--r-- 1 bill bill  176 May 19 20:00
/tmp/odp-13401-shm-ipc:ipktio_m_cons
-rw-r--r-- 1 bill bill  176 May 19 20:00
/tmp/odp-13401-shm-ipc:ipktio_m_prod
-rw-r--r-- 1 bill bill  176 May 19 20:00
/tmp/odp-13401-shm-ipc:ipktio_s_cons
-rw-r--r-- 1 bill bill  176 May 19 20:00
/tmp/odp-13401-shm-ipc:ipktio_s_prod
-rw-r--r-- 1 bill bill  177 May 19 20:00
/tmp/odp-13401-shm-ipc_packet_pool
!!!First stage  FAILED 255!!!

Any suggestions for how to debug this? The patch that causes the failure is
some restructuring of the odp_packet_free() and odp_packet_free_multi()
routines to deal with packet references, so I don't see why pktio_ipc
should be affected by that (or by the changes you posted either) but
obviously there is some sensitivity that's going on.


>
>
>
> > On Fri, May 19, 2017 at 5:02 PM, Dmitry Eremin-Solenikov <
> > dmitry.ereminsoleni...@linaro.org> wrote:
> >
> >> Since one of last updates I'm receiving timeouts from Travis CI because
> >> some tests run silently taking too much time to complete.
> >>
>
> Dmitry, I saw that with your patch for recursive atomic changes. But I
> think it's not applied.
>
> Maxim.
>
>
> >> --
> >> With best wishes
> >> Dmitry
> >>
>
>

Re: [lng-odp] api-next branch broken?

[Maxim Uvarov]

On 05/20/17 01:07, Bill Fischofer wrote:
> Something is clearly going on as I'm seeing the pktio/pktio_ipc_run.sh
> script suddenly start failing for completely unrelated changes.
> 


Can you reproduce it manually? I can not reproduce it locally
and here looks like it passed for the latest code (if not take in
account that check patch failed):

https://github.com/Linaro/odp/commits/api-next


> On Fri, May 19, 2017 at 5:02 PM, Dmitry Eremin-Solenikov <
> dmitry.ereminsoleni...@linaro.org> wrote:
> 
>> Since one of last updates I'm receiving timeouts from Travis CI because
>> some tests run silently taking too much time to complete.
>>

Dmitry, I saw that with your patch for recursive atomic changes. But I
think it's not applied.

Maxim.


>> --
>> With best wishes
>> Dmitry
>>

Re: [lng-odp] api-next branch broken?

[Bill Fischofer]

Something is clearly going on as I'm seeing the pktio/pktio_ipc_run.sh
script suddenly start failing for completely unrelated changes.

On Fri, May 19, 2017 at 5:02 PM, Dmitry Eremin-Solenikov <
dmitry.ereminsoleni...@linaro.org> wrote:

> Since one of last updates I'm receiving timeouts from Travis CI because
> some tests run silently taking too much time to complete.
>
> --
> With best wishes
> Dmitry
>

[lng-odp] api-next branch broken?

[Dmitry Eremin-Solenikov]

Since one of last updates I'm receiving timeouts from Travis CI because
some tests run silently taking too much time to complete.

-- 
With best wishes
Dmitry

Re: [lng-odp] API-NEXT branch is broken

[Bill Fischofer]

On Wed, Jan 13, 2016 at 2:35 PM, Mike Holmes  wrote:

>
>
> On 13 January 2016 at 15:33, Bill Fischofer 
> wrote:
>
>> If we can eliminate a dependency that seems like another reason to use
>> the .svgs directly.  I really don't see the need for PDF support since few
>> are going to be printing manuals these days.
>>
>
> Agree, I am hoping no one disagrees and there will be patches :)
>
>

Your wish is my command.  Patch submitted against API-NEXT.  Please
review.  Everything builds fine for me.


>
>> On Wed, Jan 13, 2016 at 2:11 PM, Mike Holmes 
>> wrote:
>>
>>>
>>>
>>> On 13 January 2016 at 14:31, Maxim Uvarov 
>>> wrote:
>>>
 Probably we need to add convert (imagemagic package?) check to
 ./configure.

>>>
>>> It is there, what has changed is that it apparently gets required even
>>> if you dont build docs.
>>> As we move to direct use of .svg I can delete all that assuming we dont
>>> want to build .pdf which appears to be true.
>>>
>>>

 Maxim.

 On 01/13/2016 19:01, Mike Holmes wrote:

> Have you installed the DEPENDENCIES for building the docs ?
>
> Did you just make or "make distcheck"
>
> On 13 January 2016 at 10:58, Bala Manoharan  > wrote:
>
> Hi,
>
> The api-next branch is broken after merge from master.
> convert atomic_queue.svg atomic_queue.png
> /bin/bash: convert: command not found
> make[2]: *** [atomic_queue.png] Error 127
>
> Regards,
> Bala
> ___
> lng-odp mailing list
> lng-odp@lists.linaro.org 
> https://lists.linaro.org/mailman/listinfo/lng-odp
>
>
>
>
> --
> Mike Holmes
> Technical Manager - Linaro Networking Group
> Linaro.org ***│ *Open source software for ARM
> SoCs
>
>
>
> ___
> lng-odp mailing list
> lng-odp@lists.linaro.org
> https://lists.linaro.org/mailman/listinfo/lng-odp
>

 ___
 lng-odp mailing list
 lng-odp@lists.linaro.org
 https://lists.linaro.org/mailman/listinfo/lng-odp

>>>
>>>
>>>
>>> --
>>> Mike Holmes
>>> Technical Manager - Linaro Networking Group
>>> Linaro.org  *│ *Open source software for ARM
>>> SoCs
>>>
>>>
>>>
>>> ___
>>> lng-odp mailing list
>>> lng-odp@lists.linaro.org
>>> https://lists.linaro.org/mailman/listinfo/lng-odp
>>>
>>>
>>
>
>
> --
> Mike Holmes
> Technical Manager - Linaro Networking Group
> Linaro.org  *│ *Open source software for ARM SoCs
>
>
>
___
lng-odp mailing list
lng-odp@lists.linaro.org
https://lists.linaro.org/mailman/listinfo/lng-odp

Re: [lng-odp] API-NEXT branch is broken

[Mike Holmes]

On 13 January 2016 at 15:33, Bill Fischofer 
wrote:

> If we can eliminate a dependency that seems like another reason to use the
> .svgs directly.  I really don't see the need for PDF support since few are
> going to be printing manuals these days.
>

Agree, I am hoping no one disagrees and there will be patches :)


>
> On Wed, Jan 13, 2016 at 2:11 PM, Mike Holmes 
> wrote:
>
>>
>>
>> On 13 January 2016 at 14:31, Maxim Uvarov 
>> wrote:
>>
>>> Probably we need to add convert (imagemagic package?) check to
>>> ./configure.
>>>
>>
>> It is there, what has changed is that it apparently gets required even if
>> you dont build docs.
>> As we move to direct use of .svg I can delete all that assuming we dont
>> want to build .pdf which appears to be true.
>>
>>
>>>
>>> Maxim.
>>>
>>> On 01/13/2016 19:01, Mike Holmes wrote:
>>>
 Have you installed the DEPENDENCIES for building the docs ?

 Did you just make or "make distcheck"

 On 13 January 2016 at 10:58, Bala Manoharan >>> > wrote:

 Hi,

 The api-next branch is broken after merge from master.
 convert atomic_queue.svg atomic_queue.png
 /bin/bash: convert: command not found
 make[2]: *** [atomic_queue.png] Error 127

 Regards,
 Bala
 ___
 lng-odp mailing list
 lng-odp@lists.linaro.org 
 https://lists.linaro.org/mailman/listinfo/lng-odp




 --
 Mike Holmes
 Technical Manager - Linaro Networking Group
 Linaro.org ***│ *Open source software for ARM
 SoCs



 ___
 lng-odp mailing list
 lng-odp@lists.linaro.org
 https://lists.linaro.org/mailman/listinfo/lng-odp

>>>
>>> ___
>>> lng-odp mailing list
>>> lng-odp@lists.linaro.org
>>> https://lists.linaro.org/mailman/listinfo/lng-odp
>>>
>>
>>
>>
>> --
>> Mike Holmes
>> Technical Manager - Linaro Networking Group
>> Linaro.org  *│ *Open source software for ARM SoCs
>>
>>
>>
>> ___
>> lng-odp mailing list
>> lng-odp@lists.linaro.org
>> https://lists.linaro.org/mailman/listinfo/lng-odp
>>
>>
>


-- 
Mike Holmes
Technical Manager - Linaro Networking Group
Linaro.org  *│ *Open source software for ARM SoCs
___
lng-odp mailing list
lng-odp@lists.linaro.org
https://lists.linaro.org/mailman/listinfo/lng-odp

Re: [lng-odp] API-NEXT branch is broken

[Bill Fischofer]

If we can eliminate a dependency that seems like another reason to use the
.svgs directly.  I really don't see the need for PDF support since few are
going to be printing manuals these days.

On Wed, Jan 13, 2016 at 2:11 PM, Mike Holmes  wrote:

>
>
> On 13 January 2016 at 14:31, Maxim Uvarov  wrote:
>
>> Probably we need to add convert (imagemagic package?) check to
>> ./configure.
>>
>
> It is there, what has changed is that it apparently gets required even if
> you dont build docs.
> As we move to direct use of .svg I can delete all that assuming we dont
> want to build .pdf which appears to be true.
>
>
>>
>> Maxim.
>>
>> On 01/13/2016 19:01, Mike Holmes wrote:
>>
>>> Have you installed the DEPENDENCIES for building the docs ?
>>>
>>> Did you just make or "make distcheck"
>>>
>>> On 13 January 2016 at 10:58, Bala Manoharan >> > wrote:
>>>
>>> Hi,
>>>
>>> The api-next branch is broken after merge from master.
>>> convert atomic_queue.svg atomic_queue.png
>>> /bin/bash: convert: command not found
>>> make[2]: *** [atomic_queue.png] Error 127
>>>
>>> Regards,
>>> Bala
>>> ___
>>> lng-odp mailing list
>>> lng-odp@lists.linaro.org 
>>> https://lists.linaro.org/mailman/listinfo/lng-odp
>>>
>>>
>>>
>>>
>>> --
>>> Mike Holmes
>>> Technical Manager - Linaro Networking Group
>>> Linaro.org ***│ *Open source software for ARM
>>> SoCs
>>>
>>>
>>>
>>> ___
>>> lng-odp mailing list
>>> lng-odp@lists.linaro.org
>>> https://lists.linaro.org/mailman/listinfo/lng-odp
>>>
>>
>> ___
>> lng-odp mailing list
>> lng-odp@lists.linaro.org
>> https://lists.linaro.org/mailman/listinfo/lng-odp
>>
>
>
>
> --
> Mike Holmes
> Technical Manager - Linaro Networking Group
> Linaro.org  *│ *Open source software for ARM SoCs
>
>
>
> ___
> lng-odp mailing list
> lng-odp@lists.linaro.org
> https://lists.linaro.org/mailman/listinfo/lng-odp
>
>
___
lng-odp mailing list
lng-odp@lists.linaro.org
https://lists.linaro.org/mailman/listinfo/lng-odp

Re: [lng-odp] API-NEXT branch is broken

[Mike Holmes]

On 13 January 2016 at 14:31, Maxim Uvarov  wrote:

> Probably we need to add convert (imagemagic package?) check to ./configure.
>

It is there, what has changed is that it apparently gets required even if
you dont build docs.
As we move to direct use of .svg I can delete all that assuming we dont
want to build .pdf which appears to be true.


>
> Maxim.
>
> On 01/13/2016 19:01, Mike Holmes wrote:
>
>> Have you installed the DEPENDENCIES for building the docs ?
>>
>> Did you just make or "make distcheck"
>>
>> On 13 January 2016 at 10:58, Bala Manoharan > > wrote:
>>
>> Hi,
>>
>> The api-next branch is broken after merge from master.
>> convert atomic_queue.svg atomic_queue.png
>> /bin/bash: convert: command not found
>> make[2]: *** [atomic_queue.png] Error 127
>>
>> Regards,
>> Bala
>> ___
>> lng-odp mailing list
>> lng-odp@lists.linaro.org 
>> https://lists.linaro.org/mailman/listinfo/lng-odp
>>
>>
>>
>>
>> --
>> Mike Holmes
>> Technical Manager - Linaro Networking Group
>> Linaro.org ***│ *Open source software for ARM
>> SoCs
>>
>>
>>
>> ___
>> lng-odp mailing list
>> lng-odp@lists.linaro.org
>> https://lists.linaro.org/mailman/listinfo/lng-odp
>>
>
> ___
> lng-odp mailing list
> lng-odp@lists.linaro.org
> https://lists.linaro.org/mailman/listinfo/lng-odp
>



-- 
Mike Holmes
Technical Manager - Linaro Networking Group
Linaro.org  *│ *Open source software for ARM SoCs
___
lng-odp mailing list
lng-odp@lists.linaro.org
https://lists.linaro.org/mailman/listinfo/lng-odp

Re: [lng-odp] API-NEXT branch is broken

[Maxim Uvarov]

Probably we need to add convert (imagemagic package?) check to ./configure.

Maxim.

On 01/13/2016 19:01, Mike Holmes wrote:

Have you installed the DEPENDENCIES for building the docs ?

Did you just make or "make distcheck"

On 13 January 2016 at 10:58, Bala Manoharan > wrote:


Hi,

The api-next branch is broken after merge from master.
convert atomic_queue.svg atomic_queue.png
/bin/bash: convert: command not found
make[2]: *** [atomic_queue.png] Error 127

Regards,
Bala
___
lng-odp mailing list
lng-odp@lists.linaro.org 
https://lists.linaro.org/mailman/listinfo/lng-odp




--
Mike Holmes
Technical Manager - Linaro Networking Group
Linaro.org ***│ *Open source software for ARM SoCs



___
lng-odp mailing list
lng-odp@lists.linaro.org
https://lists.linaro.org/mailman/listinfo/lng-odp


___
lng-odp mailing list
lng-odp@lists.linaro.org
https://lists.linaro.org/mailman/listinfo/lng-odp

Re: [lng-odp] API-NEXT branch is broken

[Mike Holmes]

Have you installed the DEPENDENCIES for building the docs ?

Did you just make or "make distcheck"

On 13 January 2016 at 10:58, Bala Manoharan 
wrote:

> Hi,
>
> The api-next branch is broken after merge from master.
> convert atomic_queue.svg atomic_queue.png
> /bin/bash: convert: command not found
> make[2]: *** [atomic_queue.png] Error 127
>
> Regards,
> Bala
> ___
> lng-odp mailing list
> lng-odp@lists.linaro.org
> https://lists.linaro.org/mailman/listinfo/lng-odp
>



-- 
Mike Holmes
Technical Manager - Linaro Networking Group
Linaro.org  *│ *Open source software for ARM SoCs
___
lng-odp mailing list
lng-odp@lists.linaro.org
https://lists.linaro.org/mailman/listinfo/lng-odp

[lng-odp] API-NEXT branch is broken

[Bala Manoharan]

Hi,

The api-next branch is broken after merge from master.
convert atomic_queue.svg atomic_queue.png
/bin/bash: convert: command not found
make[2]: *** [atomic_queue.png] Error 127

Regards,
Bala
___
lng-odp mailing list
lng-odp@lists.linaro.org
https://lists.linaro.org/mailman/listinfo/lng-odp