Re: Automatic ABI checks for new package updates in bodhi using taskotron

[Dodji Seketeli]

Kamil Paral  a écrit:

> Thanks, I reported an RFE about this:
> https://phab.qadevel.cloud.fedoraproject.org/T811

Thanks Kamil,

Unfortunately, I still cannot log into
https://phab.qadevel.cloud.fedoraproject.org/T811 using my
do...@fedoraproject.org persona, so I cannot comment on the issue.  When
I try to log in there, I am getting this error:

[HTTP/500] Internal Server Error Internal Server Error

But I can log into the Fedora infrastracture just fine, in general.
It's just the https://phab.qadevel.cloud.fedoraproject.org/T811 thingy
that I am having issue with.  I am not sure what I am doing wrong. 

I wish all the tickets handling gets migrated to pagure.io one day :-)

Cheers,

-- 
Dodji
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/qa-devel@lists.fedoraproject.org

Re: Automatic ABI checks for new package updates in bodhi using taskotron

[Dodji Seketeli]

Hello,

Sorry for the late reply.

Kamil Paral  a écrit:

>> So, let's say we initially start with packages available in
>> critpath[3], then we don't have to worry whether a package provides
>> any shared library or not. 
>
> Let me try this another way. Let's assume we can detect whether a
> certain package contains an *.so (or *.so.N+) file. Can we use this to
> decide which packages to run libabigail on?

If by "we" you mean taskotron, then the answer to your question is yes.

> Or does libabigail run checks on more files than just *.so (now that
> we're using --dso-only)?

Now that the taskotron task is invoking abipkgdiff with the --dso-only
option, if the package contains shared libraries and and other types of
binaries, then only the shared libraries are going to be ABI-compared.

> I'm asking, because I expect this request ("run my task on packages
> containing shared libraries/files or certain other kind") is going to
> be quite common in the future and I think we should cover it somehow
> (and I have an idea how). I understand that currently we can run
> libabigail on everything, and if there are no libraries inside,
> nothing bad happens. But it's not very efficient, and that's why I'm
> interested to learn how exactly to distinguish packages libabigail is
> useful for from packages libabigail just skips.

I understand.  I agree that being able to skip packages early based on
their content would likely be more efficent.

I hope this helps.

Cheers,

-- 
Dodji
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
https://lists.fedoraproject.org/admin/lists/qa-devel@lists.fedoraproject.org

Re: Automatic ABI checks for new package updates in bodhi using taskotron

[Kamil Paral]

> > Speaking of lists, you and Tim were mentioning white/blacklists, also
> > critpath set. So what is the ideal set of packages we should run on
> > initially? All packages? Only critpath packages? Only libraries included
> > in critpath? In case we should run just on libraries, any good
> > recommendation how to recognize that (better than matching "lib*" in
> > package name)? We would need to make the decision without downloading the
> > package, but I guess we could query koji or repo metadata if necessary.
> 
> With latest commit [1] in libabigail taskotron task, ABI comparison
> will occur only on shared libraries because we call abipkgdiff with
> --dso-only option. So, if a package doesn't contains any library file,
> it won't run abi checks on files available in package.There is a
> related bug[2] which has already been fixed and will be available in
> next libabigail release. So, there is no need to look specifically for
> lib* package. Also, packages providing libraries may not start/contain
> *lib* (e.g. elfutils) in its name.

I don't see any libraries in elfutils, they are in elfutils-libs, byt yes, 
you're right, we can't rely on package names.

> 
> So, let's say we initially start with packages available in
> critpath[3], then we don't have to worry whether a package provides
> any shared library or not. 

Let me try this another way. Let's assume we can detect whether a certain 
package contains an *.so (or *.so.N+) file. Can we use this to decide which 
packages to run libabigail on? Or does libabigail run checks on more files than 
just *.so (now that we're using --dso-only)?

I'm asking, because I expect this request ("run my task on packages containing 
shared libraries/files or certain other kind") is going to be quite common in 
the future and I think we should cover it somehow (and I have an idea how). I 
understand that currently we can run libabigail on everything, and if there are 
no libraries inside, nothing bad happens. But it's not very efficient, and 
that's why I'm interested to learn how exactly to distinguish packages 
libabigail is useful for from packages libabigail just skips.

Thanks.
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/qa-devel@lists.fedoraproject.org

Re: Automatic ABI checks for new package updates in bodhi using taskotron

[Kamil Paral]

> Hello Kamil,
> 
> Sorry for replying late.  I subscribed to this list, but for a reason the
> emails are still not being delivered to me.  I clicked on some more buttons
> in the interface right now, so we'll see.  In the mean time, I am using the
> web interface to reply, so please forgive the awkward formatting that might
> come out of this.

Hello Dodji, welcome.

> I think the test systems must have that kind of amount of memory.  For most
> packages, the memory consumption is reasonable.  But for some packages where
> the combination of ELF binary size and uncompressed debug info size is big,
> then, well, the tool is going to require a lot of memory.  For instance, it
> takes 13+GB of memory to compare the ABI of two instances of the libjvm.so
> library.

I've read the backlog of your discussion with tflink, but I'm not sure what the 
conclusion is. I believe we can't assign 15GB RAM to all our test VMs (and we 
can't tie a specific task to a specific set of VMs at the moment). So I guess 
we will increase our test VMs memory to some "reasonable" amount and let the 
few extremely large packages crash with OOM. (That reminds me we should make 
sure to disable swap in VMs, otherwise that would kill the host). Do you have 
any advice what this "reasonable" amount of RAM should be? 4 GB? 6 GB? So that 
95% of tasks work OK, and just the extreme ones crash. We can then add those to 
a blacklist.

Speaking of lists, you and Tim were mentioning white/blacklists, also critpath 
set. So what is the ideal set of packages we should run on initially? All 
packages? Only critpath packages? Only libraries included in critpath? In case 
we should run just on libraries, any good recommendation how to recognize that 
(better than matching "lib*" in package name)? We would need to make the 
decision without downloading the package, but I guess we could query koji or 
repo metadata if necessary.

We will need to implement white/blacklisting, ideally I'd like you to have the 
control over it, not us (so e.g. defining that in task.yaml formula). We can 
probably implement critpath support (recognizing what is in critpath and what 
is not), I'm sure it will be needed in other tasks in the future as well.

Kamil
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/qa-devel@lists.fedoraproject.org

Re: Automatic ABI checks for new package updates in bodhi using taskotron

[Kamil Paral]

> Thanks for the quick review. I have addressed review comments and added new
> diff at https://phab.qadevel.cloud.fedoraproject.org/D817?id=2081 . 

Thanks, looks good. I have no further concerns regarding the task code.

> I am not
> sure if I updated diff in right way but this is what I ended up by following
> option "Update diff" available in right side :)

Whatever works. Phabricator is not completely suited for reviewing patches from 
unknown projects, so it can be a bit cumbersome. But I needed a place where I 
can add comments to any line of the script (I think it's not possible to do on 
github). If we need to review any further changes, we can use Phab or github, 
doesn't matter.
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/qa-devel@lists.fedoraproject.org

Re: Automatic ABI checks for new package updates in bodhi using taskotron

[Dodji Seketeli]

Hello Kamil,

Sorry for replying late.  I subscribed to this list, but for a reason the 
emails are still not being delivered to me.  I clicked on some more buttons in 
the interface right now, so we'll see.  In the mean time, I am using the web 
interface to reply, so please forgive the awkward formatting that might come 
out of this.

> One additional thing that I noticed today - when I ran the abipkgdiff 
> comparison on lyx
> package, it used up to 6 GB RAM. That is really high and I believe our test 
> clients
> don't have that many available.

Ah.

I think the test systems must have that kind of amount of memory.  For most 
packages, the memory consumption is reasonable.  But for some packages where 
the combination of ELF binary size and uncompressed debug info size is big, 
then, well, the tool is going to require a lot of memory.  For instance, it 
takes 13+GB of memory to compare the ABI of two instances of the libjvm.so 
library.

Really, abipkgdiff uses the usual "static analysis" techniques (it's only that 
it acts on binaries rather than source code), so it is expected to have the 
same kind of memory requirements.

> I don't know how abipkgdiff works and whether it
> is able to adjust its memory requirements based on the amount of available 
> system memory
> (so that it would use less memory on a less equipped system and still work 
> properly). 

abipkgdiff performs static analysis of the binary.  For a given library, it 
loads the type information of each one of the compilation units that make up 
the final library.  There can be hundreds of thousands of those types in big 
libraries.  abipkgdiff has to represent them all to analyse them. It needs to 
have them all in memory because it analyses the relationships between all the 
types.  And unlike a compiler that generally sees only one translation unit at 
a time, abipkgdiff sees the type information of all the translation units at a 
time.  Just like what a debugger has to do.  And abipkgdiff has to do this 
*twice*, because it needs to load two libraries to compare the types of the 
functions and variables in that library.  So yes, that can take a lot of 
memory.  We are working on optimizations, of course, but the reality is that 
there are going to be binaries for which we'll need a lot of memory to operate 
on.  And right now, we cannot do this incrementally.

> Do you know if this is going to be a problem or not? 

If the machine doesn't have enough memory, then yes it's going to be a problem. 
 Our testing showed that in practice, for the vast majority of packages, the 
memory consumption is OK.  But there are going to be some packages that will 
require much much more memory.

> And do you know what is the peak memory
> usage for very big packages (e.g. what about comparing two kernel packages)? 

As Sinny said in another thread, abipkgdiff doesn't support the kernel yet.  So 
we haven't tested that.

> Is there any solution that we could use to limit the memory consumption of 
> abipkgdiff?

In general, I'd say there is no good solution beside us optimizing the code 
more to make it consume less over time.  And this is something I am really 
interested in doing.  I have started already and I intend to keep doing that as 
part of the normal development process.

As Sinny pointed out, in the particular case of the lyx package, the tool 
should *not* have performed any comparison because that package only contains 
an executable.  The tool should only act on libraries, at least in a first 
instalment.  The reason why it tried to compare two lyx executable is because 
it mistakenly considered that executable as a shared library.  This mistake can 
happen on Position Independent Executables, due to a bug in Libabigail.  This 
bug is fixed now, though.  So on lyx, the memory consumption should be very 
minimal now, as abipkgdiff won't do anything ;-)

I hope this helps.
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/qa-devel@lists.fedoraproject.org

Re: Automatic ABI checks for new package updates in bodhi using taskotron

[Sinny Kumari]

Hi Kamil,


On Thu, Apr 14, 2016 at 8:40 PM, Kamil Paral  wrote:

> Hi Sinny,
>
> I have created a review here:
> https://phab.qadevel.cloud.fedoraproject.org/D813
>
> I used our Phabricator, because it makes it easy to comment on particular
> lines. You can log in using your FAS email [1] and reply there, or here in
> the mailing list, it doesn't matter. I'm pleasantly surprised that despite
> our very lacking documentation, you managed to create a very good Taskotron
> task :-)
>

Thanks for the quick review. I have addressed review comments and added new
diff at https://phab.qadevel.cloud.fedoraproject.org/D817?id=2081 . I am
not sure if I  updated diff in right way but this is what I ended up by
following option "Update diff" available in right side :)


> One additional thing that I noticed today - when I ran the abipkgdiff
> comparison on lyx package, it used up to 6 GB RAM. That is really high and
> I believe our test clients don't have that many available. I don't know how
> abipkgdiff works and whether it is able to adjust its memory requirements
> based on the amount of available system memory (so that it would use less
> memory on a less equipped system and still work properly). Do you know if
> this is going to be a problem or not? And do you know what is the peak
> memory usage for very big packages (e.g. what about comparing two kernel
> packages)? Is there any solution that we could use to limit the memory
> consumption of abipkgdiff?
>

I had a talk about this with libabigail's maintainer (Dodji). If a binary
is taking more RAM then right now, we can't do anything about it. This is
because ELF information for binaries to be compared and corresponding
debuginfo gets loaded while performing ABI comparison.  Also, right now
libabigail doesn't support comparison of two kernel version.

I noticed that binary files available in lyx package are PIE (Position
Independent Executable) files which are basically executable files. As we
know, an application mainly consumes shared libraries from its dependent
packages. ABI change of shared libraries are what really matter in
packages. Therefore, I am now calling  abipkgdiff  with --dso-only  option.
This will avoid unnecessary comparison between executable files. This
reduces overall comparison time of a koji build update and hopefully
reduces memory consumption since most of shared libraries are smaller in
size (few MBs).

Right now there is a bug in libabigail that it doesn't distinguish between
PIE files and shared libraries since both are of type DSO(Dynamic Shared
object). So, right now abipkgdiff with --dso-only option does ABI
comparison for PIE files as well. Upstream bug has been reported to
distinguish between Shared libraries and PIE files
https://sourceware.org/bugzilla/show_bug.cgi?id=19961. This will be fixed
soon and will be available in next libabigail release.

Thanks,
Sinny
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/qa-devel@lists.fedoraproject.org

Re: Automatic ABI checks for new package updates in bodhi using taskotron

[Kamil Paral]

Hi Sinny,

I have created a review here:
https://phab.qadevel.cloud.fedoraproject.org/D813

I used our Phabricator, because it makes it easy to comment on particular 
lines. You can log in using your FAS email [1] and reply there, or here in the 
mailing list, it doesn't matter. I'm pleasantly surprised that despite our very 
lacking documentation, you managed to create a very good Taskotron task :-)

One additional thing that I noticed today - when I ran the abipkgdiff 
comparison on lyx package, it used up to 6 GB RAM. That is really high and I 
believe our test clients don't have that many available. I don't know how 
abipkgdiff works and whether it is able to adjust its memory requirements based 
on the amount of available system memory (so that it would use less memory on a 
less equipped system and still work properly). Do you know if this is going to 
be a problem or not? And do you know what is the peak memory usage for very big 
packages (e.g. what about comparing two kernel packages)? Is there any solution 
that we could use to limit the memory consumption of abipkgdiff? 

Thanks,
Kamil


[1] Please note we've been experiencing some problems with authentication 
recently, so it might not work for you:
https://phab.qadevel.cloud.fedoraproject.org/T761
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/qa-devel@lists.fedoraproject.org

Re: Automatic ABI checks for new package updates in bodhi using taskotron

[Tim Flink]

On Wed, 13 Apr 2016 21:14:19 +0530
Sinny Kumari  wrote:

Thank you for getting this done, I'm excited to start it running in
Taskotron.



> You need to decide whether it is better to run libabigain against
> every new
> > Koji build, or just against every new Bodhi update. From a quick
> > look, I think it makes more sense to run libabigail on every new
> > Koji build, so that people can see the results even before creating
> > the update (that requires looking into ResultsDB manually at the
> > moment). If we run it on every Koji build, the results will still
> > show up in Bodhi - Bodhi should query ResultsDB and show the
> > results for those particular builds present in the update. (We
> > might need to teach Bodhi about libabigail existence, I'm not
> > sure). Ultimately it's your choice, what makes more sense for your
> > check. 
> 
> I believe that when we say every new Koji build, we are talking about
> non-scratch build which doesn't include scratch build done by anyone.
> If my assumption is right, then yes running libabigail task on each
> koji build will be good. It is possible to do that with current
> implementation since libabigail task look for a koji build-id to
> download corresponding rpms.

I'd advocate for running libabigail against koji builds, if possible.
There are no updates in rawhide and I think it'd be useful to have abi
checks running on rawhide builds.



> > We try to have at least some basic documentation and FAQ for our
> > checks in there. Currently it's not very discoverable (we should
> > link to it at least from ResultsDB, which we currently don't) and
> > the location can change, but at least it's a link we can give to
> > people when asking basic questions about one of our tasks. Also,
> > since you're going to maintain the task and not us, please include
> > some "Contact" section where to post feedback or report bugs (e.g.
> > github issues page). If people ask us about the task and we don't
> > know the answer, we will point them to that wiki page. 
> 
> Will add contact section in wiki page.
> I wonder if it is better to have this included with other fedora qa
> tasks?

By "have this included wiht other fedora qa tasks", do you mean have
the git repo hosted in the same place?

If so, I'd say that's up to you but I expect to be moving all our repos
off of bitbucket and into something hosted in Fedora in the next month
or so. If you do decide that you want the task hosted that way, I'd
suggest waiting until we move all of our current repos so that there's
only one change.

If I misunderstood, would you rephrase the question?

Again, thanks for getting this done.

Tim



pgpWMX2pBC9l5.pgp
Description: OpenPGP digital signature
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/qa-devel@lists.fedoraproject.org

Re: Automatic ABI checks for new package updates in bodhi using taskotron

[Kamil Paral]

> Hi all,

> Last year, discussion happened around "Checking the ABI of packages submitted
> to the updates-testing Fedora repository" on Fedora devel ML [1].

> We felt that taskotron[2] will be the best place to run automatic ABI checks
> for a new package update pushed in bodhi for testing against latest stable
> package. If any ABI incompatibility is detected, we will provide package
> maintainer feedback to review ABI changes, similar to how rpmlint feedback
> is given. This will help to ship stable ABI for Fedora packages, reduced
> build failures which is seen later in packages depending on it.

> I have created a libabigail [3] taskotron task which can be integrated in
> taskotron to perform automatic ABI checks for new bodhi updates. Right now,
> this task downloads all rpms with debuginfo for specified package update and
> corresponding latest stable package available for that release. Further, it
> runs libabigail's abipkgdiff tool for ABI checks. It tells PASSED if no ABI
> change occurred, otherwise FAILED and ABI changes log can be reviewed to
> ensure changes are ok to go or not.

> Source code of libabigail taskotron task can be accessed from github[4].
> Some sample output of libabigail task run on my local machine:
> * http://fpaste.org/349740/
> * http://fpaste.org/349741/
> * http://fpaste.org/349761/

> It will be great if someone can review libabigail task and provide feedback.
> Also, I would love to know further steps required to integrate it with
> taskotron?

> Thanks,
> Sinny

> [1]
> https://lists.fedoraproject.org/archives/list/de...@lists.fedoraproject.org/thread/BBWQWPSYA66F2T4VVOLY3BGWW43W6K6C/#TDUMSBDHQDW6A4WBTRFYYH7WMQGNWLAZ
> [2] http://libtaskotron.readthedocs.org/en/develop/
> [3] https://sourceware.org/libabigail/
> [4] https://github.com/sinnykumari/task-libabigail


Hello Sinny, 
thanks for your work and sorry for late response. I'll review your taskotron 
task and let you know if there's something that should be changed or not. 
Afterwards, we will start mirroring your git repo on our taskotron servers, and 
patch our taskotron-trigger to know about task-libabigail. We can then execute 
it on every new Koji build (or Bodhi update, your choice). Your task will 
probably be the first one that we will execute regularly while not being 
written and maintained directly by us, so if there are any rough edges in the 
process, I apologize in advance :-)

You'll need to have two branches in your git:
master - this will be used on our production server 
https://taskotron.fedoraproject.org/
develop - this will be used on our dev and staging server 
http://taskotron-dev.fedoraproject.org/ and 
https://taskotron.stg.fedoraproject.org/

We will regularly pull your repo, probably with a cron job for now. The cron 
job is not yet written and the periodicity is not yet decided, but you can 
track it here:
https://phab.qadevel.cloud.fedoraproject.org/T767

You need to decide whether it is better to run libabigain against every new 
Koji build, or just against every new Bodhi update. From a quick look, I think 
it makes more sense to run libabigail on every new Koji build, so that people 
can see the results even before creating the update (that requires looking into 
ResultsDB manually at the moment). If we run it on every Koji build, the 
results will still show up in Bodhi - Bodhi should query ResultsDB and show the 
results for those particular builds present in the update. (We might need to 
teach Bodhi about libabigail existence, I'm not sure). Ultimately it's your 
choice, what makes more sense for your check.

Please also create a wiki page at 
https://fedoraproject.org/wiki/Taskotron/Tasks/libabigail similar to these
https://fedoraproject.org/wiki/Taskotron/Tasks/depcheck
https://fedoraproject.org/wiki/Taskotron/Tasks/upgradepath
linked from https://fedoraproject.org/wiki/Taskotron/Tasks .
We try to have at least some basic documentation and FAQ for our checks in 
there. Currently it's not very discoverable (we should link to it at least from 
ResultsDB, which we currently don't) and the location can change, but at least 
it's a link we can give to people when asking basic questions about one of our 
tasks. Also, since you're going to maintain the task and not us, please include 
some "Contact" section where to post feedback or report bugs (e.g. github 
issues page). If people ask us about the task and we don't know the answer, we 
will point them to that wiki page.

Can we please continue this discussion in qa-devel [1] mailing list? We can 
discuss more implementation details in there, and I'll post my review findings 
in there as well.

Thanks,
Kamil

[1] 
https://lists.fedoraproject.org/archives/list/qa-devel@lists.fedoraproject.org/
___
qa-devel mailing list
qa-devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/qa-devel@lists.fedoraproject.org