Re: [DDEB] Status on automatic debug packages (2015-08-24)

[Tomasz Rybak]

Dnia 2015-09-20, nie o godzinie 19:38 +0200, Niels Thykier pisze:
[ cut ]
> > I added DH_BUILD_DDEBS = 1 to d/rules, changed dh_strip to use
> > --ddeb-migration and not -dbg-package, and removed *-dbg from
> > d/control.
> > After building I got python-pyopencl{3}-dbgsym*.dbg.
> > Those packages contained only /usr/lib/debug/.build-id/*.debug
> > files,
> > without /usr/lib/python*/dist-packages/*.so.
> > 
> 
> Indeed, as I concluded above in reply to Jean-Michel, this is the
> intentional behaviour - at least for now.
> 
> > Is there something missing?
> > Files *.so with debugging symbols are built (there are lines
> > PYBUILD_DESTDIR_python2-dbg=debian/python-pyopencl-dbg/ in d/rules)
> > so IMO it's (just?) case of adding those to *-dggsym.deb files.
> > 
> > Best regards.
> > 
> 
> I am open to adding those at a later stage.  Though at this point, I
> would really like to get ddebs into unstable, before we scope creep
> it
> any further.

Thanks for explanation.
For now I'll leave manual debug packages for Python.
At the same time - current DH with pybuilder build all necessary files
so as soon as DDEBs take Python-specific files into consideration
I'm ready to switch.

Best regards.

-- 
Tomasz Rybak  GPG/PGP key ID: 2AD5 9860
Fingerprint A481 824E 7DD3 9C0E C40A  488E C654 FB33 2AD5 9860
http://member.acm.org/~tomaszrybak




signature.asc
Description: This is a digitally signed message part

Re: [DDEB] Status on automatic debug packages (2015-08-24)

[Tomasz Rybak]

Dnia 2015-08-25, wto o godzinie 13:41 +, Jean-Michel Vourgère
pisze:
> Hi
> 
> Nikolaus Rath wrote:
> > On Aug 24 2015, Sebastian Ramacher  wrote:
> > > What's the plan for python(3)-*-dbg packages that include both
> > > Python extensions
> > > built for the python-dbg interpreter and debug symbols? Should
> > > they also change
> > > their Section to something else?
> > 
> > 
> > .. and will they also be build automatically? Or rather, when
> > relying on
> > the automatic building, will they include the extension for the
> > debug
> > interpreter or just the debugging symbols for all extensions (built
> > for
> > debugging and regular interpreter)?
> 
> The question is also valid for python2:
> 
> When using dh --with python2 *and* build-depending on python-all-dbg,
> I'm getting [1]:
> * Some files like /usr/lib/debug/.build-id/*.debug
> * Some files like
> /usr/lib/python2.7/dist-packages/rrdtool.x86_64-linux-gnu_d.so
> 
> Am I correct in assuming that this need to be split?
> * Each arch:any binary package will get its own -dbgsym package, like
> python-rrdtool-dbgsym_1.5.4-6_amd64.deb,
> lua-rrd-dbgsym_1.5.4-6_amd64.deb, ...
> * The python debug $(arch)_d.so generated by dh_auto_build will need
> its
> own package. (See questions of Nikolaus and Sebastian).
> 
> The main question is whether or not these -dbgsym package is only of
> debug symbols.
> 

> 
> Assuming this is split, should one make a recommends: towards these
> non-main packages?
> 
> 
> Finally, if the current -dbg packages are moved out of main, either
> the
> buildd's will need to have them in their source list, or the section
> of
> the python tools to generate the debug _d.so thingies need to be
> changed.
> 
> [1] https://packages.debian.org/sid/amd64/rrdtool-dbg/filelist

Test on current sid, trying to PyOpenCL with pybuild and debug symbols.
Current version of package has *-dbg entries for both Python 2 and Python 3,
and thanks for pybuild there is no need for code dealing with those in 
d/control.

I added DH_BUILD_DDEBS = 1 to d/rules, changed dh_strip to use
--ddeb-migration and not -dbg-package, and removed *-dbg from d/control.
After building I got python-pyopencl{3}-dbgsym*.dbg.
Those packages contained only /usr/lib/debug/.build-id/*.debug files,
without /usr/lib/python*/dist-packages/*.so.

Is there something missing?
Files *.so with debugging symbols are built (there are lines
PYBUILD_DESTDIR_python2-dbg=debian/python-pyopencl-dbg/ in d/rules)
so IMO it's (just?) case of adding those to *-dggsym.deb files.

Best regards.

-- 
Tomasz Rybak  GPG/PGP key ID: 2AD5 9860
Fingerprint A481 824E 7DD3 9C0E C40A  488E C654 FB33 2AD5 9860
http://member.acm.org/~tomaszrybak




signature.asc
Description: This is a digitally signed message part

Re: [DDEB] Status on automatic debug packages (2015-08-24)

[Niels Thykier]

On 2015-09-20 19:04, Tomasz Rybak wrote:
> Dnia 2015-08-25, wto o godzinie 13:41 +, Jean-Michel Vourgère
> pisze:
>> Hi
>>[...]
>>
>> The question is also valid for python2:
>>
>> When using dh --with python2 *and* build-depending on python-all-dbg,
>> I'm getting [1]:
>> * Some files like /usr/lib/debug/.build-id/*.debug
>> * Some files like
>> /usr/lib/python2.7/dist-packages/rrdtool.x86_64-linux-gnu_d.so
>>
>> Am I correct in assuming that this need to be split?
>> * Each arch:any binary package will get its own -dbgsym package, like
>> python-rrdtool-dbgsym_1.5.4-6_amd64.deb,
>> lua-rrd-dbgsym_1.5.4-6_amd64.deb, ...
>> * The python debug $(arch)_d.so generated by dh_auto_build will need
>> its
>> own package. (See questions of Nikolaus and Sebastian).
>>
>> The main question is whether or not these -dbgsym package is only of
>> debug symbols.
>>

The current plan is for debhelper to only deal with the first type of
debug symbols (i.e. /usr/lib/debug/.build-id/*/*.debug).  I have not
looked at language specific debug files; we might do that later once the
original proposal is available in unstable.

> 
>>
>> Assuming this is split, should one make a recommends: towards these
>> non-main packages?
>>

FTR: It is not a given that you should split them.  If you have to
create a manual dbg package, you might as well include the normal debug
symbols.

>>
>> Finally, if the current -dbg packages are moved out of main, either
>> the
>> buildd's will need to have them in their source list, or the section
>> of
>> the python tools to generate the debug _d.so thingies need to be
>> changed.
>>
>> [1] https://packages.debian.org/sid/amd64/rrdtool-dbg/filelist

For reference, given the issues with moving regular -dbg packages out of
main (some of them are used as Build-Depends), manual -dbg packages will
stay in unstable.

This was concluded in a separate branch of this thread.

> 
> Test on current sid, trying to PyOpenCL with pybuild and debug symbols.
> Current version of package has *-dbg entries for both Python 2 and Python 3,
> and thanks for pybuild there is no need for code dealing with those in 
> d/control.
> 
> I added DH_BUILD_DDEBS = 1 to d/rules, changed dh_strip to use
> --ddeb-migration and not -dbg-package, and removed *-dbg from d/control.
> After building I got python-pyopencl{3}-dbgsym*.dbg.
> Those packages contained only /usr/lib/debug/.build-id/*.debug files,
> without /usr/lib/python*/dist-packages/*.so.
> 

Indeed, as I concluded above in reply to Jean-Michel, this is the
intentional behaviour - at least for now.

> Is there something missing?
> Files *.so with debugging symbols are built (there are lines
> PYBUILD_DESTDIR_python2-dbg=debian/python-pyopencl-dbg/ in d/rules)
> so IMO it's (just?) case of adding those to *-dggsym.deb files.
> 
> Best regards.
> 

I am open to adding those at a later stage.  Though at this point, I
would really like to get ddebs into unstable, before we scope creep it
any further.

Thanks,
~Niels





signature.asc
Description: OpenPGP digital signature

Re: [DDEB] Status on automatic debug packages (2015-08-24)

[Bastian Blank]

On Mon, Aug 24, 2015 at 11:12:41PM +0200, Niels Thykier wrote:
 * ddebs are Debian packages with the extension .deb that
   contain debugging symbols and are built implicitly.
   - A package foo_1.23.deb will receive a corresponding
 foo_1.23-dbgsym.ddeb package.

Are they named .deb or .ddeb?

One question remains: Can we move non-automatic build dbgsum packages[1]
to the special debug suite as well? Even after the recent change to not
automatically move them?

Thanks,
Bastian

[1]: linux for example needs to handle them specially, as they need to
contain unstripped binaries instead of detached debugging infos and
there is no support for build-ids.
-- 
Most legends have their basis in facts.
-- Kirk, And The Children Shall Lead, stardate 5029.5


signature.asc
Description: Digital signature

Re: [DDEB] Status on automatic debug packages (2015-08-24)

[Niels Thykier]

On 2015-08-27 14:57, Bastian Blank wrote:
 On Mon, Aug 24, 2015 at 11:12:41PM +0200, Niels Thykier wrote:
 * ddebs are Debian packages with the extension .deb that
   contain debugging symbols and are built implicitly.
   - A package foo_1.23.deb will receive a corresponding
 foo_1.23-dbgsym.deb package.
 
 Are they named .deb or .ddeb?
 

That would be .deb with a single D - sorry for the copy-waste mistake.
  I should probably start calling them dbgsym or adbg (for automatic
debug packages) instead to avoid confusion. :)

 One question remains: Can we move non-automatic build dbgsum packages[1]
 to the special debug suite as well? Even after the recent change to not
 automatically move them?
 
 Thanks,
 Bastian
 
 [1]: linux for example needs to handle them specially, as they need to
 contain unstripped binaries instead of detached debugging infos and
 there is no support for build-ids.
 

Technically, yes.  I also /suspect/ that the FTP masters would agree
with it (but I cannot speak on their behalf).


On the technical side, it is will be a simple matter of adding a field
with a given value in your -dbg package.

Thanks,
~Niels





signature.asc
Description: OpenPGP digital signature

Re: [DDEB] Status on automatic debug packages (2015-08-24)

[Paul Tagliamonte]

On Mon, Aug 24, 2015 at 02:28:05PM -0700, Steve Langasek wrote:
 I wonder how this list was arrived at.  Offhand, I see the libc6-dbg and
 python3.5-dbg packages are both in section 'debug', both of which are part
 of the build-dependency closure of main; I'm pretty sure we don't want them
 shunted to a separate archive.

Really good point; no one thought of this while we were sprinting on it.
Right, so, two things are now in.

nthykier has checked in a changeset to add a binary control header to
signal it's an autocreated debug package, and dak now uses that to check
to see if it should be diverted.

This means we *wont* see a great pickup in space by moving those super
ultra huge debs (like webkit's debug package) into the other mirror, but
I'm sure we can work something out with the maintainers.

Anyway, thanks for that. Patches made against dak and just getting some
testing locally now.

   Paul

-- 
 .''`.   Paul Tagliamonte paul...@debian.org|   Proud Debian Developer
: :'  :  https://people.debian.org/~paultag   |   https://pault.ag/
`. `'`   Debian - the Universal Operating System
 `-4096R / 8F04 9AD8 2C92 066C 7352  D28A 7B58 5B30 807C 2A87


signature.asc
Description: Digital signature

Re: [DDEB] Status on automatic debug packages (2015-08-24)

[Matthias Klose]

On 08/24/2015 11:12 PM, Niels Thykier wrote:
  * debhelper will be including debug-ids in the control file to make it
easier to find the necessary debug symbols for a given file.
- Thanks to paultag for this idea.
- This is merged into git and will be included in the next upload.

are these debug-ids the same as the build-id emitted by GCC? If yes, please
could you consider enabling these build-id's independent of the debhelper compat
level? I see no reason why these should be only enabled by compat 9.

Matthias

Re: [DDEB] Status on automatic debug packages (2015-08-24)

[Niels Thykier]

On 2015-08-25 11:29, Matthias Klose wrote:
 On 08/24/2015 11:12 PM, Niels Thykier wrote:
  * debhelper will be including debug-ids in the control file to make it
easier to find the necessary debug symbols for a given file.
- Thanks to paultag for this idea.
- This is merged into git and will be included in the next upload.
 
 are these debug-ids the same as the build-id emitted by GCC? If yes, please
 could you consider enabling these build-id's independent of the debhelper 
 compat
 level? I see no reason why these should be only enabled by compat 9.
 
 Matthias
 

Hi,

Indeed, they are the same as emitted by GCC and they will be added
regardless of compat level (like ddebs itself).

~Niels





signature.asc
Description: OpenPGP digital signature

Re: [DDEB] Status on automatic debug packages (2015-08-24)

[Jean-Michel Vourgère]

Hi

Nikolaus Rath wrote:
 On Aug 24 2015, Sebastian Ramacher sramac...@debian.org wrote:
 What's the plan for python(3)-*-dbg packages that include both Python 
 extensions
 built for the python-dbg interpreter and debug symbols? Should they also 
 change
 their Section to something else?
 
 
 .. and will they also be build automatically? Or rather, when relying on
 the automatic building, will they include the extension for the debug
 interpreter or just the debugging symbols for all extensions (built for
 debugging and regular interpreter)?

The question is also valid for python2:

When using dh --with python2 *and* build-depending on python-all-dbg,
I'm getting [1]:
* Some files like /usr/lib/debug/.build-id/*.debug
* Some files like
/usr/lib/python2.7/dist-packages/rrdtool.x86_64-linux-gnu_d.so

Am I correct in assuming that this need to be split?
* Each arch:any binary package will get its own -dbgsym package, like
python-rrdtool-dbgsym_1.5.4-6_amd64.deb,
lua-rrd-dbgsym_1.5.4-6_amd64.deb, ...
* The python debug $(arch)_d.so generated by dh_auto_build will need its
own package. (See questions of Nikolaus and Sebastian).

The main question is whether or not these -dbgsym package is only of
debug symbols.


Assuming this is split, should one make a recommends: towards these
non-main packages?


Finally, if the current -dbg packages are moved out of main, either the
buildd's will need to have them in their source list, or the section of
the python tools to generate the debug _d.so thingies need to be changed.

[1] https://packages.debian.org/sid/amd64/rrdtool-dbg/filelist
-- 
Nirgal

Re: [DDEB] Status on automatic debug packages (2015-08-24)

[Niels Thykier]

On 2015-08-24 23:28, Steve Langasek wrote:
 Hi Niels,
 
 On Mon, Aug 24, 2015 at 11:12:41PM +0200, Niels Thykier wrote:
 [...]
 
 I wonder how this list was arrived at.  Offhand, I see the libc6-dbg and
 python3.5-dbg packages are both in section 'debug', both of which are part
 of the build-dependency closure of main; I'm pretty sure we don't want them
 shunted to a separate archive.
 
 Analysis of Build-Dependencies[1] also shows libpetsc3.4.2-dbg is affected,
 which wasn't in your list.
 
 Thanks,
 

I looked for packages in the debug section, which did not end in -dbg.
It did not occur to me that we were using -dbg packages as a part of the
build-dependency chains.  I suspect it did not occur to the FTP masters
when they wrote the DAK patch.

Thanks,
~Niels





signature.asc
Description: OpenPGP digital signature

Re: [DDEB] Status on automatic debug packages (2015-08-24)

[Steve Langasek]

Hi Niels,

On Mon, Aug 24, 2015 at 11:12:41PM +0200, Niels Thykier wrote:
  * Up to 5 packages need to change section (see Implementation-
details below)
- See #796834, #796836, #796839, #796840, and #796842.

snip

 Implementation-details
 ==
 
 There are some details behind the implementation that might be of
 general interest.
 
  * All packages in the debug section will be moved from unstable to
the new unstable-debug (separate mirror list)
- Including manual debug packages

I wonder how this list was arrived at.  Offhand, I see the libc6-dbg and
python3.5-dbg packages are both in section 'debug', both of which are part
of the build-dependency closure of main; I'm pretty sure we don't want them
shunted to a separate archive.

Analysis of Build-Dependencies[1] also shows libpetsc3.4.2-dbg is affected,
which wasn't in your list.

Thanks,
-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org

[1] grep-dctrl -sBuild-Depends -FBuild-Depends  -r '.*-dbg' \
/chroots/sid/var/lib/apt/lists/*Sources \
| grep -vE 'python3?(-all)?-dbg|libc[0-9.]+-dbg' | less


signature.asc
Description: Digital signature

[DDEB] Status on automatic debug packages (2015-08-24)

[Niels Thykier]

Hi,

Here is the post-debconf status on automatic debug packages (ddebs).
 The previous status from 2015-06-29 can be found in [0].

What is it?
===

* ddebs are Debian packages with the extension .deb that
  contain debugging symbols and are built implicitly.
  - A package foo_1.23.deb will receive a corresponding
foo_1.23-dbgsym.ddeb package.
  - ddebs are built automatically by dh_strip.

Where are we?
=

 * ddebs are now properly included in the .changes file
   - This is fixed with debhelper/9.20150811 and dpkg/1.18.2

 * dak has been patched and is mostly ready!
   - A huge thanks to paultag and ansgar for their work here

 * DSA and the FTP masters are setting up a mirror network for it.
   - Filed as RT#5913
   - (If you have no idea about what a mirror network is, then do not
  worry - I still do not understand it either)

 * debhelper will be including debug-ids in the control file to make it
   easier to find the necessary debug symbols for a given file.
   - Thanks to paultag for this idea.
   - This is merged into git and will be included in the next upload.

What needs to be done
=

 * We need to finish the mirror network setup
   - Thanks to weasel, we have a webserver now that is just missing an
 archive on it.
   - As I do not quite understand the concepts, I am not entirely sure
 if this item is completely done after the FTP masters sync out an
 archive.  But it should be sufficient for testing purposes.

 * FTP-masters have to write a final patch to have ddebs by-pass NEW.
   - This has been deliberately punted until the mirror network was in
 place.

 * Up to 5 packages need to change section (see Implementation-
   details below)
   - See #796834, #796836, #796839, #796840, and #796842.

 * debhelper needs an upload to build ddebs by default.

With these done, we should be ready to start upload ddebs to Debian.
There will be some post processing after this as well (e.g. adding them
to testing will require patching Britney).

Implementation-details
==

There are some details behind the implementation that might be of
general interest.

 * All packages in the debug section will be moved from unstable to
   the new unstable-debug (separate mirror list)
   - Including manual debug packages

 * This will reduce the size required to mirror unstable and also reduce
   the total number of binary packages in unstable

 * If you are a regular user of debug symbols, you will have to add the
   unstable-debug mirror to install them.

 * My understanding is that DAK does *not* use its overrides when
   checking whether a package in the debug section.

Thanks,
~Niels

[0] https://lists.debian.org/debian-devel/2015/06/msg00406.html



signature.asc
Description: OpenPGP digital signature

Re: [DDEB] Status on automatic debug packages (2015-08-24)

[Sebastian Ramacher]

Hi

On 2015-08-24 23:12:41, Niels Thykier wrote:
  * Up to 5 packages need to change section (see Implementation-
details below)
- See #796834, #796836, #796839, #796840, and #796842.

What's the plan for python(3)-*-dbg packages that include both Python extensions
built for the python-dbg interpreter and debug symbols? Should they also change
their Section to something else?

Cheers
-- 
Sebastian Ramacher


signature.asc
Description: Digital signature

Re: [DDEB] Status on automatic debug packages (2015-08-24)

[Nikolaus Rath]

On Aug 24 2015, Sebastian Ramacher sramac...@debian.org wrote:
 Hi

 On 2015-08-24 23:12:41, Niels Thykier wrote:
  * Up to 5 packages need to change section (see Implementation-
details below)
- See #796834, #796836, #796839, #796840, and #796842.

 What's the plan for python(3)-*-dbg packages that include both Python 
 extensions
 built for the python-dbg interpreter and debug symbols? Should they also 
 change
 their Section to something else?


.. and will they also be build automatically? Or rather, when relying on
the automatic building, will they include the extension for the debug
interpreter or just the debugging symbols for all extensions (built for
debugging and regular interpreter)?


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

 »Time flies like an arrow, fruit flies like a Banana.«


signature.asc
Description: PGP signature

Re: Automatic debug packages

[Carsten Hey]

* Emil Langrock [2011-03-08 00:48 +0100]:
 I browsed a little bit in the goals which were planned for squeeze and noticed
 that the debug packages aka ddebs[1] weren't implemented in the debian
 infrastructure.

A prerequisite to automatically add debug packages for all architectures
is to change the way how packages are uploaded and/or build[1].  In the
upcoming ftpmaster meeting[2] from the 21st until the 27th of March, one
possible way to change it (or rather a part of it) will be discussed:

| * Throwaway DD built .debs (well, let's have the fight^Wdiscussion)

I suggest further discussing debug packages after we know the result of
the ftpmaster meeting.

I also suggest to add discussing the required changes for automatic
debug packages on ftpmaster side to the meeting's agenda.


Regards
Carsten

 [1] Currently a Debian Developer/Maintainer builds locally and uploads
 the source package and one binary package.  Then the build daemons
 build binary packages for the other architectures.  Finally, all build
 packages become part of the archive.

 [2] http://lists.debian.org/debian-devel/2011/02/msg00064.html


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20110308150518.ga21...@furrball.stateful.de

Re: Automatic debug packages

[Philipp Kern]

On 2011-03-08, Carsten Hey cars...@debian.org wrote:
 * Emil Langrock [2011-03-08 00:48 +0100]:
 I browsed a little bit in the goals which were planned for squeeze and 
 noticed
 that the debug packages aka ddebs[1] weren't implemented in the debian
 infrastructure.
 A prerequisite to automatically add debug packages for all architectures
 is to change the way how packages are uploaded and/or build[1].  In the
 upcoming ftpmaster meeting[2] from the 21st until the 27th of March, one
 possible way to change it (or rather a part of it) will be discussed:

I don't think that's true.  In fact I also suggested back then that it should
just part of the normal build process.  Then DDs would upload ddebs just like
the buildds would.

The throw-away part isn't really connected with that.

Kind regards
Philipp Kern


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/slrnincmcb.uph.tr...@kelgar.0x539.de

Re: Automatic debug packages

[Josselin Mouette]

Le mardi 08 mars 2011 à 16:30 +, Philipp Kern a écrit : 
 I don't think that's true.  In fact I also suggested back then that it should
 just part of the normal build process.  Then DDs would upload ddebs just 
 like
 the buildds would.

FWIW, this is how the proposed changes to the toolchain work. It
requires some changes in dak, but nothing related to throwing away
packages.

-- 
 .''`.
: :' : “You would need to ask a lawyer if you don't know
`. `'   that a handshake of course makes a valid contract.”
  `---  J???rg Schilling


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/1299605388.18970.76.camel@meh

Re: Automatic debug packages

[Carsten Hey]

* Philipp Kern [2011-03-08 16:30 +]:
 On 2011-03-08, Carsten Hey cars...@debian.org wrote:
  A prerequisite to automatically add debug packages for all
  architectures is to change the way how packages are uploaded and/or
  build[1].  In the upcoming ftpmaster meeting[2] from the 21st until
  the 27th of March, one possible way to change it (or rather a part
  of it) will be discussed:

 I don't think that's true.  In fact I also suggested back then that it
 should just part of the normal build process.  Then DDs would upload
 ddebs just like the buildds would.

I read that there are plans to work on translation packages (TDeps)
before Wheezy is released.

Adding TDeps to Debian is in many aspects similar to adding automatic
debug packages (DDeps).  Given an reasonable abstract view on the
possible implementations of both, they can be grouped into:

 * An upload by a DD includes all additional packages.  Whether DD
   uploaded packages are thrown away does not matter.  This is the
   variant you mentioned.
 * An upload by a DD does not include additional packages.  DD uploaded
   packages are used.  Additional packages are created on the Debian
   infrastructure:
- DDeps: DD uploaded packages are rebuilt to extract debugging
  packages, the resulting binary package is thrown away.
  debug.debian.net is an implementation of this.
- TDeps: The message catalog is extracted from the uploaded binary
  package (or it is rebuilt, but this would be pointless) and the
  binary package is repacked.
 * An upload by a DD does not include additional packages.  DD uploaded
   packages are thrown away and rebuilt.  Ubuntu does this (or rather
   something equivalent) since ages to create debug packages.

Using implementations from the same group for DDeps and TDeps seems to
be a sane choice.

 The throw-away part isn't really connected with that.

If we decide to use a implementation from the first or the second group,
the throw-away part is indeed not connected with that.

If we decide to use a implementation from the third group, the
throw-away part is a prerequisite.

Before the throw-away part is decided, discussing the different ways of
implementing and choosing one could be a waste of time.

Questions to consider during such a discussion include:
 * Do we want users which build private packages to build also DDeps and
   TDeps?
 * Do we want to have a different building process (or build options)
   for to be uploaded packages?


Regards
Carsten


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20110308234028.ga27...@furrball.stateful.de

Re: Automatic debug packages

[Samuel Thibault]

  * Do we want users which build private packages to build also DDeps and
TDeps?

DDeps from private builds are useful to track bugs.

Samuel


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20110308234936.gq4...@const.famille.thibault.fr

Automatic debug packages

[Emil Langrock]

Hi,

I browsed a little bit in the goals which were planned for squeeze and noticed 
that the debug packages aka ddebs[1] weren't implemented in the debian 
infrastructure. I thought that many things happened [2] and there were also 
some wrappers implemented [3] to automatically generate those packages. Even a 
project as part of google summer of code 2009 was finished and its result were 
published [4]. But it seems that none of it is really accepted.

What is the direction we should follow for Wheezy[5]? Should we remove our 
special -dbg packages or at least stop to create new -dbg packages?

Kind regards,
Emil

[1] http://wiki.debian.org/AutomaticDebugPackages
[2] http://debug.debian.net/
[3] https://launchpad.net/ubuntu/+source/pkg-create-dbgsym
[4] http://code.google.com/p/google-summer-of-code-2009-
debian/downloads/detail?name=Emilio_PozueloMonfort.tar.bz2can=2q=
[5] http://release.debian.org/wheezy/goals.txt


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/201103080048.28079.emil.langr...@gmx.de

Re: Automatic Debug Packages

[Theodore Tso]

On Fri, Aug 14, 2009 at 05:50:50PM -0700, Russ Allbery wrote:
 Peter Samuelson pe...@p12n.org writes:
  [Emilio Pozuelo Monfort]
 
  We haven't agreed on whether there should be one ddeb per source or
  per binary package, so I would leave this still opened.
 
  Maybe I'm losing track of things here, but it seems to me that everyone
  except you is saying one ddeb per binary.  And then you say sure, we
  could do that if we need to.  How many times has this happened so far
  in the thread?  I haven't been keeping count.
 
 Joerg was also advocating one ddeb per source package in the summary
 message that he sent about the ftp-master approach, and Emilio has
 mentioned a few times that ftp-master needs to buy in on that decision
 (which I agree with).  I'm not sure if I'm missing some concern from the
 ftp-master side.

So if we have one ddeb per source package, which generates multiple
binary .debs for different libraries --- say, libext2fs, libcom_err,
and libss, to take a completely random example --- and the user
installs different versions of said libraries coming from different
versions of the source package, won't there be a problem if there is
only a single ddeb per source package?  I assume you can't install
multiple ddebs coming from different source packages at the same time,
since the pathnames would conflict, right?

- Ted


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Russ Allbery]

Theodore Tso ty...@mit.edu writes:
 On Fri, Aug 14, 2009 at 05:50:50PM -0700, Russ Allbery wrote:

 Joerg was also advocating one ddeb per source package in the summary
 message that he sent about the ftp-master approach, and Emilio has
 mentioned a few times that ftp-master needs to buy in on that decision
 (which I agree with).  I'm not sure if I'm missing some concern from
 the ftp-master side.

 So if we have one ddeb per source package, which generates multiple
 binary .debs for different libraries --- say, libext2fs, libcom_err, and
 libss, to take a completely random example --- and the user installs
 different versions of said libraries coming from different versions of
 the source package, won't there be a problem if there is only a single
 ddeb per source package?  I assume you can't install multiple ddebs
 coming from different source packages at the same time, since the
 pathnames would conflict, right?

It's possible with the binary-id method of storing debug symbols the paths
wouldn't conflict, but yes, this is one of my concerns.

-- 
Russ Allbery (r...@debian.org)   http://www.eyrie.org/~eagle/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Steve Langasek]

On Thu, Aug 13, 2009 at 09:42:55AM -0500, Manoj Srivastava wrote:
 On Thu, Aug 13 2009, Emilio Pozuelo Monfort wrote:

  Yes, dpkg, apt-get, aptitude and synaptic all work perfectly fine.

 The debug command addresses my concerns here.

You know this command doesn't actually exist, right?  AFAIK it's only
referenced in an Ubuntu wiki spec, and this part of that spec doesn't appear
to have been implemented.

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Manoj Srivastava]

On Fri, Aug 14 2009, Steve Langasek wrote:

 On Thu, Aug 13, 2009 at 09:42:55AM -0500, Manoj Srivastava wrote:
 On Thu, Aug 13 2009, Emilio Pozuelo Monfort wrote:

  Yes, dpkg, apt-get, aptitude and synaptic all work perfectly fine.

 The debug command addresses my concerns here.

 You know this command doesn't actually exist, right?  AFAIK it's only
 referenced in an Ubuntu wiki spec, and this part of that spec doesn't appear
 to have been implemented.

As far as I can see none of this is actually present now. The
 archive scripts have not been modified, there is not debug/* archive
 section, there are no downloadable share, yada et al helper packages
 have not yet been modified.

So I would actually have been very surprised were
 aptitude/synaptic ahead of the curve and had current support.

This whole thread seems to be about getting policy to write up
 things, with most of the proposal currently not implemented in
 Debian.  Now, I think you meant to warn me that some of the currently
 proposed features do not have code in other distributions either, and
 that's fair warning.

manoj
-- 
There are two problems with a major hangover.  You feel like you are
going to die and you're afraid that you won't.
Manoj Srivastava sriva...@debian.org http://www.debian.org/~srivasta/  
1024D/BF24424C print 4966 F272 D093 B493 410B  924B 21BA DABB BF24 424C


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Steve Langasek]

On Fri, Aug 14, 2009 at 12:55:38PM -0500, Manoj Srivastava wrote:
   Yes, dpkg, apt-get, aptitude and synaptic all work perfectly fine.

  The debug command addresses my concerns here.

  You know this command doesn't actually exist, right?  AFAIK it's only
  referenced in an Ubuntu wiki spec, and this part of that spec doesn't appear
  to have been implemented.

 As far as I can see none of this is actually present now. The
  archive scripts have not been modified, there is not debug/* archive
  section, there are no downloadable share, yada et al helper packages
  have not yet been modified.

 So I would actually have been very surprised were
  aptitude/synaptic ahead of the curve and had current support.

 This whole thread seems to be about getting policy to write up
  things, with most of the proposal currently not implemented in
  Debian.  Now, I think you meant to warn me that some of the currently
  proposed features do not have code in other distributions either, and
  that's fair warning.

Er, the point is that you've plucked this apt-get debug out of the
*Ubuntu* spec, https://wiki.ubuntu.com/AptElfDebugSymbols; that spec is
considered implemented, Ubuntu doesn't have such a debug command, and
Emilio's proposal at http://wiki.debian.org/AutomaticDebugPackages makes
no mention of implementing this.  Yet your messages come across as though
you think it's a foregone conclusion that this command is going to be
implemented.

So given that specification of apt-get commandline options *definitely*
doesn't belong in Policy, and no one has agreed to implement 'apt-get
debug', I think you might want to make your concerns about the package
management tools explicit.

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Tollef Fog Heen]

]] Emilio Pozuelo Monfort 

|  Because you're trying to debug a binary on your system that's linked
|  against it.
| 
| Then you should work on making your package build with the new library, since 
it
| will be FTBFS anyway :)

No, it won't.  You're confusing changed ABI from changed API.

Say you have foo, linked against libbar1, this works fine. foo, linked
against libbar2 (where the ABI was bumped because some part of the ABI
changed), doesn't work.  Being able to debug both of those to see where
their behaviour differs has value.

-- 
Tollef Fog Heen
UNIX is user friendly, it's just picky about who its friends are


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Manoj Srivastava]

On Fri, Aug 14 2009, Steve Langasek wrote:


 Er, the point is that you've plucked this apt-get debug out of the
 *Ubuntu* spec, https://wiki.ubuntu.com/AptElfDebugSymbols; that spec
 is considered implemented, Ubuntu doesn't have such a debug command,
 and Emilio's proposal at
 http://wiki.debian.org/AutomaticDebugPackages makes no mention of
 implementing this.  Yet your messages come across as though you think
 it's a foregone conclusion that this command is going to be
 implemented.

Well, I was nort looking at the Ubuntu spec at all, but I might
 have been mislead into a false sense of security:

From:  Philipp Kern tr...@philkern.de
Subject:  Re: Automatic Debug Packages
Message-ID:  slrnh87q12.7jj.tr...@kelgar.0x539.de

 This use case is IMHO implicitly addressed by making them
 downloadable and installable on the local system.

From: Emilio Pozuelo Monfort poch...@gmail.com
Message-ID: 4a84127e.1050...@gmail.com

 And aptitude and dpkg will know how to install ddebs, somehow?
  and synaptic, etc?
 Yes, dpkg, apt-get, aptitude and synaptic all work perfectly fine.


Given that assurance that aptitude and synaptic will know how to
 install ddebs, I said that satisfies my concern. I don't actually care
 what the command keyword used is, it may well be
   aptitude Avara-Kadavara foo-debug for all I care.

Are you saying that the assurance had no merit?

 So given that specification of apt-get commandline options *definitely*
 doesn't belong in Policy, and no one has agreed to implement 'apt-get
 debug', I think you might want to make your concerns about the package
 management tools explicit.

Well, before this issue is nailed down and writ into policy, the
 use cases for on-line webdav/nfs/share based debugging (including
 issues of where things are cached, how long they are cached, whether
 the bandwidth requirements make it infeasible), as well as the download
 debug packages method details need to be hammered out.

I do think that we need to have some kind of broad architecture
 of the solution in place for the common use cases before we can
 actually craft policy, and so far, I am very fuzzy about the aolution
 architecture. 

manoj
-- 
In order to discover who you are, first learn who everybody else is;
you're what's left.
Manoj Srivastava sriva...@debian.org http://www.debian.org/~srivasta/  
1024D/BF24424C print 4966 F272 D093 B493 410B  924B 21BA DABB BF24 424C


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Peter Samuelson]

[Emilio Pozuelo Monfort]
 We haven't agreed on whether there should be one ddeb per source or
 per binary package, so I would leave this still opened.

Maybe I'm losing track of things here, but it seems to me that everyone
except you is saying one ddeb per binary.  And then you say sure, we
could do that if we need to.  How many times has this happened so far
in the thread?  I haven't been keeping count.
-- 
Peter Samuelson | org-tld!p12n!peter | http://p12n.org/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Russ Allbery]

Peter Samuelson pe...@p12n.org writes:
 [Emilio Pozuelo Monfort]

 We haven't agreed on whether there should be one ddeb per source or
 per binary package, so I would leave this still opened.

 Maybe I'm losing track of things here, but it seems to me that everyone
 except you is saying one ddeb per binary.  And then you say sure, we
 could do that if we need to.  How many times has this happened so far
 in the thread?  I haven't been keeping count.

Joerg was also advocating one ddeb per source package in the summary
message that he sent about the ftp-master approach, and Emilio has
mentioned a few times that ftp-master needs to buy in on that decision
(which I agree with).  I'm not sure if I'm missing some concern from the
ftp-master side.

-- 
Russ Allbery (r...@debian.org)   http://www.eyrie.org/~eagle/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Michael Bienia]

On 2009-08-13 11:32:17 +0800, Paul Wise wrote:
 Not having anything to do with Ubuntu, I don't know anything about the
 details, but they have had automatic debug packages and automated
 crash report stuff for quite a while, a couple of years IIRC. The
 specs for that are here:
 
 https://launchpad.net/ubuntu/+spec/apt-get-debug-symbols
 https://launchpad.net/distros/ubuntu/+spec/automated-problem-reports

Some more links and information:

- The ddeb repository is at http://ddebs.ubuntu.com/
- They are created on the Ubuntu buildds with pkg-create-dbgsym which
  diverts dh_strip.
  Source at https://launchpad.net/ubuntu/+source/pkg-create-dbgsym

They are used by the automatic retracing service of crashes filed in
Launchpad with apport. Example for such a bug
https://bugs.launchpad.net/ubuntu/+source/pango1.0/+bug/327446

But one can also use them to locally retrace crashes with
apport-retrace:
,
| Description: tools for reprocessing Apport crash reports
|  apport-retrace recombines an Apport crash report (either a file or a
|  Launchpad bug) and debug symbol packages (.ddebs) into fully symbolic
|  stack traces.
|  .
|  This package also ships apport-chroot. This tool can create and
|  manage chroots for usage with apport-retrace. If the fakeroot and
|  fakechroot libraries are available (either by installing the packages
|  or by merely putting their libraries somewhere and setting two
|  environment variables), the entire process of retracing crashes in
|  chroots can happen with normal user privileges.
| Homepage: https://wiki.ubuntu.com/Apport
`
Source package at https://launchpad.net/ubuntu/+source/apport

Michael


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Emilio Pozuelo Monfort]

Russ Allbery wrote:
 Hm, that's interesting, but I suspect that few enough of our users will be
 able to use such a thing that we shouldn't let that influence any other
 design choices.  Most shares are not going to be able to be mounted
 through firewalls, for example, so that form of the debug symbols won't be
 available to quite a few users.
 
 Or maybe by share you meant something that was more like a file download
 service over HTTP than, say, NFS?

I've been playing with WebDAV, which is an extension to HTTP. So I guess that
will work with firewalls?

Anyway I haven't talked to DSA yet, so we will have to see what is possible and
what is not.

Emilio



signature.asc
Description: OpenPGP digital signature

Re: Automatic Debug Packages

[Philipp Kern]

On 2009-08-13, Manoj Srivastava sriva...@debian.org wrote:
 Ah, and it looks like the automated crash reporting offers to download the
 -dbgsym packages and install them.
 Reading the spec, it seems to me that the primary motivation was
  for users to provide crash dumps with bug reports, and not much screen
  time is given to users debugging their own applications linked to
  vendor libraries, or for the developer user  in general. I think that
  use case should be addressed as well.

This use case is IMHO implicitly addressed by making them downloadable
and installable on the local system.

And I have to agree with Emilio that I don't see the point of a 1:1
relationship of ddeb to binary package just for the sake of library
transitions.  I wonder if we could just unpack the debugging build-id
objects to some other location than globally and point gdb to that
in addition to the global debug store.

Someone should've pointed a summary of how Ubuntu does it, it seems.

Kind regards,
Philipp Kern



-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Emilio Pozuelo Monfort]

Manoj Srivastava wrote:
 On Wed, Aug 12 2009, Emilio Pozuelo Monfort wrote:
 
 
 There will still be a repository with all the .ddebs.
 
 And aptitude and dpkg will know how to install ddebs, somehow?
  and synaptic, etc?

Yes, dpkg, apt-get, aptitude and synaptic all work perfectly fine.

They don't care about the extension at all.

 But also we will have a share that will ship all the debugging symbols
 in a build id file hierarchy structure (so something like
 .build-id/xx/xxx.debug). You can mount it in your system and use
 it as if you had installed every -ddeb available in the
 archive.
 
 So all debugging has to happen online? Many places have a
  prohibition against mounting a file system from outside the firewall,
  though installing packages that have been vetted is permissible

No, we provide it for whoever wants to use it. There will still be an archive
with all the packages, so if you prefer that or if you won't have networking
when the need of debugging arises, or if you don't like the other way, use the
packages.

 Furthermore, if disk space permits it, we will provide symbols for
 more than one version (i.e. not only the current package in the
 archive, but maybe the last three or whatever we can do), since build
 ids permit it.
 
 With packages, mirrored repositories may have a different
  retention policy, and not rely on the packages one has installed
  disappearing off the remote filesystem.
 
 The system you propose works great for the use case you have
  envisaged: Debugging on a low security instllation with always on
  broadband connection to the network; but surely that is not the only
  model we provide.

As I've said, there will still be a debug archive. I don't see what's the
problem with providing *both* an archive and a share, really. If you can't use
the latter, that's fine, use the packages.

 I am also wondering about the obstacles placed in the path of
  packages that will not be covered by the automated system. While we
  were  talking about generating .debs, that was some work, but not any
  different from creating a package.  Now, in order to create a hand
  crafted ddeb, what does one do? Add a nerw package to debian/control,
  build it, rename the package, edit ./debian/files before
  dpkg-genchanges is called  -- this is more complex than just calling
  dpkg-buildpackage and be done with it.

You can do it the hard way. Or you can tell dpkg-gencontrol how the package
should be named with the -n option. It will do the correct thing, and add the
correct filename to debian/files. Furthermore, we could expand dpkg-gencontrol
to accept a --extension option, so that you don't need to look for the package
version and arch.

Cheers,
Emilio



signature.asc
Description: OpenPGP digital signature

Re: Automatic Debug Packages

[Emilio Pozuelo Monfort]

Julien Cristau wrote:
 On Thu, Aug 13, 2009 at 02:58:45 +0200, Emilio Pozuelo Monfort wrote:
 
 If that bothers you, you can use the share we plan to provide.

 I'd like to still be able to debug offline, thank you very much.  So far
 you've avoided answering the question, though: why one ddeb per source
 instead of per binary?

The current status quo for source packages that build a -dbg package is not very
different AFAICS. But I don't really care much one way or the other, I'll
implement what the project decides. You'll need to convince the ftpmasters 
though.

Emilio



signature.asc
Description: OpenPGP digital signature

Re: Automatic Debug Packages

[Emilio Pozuelo Monfort]

Roger Leigh wrote:
 On Thu, Aug 13, 2009 at 02:58:45AM +0200, Emilio Pozuelo Monfort wrote:
 Roger Leigh wrote:
 This fails to address the rather valid concern brought up about
 having different versions of libraries and binaries installed
 from the same source package.  Having one .ddeb per binary would
 solve this elegantly.
 Except that in that case, the old library will be NBS and thus I see no point
 why you would want to keep it installed. The only reason would be if it was
 meant to stay around, but in that case I'm sure the source package names 
 would
 be different.
 
 The scenario has been described already in the thread.

And I don't consider NBS packages a good reason to change it.

 It's also rather space-inefficient for the user.
 If that bothers you, you can use the share we plan to provide.
 
 No thanks, I like my debug symbols in a nice convenient packaged
 format, signed by the archive admins.

And we will provide that.

 While you might plan to be providing some fancy (yet enigmatic) service
 based upon the debug deb content,

 I still want them installable,

Sure

 preferably automatically getting all dependent debug symbols as well
 using apt.

I want to provide {apt-get,aptitude} debug commands, but that's orthogonal to
this discussion.

 Preferably with a .deb extension;

Why does that matter to you?

 I see no reason why they
 can't be first-class Debian packages

They are.

 in addition to being used for
 mysterious as-yet-unspecified purposes.

A share would be provided shipping all the debugging symbols that use build
ids (so it's easy to mount it in /usr/lib/debug/.build-id/ and have all the
debugging symbols available to debuggers and anything that needs them).

If that's mysterious, tell me where. That has been in the wiki page linked from
the message that started this thread.

 I can't say I'm particularly enthused with the apparent lack of
 consideration for most of the issues with the proposal brought up in
 this thread.

No comments.

Emilio



signature.asc
Description: OpenPGP digital signature

Re: Automatic Debug Packages

[Eugene V. Lyubimkin]

Hello thread! /me puts on a package manager developer hat.

Sorry, I haven't read the whole thread, it's huge.

I think that diversion of debug packages out of current deb format is a
completely wrong direction. Do you want to teach all tools that get some info
about Debian packages that there is new 'ddeb' format packages? New .ddeb
extension? For a what sake?

Oh, no, and this covers not 3 packages. Let's count. The following command
counts reverse {predepends,depends,recommends} to apt (which contains
libapt-pkg) and to libcupt:

$ cupt rdepends apt libcupt-perl 2/dev/null | grep Reverse | wc -l
63


To extract the info 'is this debug package?' you can use a good and easy
regular expression '.*-debug$' applied to package name (or any other suffix
you want).

Are there other reasons?

[Roger Leigh]
 I see no reason why they
 can't be first-class Debian packages
Fully agree.

While I support automatic generation of debug packages, creating a new format
for them sounds for me as creating new RFC for e-mails which bodies contain no
spaces and no Bcc header allowed. Why? To filter 'automatic debug mails'.

-- 
Eugene V. Lyubimkin aka JackYF, JID: jackyf.devel(maildog)gmail.com
C++/Perl developer, Debian Maintainer



signature.asc
Description: OpenPGP digital signature

Re: Automatic Debug Packages

[Emilio Pozuelo Monfort]

Eugene V. Lyubimkin wrote:
 Hello thread! /me puts on a package manager developer hat.
 
 Sorry, I haven't read the whole thread, it's huge.
 
 I think that diversion of debug packages out of current deb format is a
 completely wrong direction. Do you want to teach all tools that get some info
 about Debian packages that there is new 'ddeb' format packages? New .ddeb
 extension? For a what sake?

Maybe you should spend some time and read the thread before stating such things.
Neither dpkg nor apt, aptitude, apt-get and company need to know anything new.
They just work perfectly fine. The format is the same.

Cheers,
Emilio



signature.asc
Description: OpenPGP digital signature

Re: Automatic Debug Packages

[Manoj Srivastava]

On Thu, Aug 13 2009, Emilio Pozuelo Monfort wrote:


 Yes, dpkg, apt-get, aptitude and synaptic all work perfectly fine.

The debug command addresses my concerns here.

 As I've said, there will still be a debug archive. I don't see what's
 the problem with providing *both* an archive and a share, really. If
 you can't use the latter, that's fine, use the packages.

Oh, that's good, then.

 Furthermore, we could expand dpkg-gencontrol to accept a --extension
 option, so that you don't need to look for the package version and
 arch.

That would be really nice.

So, at this point, I have a somewhat pedantic issue with calling
 the same package format with two names (.deb and .ddeb), but this is
 mostly aesthetics, and not a real technical objection. 


manoj
-- 
It is impossible to make anything foolproof, because fools are so
ingenious.
Manoj Srivastava sriva...@debian.org http://www.debian.org/~srivasta/  
1024D/BF24424C print 4966 F272 D093 B493 410B  924B 21BA DABB BF24 424C


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Emilio Pozuelo Monfort]

Russ Allbery wrote:
 https://wiki.ubuntu.com/AptElfDebugSymbols is the specification.  It does
 use *.ddeb.  There isn't any clear statement about how *.ddeb packages
 differ from *.deb packages.  It looks like, by and large, they don't,
 except they may not need to contain the same set of things.  The packages
 are not in debian/control or the things fed from it, but are in *.changes.

The format is the same of .deb packages, only the extension changes.

 Ubuntu is creating one debug package per binary package, as I and a few
 other people have said we prefer. However, they're using the
 gnu_debuglink method, not the id method, so they don't have the one
 problem with that method previously identified in this thread when the
 same binary is copied into multiple packages.

Nor the advantages. Anyway I don't care about one or many ddeb packages,
convince the ftpmasters and I'll do one per binary package, resolving the build
id file conflicts with replaces.

 The debug packages depend on the packages for which they have symbols,
 which solves the problem of not installing debug packages that both
 provide symbols for the same binary.

I don't get the problem, but I agree having one ddeb per binary package makes
package relationships between ddebs and normal packages easier.

 The proposal appears to only work for packages using debhelper, although
 the steps are laid out so presumably they could be done manually if need
 be.

It's more complicated in their case because they do it only on the buildds by
diverting debhelper (dh_strip to be precise). We, on the other hand, will modify
debhelper directly so that ddebs are built anywhere. If you don't use debhelper,
that's OK, as the proposal is not debhelper specific. You can magically build
the ddebs with any other tools you want, or you can build them manually, and
they will be accepted.

 Ubuntu uses -dbgsym as the magic namespace.  I don't know if it would be
 good for us to do the same or to use a different namespace to avoid
 problems for them in cases where we decide to build the package manually
 via debian/control and debian/rules.

I can ask Martin Pitt (who implemented and maintains the Ubuntu ddeb
infrastructure), but it's just a detail to us.

 It looks like the plan with *.ddeb is to treat them specially in the
 archive software and divert them into a different tree in the archive
 instead of using a separate archive area, which I think they're doing
 now from that page.  It also looks like one of the justifications for the
 separate package is to hide them in the apt front-end so that users don't
 see all the additional packages.  I'm personally skeptical this is a good
 idea, although I can see the merits of not returning them in apt-cache
 search.

That would be an apt change that I don't have in mind to do. We may or may not
do it, but that's orthogonal to the topic I think.

 Ah, and it looks like the automated crash reporting offers to download the
 -dbgsym packages and install them.

apport-retrace does that, yes. Would be nice to integrate apport in Debian (an
ITP has recently being filled).

Cheers,
Emilio



signature.asc
Description: OpenPGP digital signature

Re: Automatic Debug Packages

[Eugene V. Lyubimkin]

Emilio Pozuelo Monfort wrote:
 Eugene V. Lyubimkin wrote:
 Hello thread! /me puts on a package manager developer hat.

 Sorry, I haven't read the whole thread, it's huge.

 I think that diversion of debug packages out of current deb format is a
 completely wrong direction. Do you want to teach all tools that get some info
 about Debian packages that there is new 'ddeb' format packages? New .ddeb
 extension? For a what sake?
 
 Maybe you should spend some time and read the thread before stating such 
 things.
 Neither dpkg nor apt, aptitude, apt-get and company need to know anything new.
 They just work perfectly fine. The format is the same.
Really? So, they are already first-class deb packages? According to the your
first message in the whole thread, the answer is no. I just re-read (for
sure) the Joerg's mail back in this subthread, and my answer is still no. I
re-read some random mails and saw '.ddeb' here and there. And, what's then
second (yours) and third (Roger's) letters ago directly in this thread speak
about? You has put far more bigger and constructive answer (yet unsatisfiable)
to Roger points, and my points is basically subset of them, re-explained from
package manager PoV.

You also told that '...dpkg nor apt, aptitude, apt-get and company... just
work perfectly fine'. What percent of their (and other tools') functionality
did you test to say that?

-- 
Eugene V. Lyubimkin aka JackYF, JID: jackyf.devel(maildog)gmail.com
C++/Perl developer, Debian Maintainer



signature.asc
Description: OpenPGP digital signature

Re: Automatic Debug Packages

[Russ Allbery]

Philipp Kern tr...@philkern.de writes:

 And I have to agree with Emilio that I don't see the point of a 1:1
 relationship of ddeb to binary package just for the sake of library
 transitions.  I wonder if we could just unpack the debugging build-id
 objects to some other location than globally and point gdb to that in
 addition to the global debug store.

The thing is, there's no significant *drawback* to a 1:1 relationship that
I can see, and it makes this use case easier.  The only drawback stated so
far is that one may need to add Replaces in the rare case of an identical
binary in multiple packages.  I doubt there are more than 30 or 40 such
packages in the entire archive.  So it seems like a bad idea to me to
break this.

 Someone should've pointed a summary of how Ubuntu does it, it seems.

Yes.  Please don't assume Debian Developers have any idea what Ubuntu is
doing.  :)

-- 
Russ Allbery (r...@debian.org)   http://www.eyrie.org/~eagle/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Russ Allbery]

Emilio Pozuelo Monfort poch...@gmail.com writes:
 Russ Allbery wrote:

 https://wiki.ubuntu.com/AptElfDebugSymbols is the specification.  It
 does use *.ddeb.  There isn't any clear statement about how *.ddeb
 packages differ from *.deb packages.  It looks like, by and large, they
 don't, except they may not need to contain the same set of things.  The
 packages are not in debian/control or the things fed from it, but are
 in *.changes.

 The format is the same of .deb packages, only the extension changes.

I assumed that.  I'm more concerned right now with the Policy compliance
of the contents.  Ubuntu talks about using a separate package extension
because they don't need to be full packages, but there doesn't seem to be
any particular motivating reason for that -- in other words, I still don't
see any reason why they can't be regular Debian packages.

 Anyway I don't care about one or many ddeb packages, convince the
 ftpmasters and I'll do one per binary package, resolving the build id
 file conflicts with replaces.

Excellent, thank you.  That's certainly my intention.

 The debug packages depend on the packages for which they have symbols,
 which solves the problem of not installing debug packages that both
 provide symbols for the same binary.

 I don't get the problem, but I agree having one ddeb per binary package
 makes package relationships between ddebs and normal packages easier.

There are multiple places in the archive where two binary packages install
binaries with the same name but different contents, where for some reason
alternatives cannot be used.  If using the binary-id method, this doesn't
matter, but if using the gnu_debuglink method, those debug packages also
have to conflict.  Having the debug package depend on the corresponding
binary package takes care of this.  (So does, of course, using binary-id
everywhere.)

-- 
Russ Allbery (r...@debian.org)   http://www.eyrie.org/~eagle/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Philipp Kern]

On 2009-08-13, Eugene V. Lyubimkin jackyf.de...@gmail.com wrote:
 Maybe you should spend some time and read the thread before stating such 
 things.
 Really? So, they are already first-class deb packages?

Maybe you should spend some time and read the thread before stating such things.

 What percent of their (and other tools') functionality did you test to say
 that?

It's already used in production by Ubuntu.  With no changes made to them.

Kind regards,
Philipp Kern


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Eugene V. Lyubimkin]

Philipp Kern wrote:
 On 2009-08-13, Eugene V. Lyubimkin jackyf.de...@gmail.com wrote:
 Maybe you should spend some time and read the thread before stating such 
 things.
 Really? So, they are already first-class deb packages?
 
 Maybe you should spend some time and read the thread before stating such 
 things.
The thread contains several mails which still special-casing these packages.
Am I right you now state that ddebs should fully conform to Debian Policy?

 What percent of their (and other tools') functionality did you test to say
 that?
 
 It's already used in production by Ubuntu.  With no changes made to them.
Ubuntu also switched to /bin/dash as /bin/sh several Ubuntu releases ago. Many
bashisms were filed/fixed since that time. So, as Emilio says, 'this is
orthogonal'.

-- 
Eugene V. Lyubimkin aka JackYF, JID: jackyf.devel(maildog)gmail.com
C++/Perl developer, Debian Maintainer



signature.asc
Description: OpenPGP digital signature

Re: Automatic Debug Packages

[Russ Allbery]

Emilio Pozuelo Monfort poch...@gmail.com writes:

 I've been playing with WebDAV, which is an extension to HTTP. So I guess
 that will work with firewalls?

Yes, it should.

The only lingering concern that I have about a share is how it plays with
the files installed on your system via package.  Can gdb be pointed to
multiple different debug information stores at different paths?  Mounting
a remote share over top of /usr/lib/debug obviously wouldn't work in many
cases (and WebDAV you're unlikely to mount as a file system at all).

-- 
Russ Allbery (r...@debian.org)   http://www.eyrie.org/~eagle/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Roger Leigh]

On Thu, Aug 13, 2009 at 02:08:58PM -0700, Russ Allbery wrote:
 Emilio Pozuelo Monfort poch...@gmail.com writes:
 
  I've been playing with WebDAV, which is an extension to HTTP. So I guess
  that will work with firewalls?
 
 Yes, it should.
 
 The only lingering concern that I have about a share is how it plays with
 the files installed on your system via package.  Can gdb be pointed to
 multiple different debug information stores at different paths?  Mounting
 a remote share over top of /usr/lib/debug obviously wouldn't work in many
 cases (and WebDAV you're unlikely to mount as a file system at all).

My other questions about this relate to bandwidth usage:

• does it keep symbols between gdb runs, or do they need downloading
  every time?

• if it keeps them, where are they stored, how long for and how do
  they get cleaned up?

• if it doesn't keep them, doesn't this have the potential to waste
  gigabytes of my monthly DSL bandwidth quota, since debugging
  symbols can be huge and you might run gdb repeatedly on many
  occasions?

With having actual debug packages installed, it's easy to know how
long they will remain for, and to remove them when done.  Knowing
the above would be useful.

For the users who will benefit from one-time reporting of problems,
this probably won't be an issue.  But, if you get repeated crashes
you might need to get the symbols many many times.  For developers
doing a lot of development work, you aren't going to want to fetch
the symbols again and again.


Regards,
Roger

-- 
  .''`.  Roger Leigh
 : :' :  Debian GNU/Linux http://people.debian.org/~rleigh/
 `. `'   Printing on GNU/Linux?   http://gutenprint.sourceforge.net/
   `-GPG Public Key: 0x25BFB848   Please GPG sign your mail.


signature.asc
Description: Digital signature

Re: Automatic Debug Packages

[Emilio Pozuelo Monfort]

Russ Allbery wrote:
 The only lingering concern that I have about a share is how it plays with
 the files installed on your system via package.  Can gdb be pointed to
 multiple different debug information stores at different paths?  Mounting
 a remote share over top of /usr/lib/debug obviously wouldn't work in many
 cases (and WebDAV you're unlikely to mount as a file system at all).

It can't read debugging symbols from several places, but you can change the
global debug directory where it reads them, which by default is /usr/lib/debug.
So you could mount the share in $HOME or wherever, and point gdb to it.

Cheers,
Emilio



signature.asc
Description: OpenPGP digital signature

Re: Automatic Debug Packages

[Steve Langasek]

On Wed, Aug 12, 2009 at 11:17:55PM +0200, Joerg Jaspert wrote:
 First, the naming:
 It is indeed us Ftpmasters who want them named .ddeb. For two reasons -
 a simple one is that it makes the handling much easier, if you can match
 on *.ddeb$. Another one is that they aren't real debs like .deb. They
 aren't meant for normal user consumption, but only for special
 situations. For most users possibly completly automated and
 transparently in the background. Seperating them as .ddeb helps making
 it clear it is something different than what you know from usual debian
 packages. More so than a -debug in the name alone.

I don't think this is going to make it very clear to most users (what user
is going to look at the filename?), and so far I don't see much reason that
they should *be* different than .debs.

 Also, ddebs should probably be defined in policy as not having
 maintainer scripts.

That's a reason to document them separately in policy, certainly, though I
don't think this has been mentioned before now as a requirement?

 Second the storage:
 Archive side they will be put into the normal
 directories right beside the source and other binaries from the
 package. They will, however, not be exported to the public view of the
 archive. Instead we will export a second directory which contains them,
 which can then be mirrored seperately from mirrors that do want to have
 them.

Ehm...

$ du -sh /srv/ftp.debian.org/ftp
410G/srv/ftp.debian.org/ftp
$ df -h /srv/ftp.debian.org/
FilesystemSize  Used Avail Use% Mounted on
/dev/cciss/c0d0p1 1.4T  1.1T  205G  85% /
$

I would guess a full set of ddebs for unstable would take *at least* as much
space as everything currently in the archive (oldstable - experimental).  Are
there plans for a hardware update on ftp-master to accomodate this sudden
explosion in archive size?

 Quantity of .ddebs:
 Usually there should only be one .ddeb per source. Of course there are
 always exceptions from the rule, so Maintainers may chose to have one
 per binary package. This should only be taken when the size of the debug
 package gets *huge* otherwise. It is hard to set definite numbers here,
 but a 5mb package would surely not be a reason to split into two 2.5mb
 ones.

Why should this be the norm?

- they're outside the main archive, so by default the Packages file size has
  no impact on the end user

- much of the data in the Packages file should be irrelevant for ddebs
  (e.g., short auto-generated Package descriptions maybe?), so even at a 1:1
  ratio the ddebs Packages file should be a quicker download for users

- grouping by source package requires an extra indirection when figuring out
  the correct ddeb to download (file - binary package - source package)

- Maintainers may choose opens up a broad range of cases where maintainers
  will have to manually implement this, particularly if they want ddebs to
  only be installed when the corresponding deb is installed; whereas with
  per-binary-package ddebs these cases could be automated.

 The packages do also not need to be listed in debian/control, if they
 follow the one ddeb per source general rule. If there are more of them
 then they will need to be explicitly defined.

Why should this be the case?

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Emilio Pozuelo Monfort]

Russ Allbery wrote:
 Josselin Mouette j...@debian.org writes:
 If we use build IDs (and this has quite some advantages, like being able
 to do more than just dump the ddebs on a repository), this can lead to
 having the same detached debugging symbols in two binary packages, since
 sometimes a binary is built twice the same exact way and put in two
 different binary packages.
 
 Hm, really?  The toolchains that I'm familiar with basically never produce
 the same binary twice; something is always slightly different from
 timestamp information.  Could you give an example of such a case in the
 archive right now where identical binaries are in multiple packages so
 that I can better understand how this happens?

There are things the linker takes into account for calculating the build id. The
timestamp isn't one of them.

E.g.:

emi...@saturno:/tmp$ echo main(){}  a.c; cp a.c b.c; gcc -o a a.c; gcc -o b
b.c; eu-readelf -n a b | grep Build
Build ID: c164b7f24a18b665601cd1d1b9fa0af6afb728bb
Build ID: c164b7f24a18b665601cd1d1b9fa0af6afb728bb

So in cases where you build a package twice with different configure flags, it's
be possible that those don't affect some binaries, getting the same build ids.

E.g., after a build of evince/2.26.1-2 from testing (the package in unstable has
been changed to ship the backends in the library package so this is no longer
the case, but to show this happens):

r...@saturno:~/evince-2.26.1# eu-readelf -n
debian/evince/usr/lib/evince/1/backends/*so
debian/evince-gtk/usr/lib/evince/1/backends/*so | grep Build | sort | uniq -c
  2 Build ID: 0232654930d90461896f3d58fe08178082a217df
  2 Build ID: 08de63310c0ff98c5aac6392d95c7bd6fc502c8b
  2 Build ID: 71b914ea23bb199d9d98de2a15a9d07e982a3ae0
  2 Build ID: 7a40178124bf7698de230b2298378f08795ddbe5
  2 Build ID: 8de5ebfcb2bfceb9ed19a57d6bbc918392e152ec
  2 Build ID: c0b63d2ecd7432f0a01441e0794306651c88f5f7
  2 Build ID: c3b7f89bda6381e5849819032f842e6870e184b5
  2 Build ID: dffdcca3f7a89b4b9da333d7cc638a96ed8b1bc8


Cheers,
Emilio



signature.asc
Description: OpenPGP digital signature

Re: Automatic Debug Packages

[Guillem Jover]

Hi!

On Tue, 2009-08-11 at 13:03:13 -0700, Russ Allbery wrote:
 Open questions:

 * Can we require a one-to-one correspondance between binary package names
   and debug package names that provide symbols for that binary package?  I
   think we should; I think it would make the system more straightforward.

Being able to debug your system with a mixture of applications using an
old and new shared library coming from the same source package for which
the soversion got bumped is pretty helpful, w/o needing to downgrade the
whole debugging package every time one wants to debug applications
using either.

regards,
guillem


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Joerg Jaspert]

On 11826 March 1977, Emilio Pozuelo Monfort wrote:

 The proposal is (very briefly) to make dak accept .ddeb packages (containing
 debugging symbols using build-ids), and to then modify helper tools to
 automatically generate them and add them to the changes file. I've written 
 down
 the details in the wiki [2], and I'll appreciate it if you could give some
 feeback. I don't want to trash this completely though, so no drastic changes
 preferred :)

What a long thread, brrr. So lets take the start message of it and reply
to a few points that have been all over in it.


First, the naming:
It is indeed us Ftpmasters who want them named .ddeb. For two reasons -
a simple one is that it makes the handling much easier, if you can match
on *.ddeb$. Another one is that they aren't real debs like .deb. They
aren't meant for normal user consumption, but only for special
situations. For most users possibly completly automated and
transparently in the background. Seperating them as .ddeb helps making
it clear it is something different than what you know from usual debian
packages. More so than a -debug in the name alone.

Also, ddebs should probably be defined in policy as not having
maintainer scripts.

Additionally the naming should include a -$DEBUG, so it will be
$package-$DEBUG_$version_$arch.ddeb.
$DEBUG should be a string not otherwise used, debug would probably work
well.



Second the storage:
Archive side they will be put into the normal
directories right beside the source and other binaries from the
package. They will, however, not be exported to the public view of the
archive. Instead we will export a second directory which contains them,
which can then be mirrored seperately from mirrors that do want to have
them.

Now, as that will be a seperately exported mirror tree it leaves license
compliance (provide binaries, provide source). Thats discussable - as we
only provide debug symbols/code/whatever_funny_language_uses_for_it we
might not fall below that requirement. Most likely we ensure compliance
by having symlinks back into the normal /debian/ mirror tree of a
mirror, and requiring each mirror to only mirror this if they also have
/debian/. (This was one way the sarge amd64 archive was presented to the
mirrors. Saved some of them quite a lot of space, not needing to
duplicate the source from Debian.)


Contents of a .ddeb:
The contents of a .ddeb should strictly be limited to debugging symbols
- or whatever their equivalent is in other programming
languages/environments. No user needed/runable parts should be there.
Which will keep -dbg packages like the python interpreters in the normal
archive.
Additionally a .ddeb must either include a
/usr/share/doc/$package-$debug/ directory or a symlink to a
/usr/share/doc/$package directory of a package it depends on.



Quantity of .ddebs:
Usually there should only be one .ddeb per source. Of course there are
always exceptions from the rule, so Maintainers may chose to have one
per binary package. This should only be taken when the size of the debug
package gets *huge* otherwise. It is hard to set definite numbers here,
but a 5mb package would surely not be a reason to split into two 2.5mb
ones.


Other stuff:
Most of the other stuff is not of much concern to us.
As to how the files within ddebs should be named, this isn't an archive
side matter (although I personally favor to put files named after their
build-ids into the .ddebs).

The packages do also not need to be listed in debian/control, if they
follow the one ddeb per source general rule. If there are more of them
then they will need to be explicitly defined.
They do *ALL* have to appear in the .changes file uploaded.


-- 
bye, Joerg, your FTPMaster of the evening
Hätten die Affen, von denen wir angeblich abstammen, geahnt, dass durch
die Evolution eines Tages aus Ihren Reihen Politiker entstehen würden,
wären sie auf Ihren Bäumen geblieben und hätten niemals versucht den
aufrechten Gang zu erlernen.
(J. Sheridan, Babylon5)


pgpp5E43bfWNQ.pgp
Description: PGP signature

Re: Automatic Debug Packages

[Roger Leigh]

On Wed, Aug 12, 2009 at 11:17:55PM +0200, Joerg Jaspert wrote:
 Quantity of .ddebs:
 Usually there should only be one .ddeb per source. Of course there are
 always exceptions from the rule, so Maintainers may chose to have one
 per binary package. This should only be taken when the size of the debug
 package gets *huge* otherwise. It is hard to set definite numbers here,
 but a 5mb package would surely not be a reason to split into two 2.5mb
 ones.

This fails to address the rather valid concern brought up about
having different versions of libraries and binaries installed
from the same source package.  Having one .ddeb per binary would
solve this elegantly.

It's also rather space-inefficient for the user.


Regards,
Roger

-- 
  .''`.  Roger Leigh
 : :' :  Debian GNU/Linux http://people.debian.org/~rleigh/
 `. `'   Printing on GNU/Linux?   http://gutenprint.sourceforge.net/
   `-GPG Public Key: 0x25BFB848   Please GPG sign your mail.


signature.asc
Description: Digital signature

Re: Automatic Debug Packages

[Emilio Pozuelo Monfort]

Roger Leigh wrote:
 On Wed, Aug 12, 2009 at 11:17:55PM +0200, Joerg Jaspert wrote:
 Quantity of .ddebs:
 Usually there should only be one .ddeb per source. Of course there are
 always exceptions from the rule, so Maintainers may chose to have one
 per binary package. This should only be taken when the size of the debug
 package gets *huge* otherwise. It is hard to set definite numbers here,
 but a 5mb package would surely not be a reason to split into two 2.5mb
 ones.
 
 This fails to address the rather valid concern brought up about
 having different versions of libraries and binaries installed
 from the same source package.  Having one .ddeb per binary would
 solve this elegantly.

Except that in that case, the old library will be NBS and thus I see no point
why you would want to keep it installed. The only reason would be if it was
meant to stay around, but in that case I'm sure the source package names would
be different.

 It's also rather space-inefficient for the user.

If that bothers you, you can use the share we plan to provide.

Cheers,
Emilio



signature.asc
Description: OpenPGP digital signature

Re: Automatic Debug Packages

[Russ Allbery]

Emilio Pozuelo Monfort poch...@gmail.com writes:
 Roger Leigh wrote:

 This fails to address the rather valid concern brought up about having
 different versions of libraries and binaries installed from the same
 source package.  Having one .ddeb per binary would solve this
 elegantly.

 Except that in that case, the old library will be NBS and thus I see no
 point why you would want to keep it installed. The only reason would be
 if it was meant to stay around, but in that case I'm sure the source
 package names would be different.

Because you're trying to debug a binary on your system that's linked
against it.

 It's also rather space-inefficient for the user.

 If that bothers you, you can use the share we plan to provide.

I'm very curious to see more details about how this is going to work.  It
sounds like we may need to hold off making any decisions or Policy changes
here until the details of that system is worked out if the normal delivery
system for the things in .ddebs won't be via package installation.

-- 
Russ Allbery (r...@debian.org)   http://www.eyrie.org/~eagle/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Julien Cristau]

On Thu, Aug 13, 2009 at 02:58:45 +0200, Emilio Pozuelo Monfort wrote:

 If that bothers you, you can use the share we plan to provide.
 
I'd like to still be able to debug offline, thank you very much.  So far
you've avoided answering the question, though: why one ddeb per source
instead of per binary?

Cheers,
Julien


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Emilio Pozuelo Monfort]

Russ Allbery wrote:
 Emilio Pozuelo Monfort poch...@gmail.com writes:
 Roger Leigh wrote:
 
 This fails to address the rather valid concern brought up about having
 different versions of libraries and binaries installed from the same
 source package.  Having one .ddeb per binary would solve this
 elegantly.
 
 Except that in that case, the old library will be NBS and thus I see no
 point why you would want to keep it installed. The only reason would be
 if it was meant to stay around, but in that case I'm sure the source
 package names would be different.
 
 Because you're trying to debug a binary on your system that's linked
 against it.

Then you should work on making your package build with the new library, since it
will be FTBFS anyway :)

I don't consider this a real issue.

 It's also rather space-inefficient for the user.
 
 If that bothers you, you can use the share we plan to provide.
 
 I'm very curious to see more details about how this is going to work.  It
 sounds like we may need to hold off making any decisions or Policy changes
 here until the details of that system is worked out if the normal delivery
 system for the things in .ddebs won't be via package installation.

There will still be a repository with all the .ddebs. But also we will have a
share that will ship all the debugging symbols in a build id file hierarchy
structure (so something like .build-id/xx/xxx.debug). You can mount it in
your system and use it as if you had installed every -ddeb available in the
archive. Furthermore, if disk space permits it, we will provide symbols for more
than one version (i.e. not only the current package in the archive, but maybe
the last three or whatever we can do), since build ids permit it.

With this in place, we can integrate reporting tools (bug-buddy, drkonqi,
apport) to use that service to magically provide meaningful bug reports with
complete backtraces.

If you use this, you won't need to get a backtrace, realize you're missing some
symbols, install some more debug packages, rinse, repeat... :)

Hope this was clear,
Emilio



signature.asc
Description: OpenPGP digital signature

Re: Automatic Debug Packages

[Roger Leigh]

On Thu, Aug 13, 2009 at 02:58:45AM +0200, Emilio Pozuelo Monfort wrote:
 Roger Leigh wrote:
  On Wed, Aug 12, 2009 at 11:17:55PM +0200, Joerg Jaspert wrote:
  Quantity of .ddebs:
  Usually there should only be one .ddeb per source. Of course there are
  always exceptions from the rule, so Maintainers may chose to have one
  per binary package. This should only be taken when the size of the debug
  package gets *huge* otherwise. It is hard to set definite numbers here,
  but a 5mb package would surely not be a reason to split into two 2.5mb
  ones.
  
  This fails to address the rather valid concern brought up about
  having different versions of libraries and binaries installed
  from the same source package.  Having one .ddeb per binary would
  solve this elegantly.
 
 Except that in that case, the old library will be NBS and thus I see no point
 why you would want to keep it installed. The only reason would be if it was
 meant to stay around, but in that case I'm sure the source package names would
 be different.

The scenario has been described already in the thread.

  It's also rather space-inefficient for the user.
 
 If that bothers you, you can use the share we plan to provide.

No thanks, I like my debug symbols in a nice convenient packaged
format, signed by the archive admins.

While you might plan to be providing some fancy (yet enigmatic) service
based upon the debug deb content, I still want them installable,
preferably automatically getting all dependent debug symbols as well
using apt.  Preferably with a .deb extension; I see no reason why they
can't be first-class Debian packages in addition to being used for
mysterious as-yet-unspecified purposes.

I can't say I'm particularly enthused with the apparent lack of
consideration for most of the issues with the proposal brought up in
this thread.


Regards,
Roger

-- 
  .''`.  Roger Leigh
 : :' :  Debian GNU/Linux http://people.debian.org/~rleigh/
 `. `'   Printing on GNU/Linux?   http://gutenprint.sourceforge.net/
   `-GPG Public Key: 0x25BFB848   Please GPG sign your mail.


signature.asc
Description: Digital signature

Re: Automatic Debug Packages

[Russ Allbery]

Emilio Pozuelo Monfort poch...@gmail.com writes:
 Russ Allbery wrote:

 Except that in that case, the old library will be NBS and thus I see
 no point why you would want to keep it installed. The only reason
 would be if it was meant to stay around, but in that case I'm sure the
 source package names would be different.

 Because you're trying to debug a binary on your system that's linked
 against it.

 Then you should work on making your package build with the new library,
 since it will be FTBFS anyway :)

I was not under the impression that this system was intended only for the
use of Debian Developers.

 I don't consider this a real issue.

I do.  I think it's a fairly significant one.

 There will still be a repository with all the .ddebs. But also we will
 have a share that will ship all the debugging symbols in a build id file
 hierarchy structure (so something like .build-id/xx/xxx.debug). You
 can mount it in your system and use it as if you had installed every
 -ddeb available in the archive. Furthermore, if disk space permits it,
 we will provide symbols for more than one version (i.e. not only the
 current package in the archive, but maybe the last three or whatever we
 can do), since build ids permit it.

 With this in place, we can integrate reporting tools (bug-buddy,
 drkonqi, apport) to use that service to magically provide meaningful bug
 reports with complete backtraces.

Hm, that's interesting, but I suspect that few enough of our users will be
able to use such a thing that we shouldn't let that influence any other
design choices.  Most shares are not going to be able to be mounted
through firewalls, for example, so that form of the debug symbols won't be
available to quite a few users.

Or maybe by share you meant something that was more like a file download
service over HTTP than, say, NFS?

 If you use this, you won't need to get a backtrace, realize you're
 missing some symbols, install some more debug packages, rinse,
 repeat... :)

I thought you were planning on automating *this*, which I think is more
useful than providing a share.  It seems like it would be fairly
straightforward in conjunction with the Contents database and the ldd
output on the binary that crashed.

-- 
Russ Allbery (r...@debian.org)   http://www.eyrie.org/~eagle/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Manoj Srivastava]

On Wed, Aug 12 2009, Emilio Pozuelo Monfort wrote:


 There will still be a repository with all the .ddebs.

And aptitude and dpkg will know how to install ddebs, somehow?
 and synaptic, etc?

 But also we will have a share that will ship all the debugging symbols
 in a build id file hierarchy structure (so something like
 .build-id/xx/xxx.debug). You can mount it in your system and use
 it as if you had installed every -ddeb available in the
 archive.

So all debugging has to happen online? Many places have a
 prohibition against mounting a file system from outside the firewall,
 though installing packages that have been vetted is permissible

. 

 Furthermore, if disk space permits it, we will provide symbols for
 more than one version (i.e. not only the current package in the
 archive, but maybe the last three or whatever we can do), since build
 ids permit it.

With packages, mirrored repositories may have a different
 retention policy, and not rely on the packages one has installed
 disappearing off the remote filesystem.

The system you propose works great for the use case you have
 envisaged: Debugging on a low security instllation with always on
 broadband connection to the network; but surely that is not the only
 model we provide.

I am also wondering about the obstacles placed in the path of
 packages that will not be covered by the automated system. While we
 were  talking about generating .debs, that was some work, but not any
 different from creating a package.  Now, in order to create a hand
 crafted ddeb, what does one do? Add a nerw package to debian/control,
 build it, rename the package, edit ./debian/files before
 dpkg-genchanges is called  -- this is more complex than just calling
 dpkg-buildpackage and be done with it.

So I am wondering if the selction by package name is not good
 enough, and  whether we really need selection using *.ddeb$  --
  /.*-ddeb_.*\.deb$/ is not that much more complex a regular expression,
  and it brings with it the ability to  manually create the debug symbol
  packages easily, using dpkg-bvuildpackage.

I too am wondering if we should defer the polivy change until
 the details get shaken out with a partial deployment of the scheme.


manoj
-- 
Don't tell me I'm burning the candle at both ends -- tell me where to
get more wax!!
Manoj Srivastava sriva...@debian.org http://www.debian.org/~srivasta/  
1024D/BF24424C print 4966 F272 D093 B493 410B  924B 21BA DABB BF24 424C


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Paul Wise]

On Thu, Aug 13, 2009 at 10:10 AM, Manoj Srivastavasriva...@debian.org wrote:

        I too am wondering if we should defer the polivy change until
  the details get shaken out with a partial deployment of the scheme.

Full deployment already happened (in Ubuntu).

-- 
bye,
pabs

http://wiki.debian.org/PaulWise


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Russ Allbery]

Paul Wise p...@debian.org writes:
 On Thu, Aug 13, 2009 at 10:10 AM, Manoj Srivastavasriva...@debian.org wrote:

        I too am wondering if we should defer the polivy change until
  the details get shaken out with a partial deployment of the scheme.

 Full deployment already happened (in Ubuntu).

As .ddebs?  What's the policy about what can go in them and how are they
integrated with the packaging tools?  And could you point me at the Ubuntu
share for the debugging information so that I can see what protocols it's
using?

Prior experience is *great*.  We can learn from the experiences of Ubuntu.

-- 
Russ Allbery (r...@debian.org)   http://www.eyrie.org/~eagle/


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Paul Wise]

On Thu, Aug 13, 2009 at 11:25 AM, Russ Allberyr...@debian.org wrote:

 As .ddebs?  What's the policy about what can go in them and how are they
 integrated with the packaging tools?  And could you point me at the Ubuntu
 share for the debugging information so that I can see what protocols it's
 using?

 Prior experience is *great*.  We can learn from the experiences of Ubuntu.

Not having anything to do with Ubuntu, I don't know anything about the
details, but they have had automatic debug packages and automated
crash report stuff for quite a while, a couple of years IIRC. The
specs for that are here:

https://launchpad.net/ubuntu/+spec/apt-get-debug-symbols
https://launchpad.net/distros/ubuntu/+spec/automated-problem-reports

-- 
bye,
pabs

http://wiki.debian.org/PaulWise


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Russ Allbery]

Paul Wise p...@debian.org writes:

 Not having anything to do with Ubuntu, I don't know anything about the
 details, but they have had automatic debug packages and automated
 crash report stuff for quite a while, a couple of years IIRC. The
 specs for that are here:

 https://launchpad.net/ubuntu/+spec/apt-get-debug-symbols
 https://launchpad.net/distros/ubuntu/+spec/automated-problem-reports

Ah, thank you!

https://wiki.ubuntu.com/AptElfDebugSymbols is the specification.  It does
use *.ddeb.  There isn't any clear statement about how *.ddeb packages
differ from *.deb packages.  It looks like, by and large, they don't,
except they may not need to contain the same set of things.  The packages
are not in debian/control or the things fed from it, but are in *.changes.

Ubuntu is creating one debug package per binary package, as I and a few
other people have said we prefer.  However, they're using the
gnu_debuglink method, not the id method, so they don't have the one
problem with that method previously identified in this thread when the
same binary is copied into multiple packages.

The debug packages depend on the packages for which they have symbols,
which solves the problem of not installing debug packages that both
provide symbols for the same binary.

The proposal appears to only work for packages using debhelper, although
the steps are laid out so presumably they could be done manually if need
be.

Ubuntu uses -dbgsym as the magic namespace.  I don't know if it would be
good for us to do the same or to use a different namespace to avoid
problems for them in cases where we decide to build the package manually
via debian/control and debian/rules.

It looks like the plan with *.ddeb is to treat them specially in the
archive software and divert them into a different tree in the archive
instead of using a separate archive area, which I think they're doing
now from that page.  It also looks like one of the justifications for the
separate package is to hide them in the apt front-end so that users don't
see all the additional packages.  I'm personally skeptical this is a good
idea, although I can see the merits of not returning them in apt-cache
search.

Ah, and it looks like the automated crash reporting offers to download the
-dbgsym packages and install them.

I don't see any sign in this of a share.

-- 
Russ Allbery (r...@debian.org)   http://www.eyrie.org/~eagle/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Manoj Srivastava]

On Wed, Aug 12 2009, Russ Allbery wrote:

 Paul Wise p...@debian.org writes:
 On Thu, Aug 13, 2009 at 10:10 AM, Manoj Srivastavasriva...@debian.org 
 wrote:

        I too am wondering if we should defer the polivy change until
  the details get shaken out with a partial deployment of the scheme.

 Full deployment already happened (in Ubuntu).

 As .ddebs?  What's the policy about what can go in them and how are they
 integrated with the packaging tools?  And could you point me at the Ubuntu
 share for the debugging information so that I can see what protocols it's
 using?

Did ubuntu have to modify the default package manager (synaptic,
 right?) in order to allow the user to install the ddebs locally? I
 would be interested in the details of how to hook up to the debug
 packages archive in Ubuntu (I shall try installing an Ubuntu virtual
 machine this weekend to try this out).

 Prior experience is *great*.  We can learn from the experiences of Ubuntu.

I would also like to learn of the coverage Ubuntu managed to
 achieve, given that they have full deployment. What is the percentage of
 packages they managed to auto create?  This is indeed very valuable
 information, I  am surprised no one has been mentioning any figures at
 all about this full deployment in Ubuntu.

manoj
-- 
A mind is a terrible thing to have leaking out your ears. The League
of Sadistic Telepaths
Manoj Srivastava sriva...@debian.org http://www.debian.org/~srivasta/  
1024D/BF24424C print 4966 F272 D093 B493 410B  924B 21BA DABB BF24 424C


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Manoj Srivastava]

On Wed, Aug 12 2009, Russ Allbery wrote:

 Paul Wise p...@debian.org writes:

 Not having anything to do with Ubuntu, I don't know anything about the
 details, but they have had automatic debug packages and automated
 crash report stuff for quite a while, a couple of years IIRC. The
 specs for that are here:

 https://launchpad.net/ubuntu/+spec/apt-get-debug-symbols 
 https://launchpad.net/distros/ubuntu/+spec/automated-problem-reports

 Ah, thank you!

 https://wiki.ubuntu.com/AptElfDebugSymbols is the specification.  It
 does use *.ddeb.  There isn't any clear statement about how *.ddeb
 packages differ from *.deb packages.  It looks like, by and large,
 they don't, except they may not need to contain the same set of
 things.  The packages are not in debian/control or the things fed from
 it, but are in *.changes.

Thanks for the link.

They mention:
--8---cut here---start-8---
# They (ddebs)  can be arranged in a proper pool structure with a
  Packages file etc., so that existing tools to mirror, download, and
  ship debs can be reused.  
# Users can actually install them if they want to. 

 ...

An apt frontend will be provided to conveniently install debug symbols:
e. g. apt-get debug apache2 would install the -dbgsym ddebs for apache2
and all its dependencies. This will allow developers to actually install
the .ddebs 
--8---cut here---end---8---


This is what I find interesting. So, not quite
 aptitude/synhaptic, but analogous to apt-get source. This does answer
 some of the questions I had about implementation.

 Ubuntu is creating one debug package per binary package, as I and a few
 other people have said we prefer.  However, they're using the
 gnu_debuglink method, not the id method, so they don't have the one
 problem with that method previously identified in this thread when the
 same binary is copied into multiple packages.

Or the facility to keep debug symbols around for multiple
 versions of shared libraries (with the same soname), which is an
 advantage the build id method has.

 The proposal appears to only work for packages using debhelper, although
 the steps are laid out so presumably they could be done manually if need
 be.

Yes, though some sleight of hand might be needed do build a
 package which is not in control (dpkg-deb --nocheck), or if the package
 is in debian/control, debian/files would have to have the .deb line
 removed, and dpkg-distaddfile used to register the ddeb file name).

 Ubuntu uses -dbgsym as the magic namespace.  I don't know if it would be
 good for us to do the same or to use a different namespace to avoid
 problems for them in cases where we decide to build the package manually
 via debian/control and debian/rules.

I would still like to see coverage figures to figure out what
 percentage of the archive will need this.


 It looks like the plan with *.ddeb is to treat them specially in the
 archive software and divert them into a different tree in the archive
 instead of using a separate archive area, which I think they're doing
 now from that page.  It also looks like one of the justifications for
 the separate package is to hide them in the apt front-end so that
 users don't see all the additional packages.  I'm personally skeptical
 this is a good idea, although I can see the merits of not returning
 them in apt-cache search.

I am not sure I see that. When I ask apt cache for packages that
 currently have a debug package, I do not see the rpesence of
 information about the debug package as garbage; it is nice to know that
 the debg information exists.


 Ah, and it looks like the automated crash reporting offers to download the
 -dbgsym packages and install them.

Reading the spec, it seems to me that the primary motivation was
 for users to provide crash dumps with bug reports, and not much screen
 time is given to users debugging their own applications linked to
 vendor libraries, or for the developer user  in general. I think that
 use case should be addressed as well.

 I don't see any sign in this of a share.

I am not sure I see much utility in a share, personally, since I
 have not really had an installation where I spent any time in where the
 mount would not have been prevented by perimeter defenses and security
 policies.

manoj
-- 
Help save the world!  --Larry Wall in README
Manoj Srivastava sriva...@debian.org http://www.debian.org/~srivasta/  
1024D/BF24424C print 4966 F272 D093 B493 410B  924B 21BA DABB BF24 424C


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Sune Vuorela]

On 2009-08-10, Manoj Srivastava sriva...@debian.org wrote:
 On Mon, Aug 10 2009, Sune Vuorela wrote:

 On 2009-08-10, Manoj Srivastava sriva...@debian.org wrote:
 I would also add that the debug symbols should live in
  /usr/lib/debug/ . /full/path/to/lib_or_binary, blessing the current
  practice.

 You are missing the new features of build-id as written earlier by
 insisting on this.

 Please, Do enlighten me.

From earlier in this thread:

|instead of objcopy and stuff people current do, either manually or with
|the help of dh_strip and wrap it in a foo-dbg.deb by using the
|.gnu_debuglink to and the expected paths by gdb and friends on where to
|find it, instead use the newfangled build id and put it in a appropriate
|hashed directory structure, as gdb and friends also are able to pick up,
|and then wrap it in a foo.ddeb package.
|
|There is nothing actually magic in it, except making it easy to use, but
|for more technical details, 
|see the --build-id option to ld(1) for more details about what build-id is
|and the gdb manual chapter 15.2 for more info about debugging
|information in seperate files.

http://permalink.gmane.org/gmane.linux.debian.devel.general/143053

/Sune


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Manoj Srivastava]

On Tue, Aug 11 2009, Sune Vuorela wrote:

 On 2009-08-10, Manoj Srivastava sriva...@debian.org wrote:
 On Mon, Aug 10 2009, Sune Vuorela wrote:

 On 2009-08-10, Manoj Srivastava sriva...@debian.org wrote:
 I would also add that the debug symbols should live in
  /usr/lib/debug/ . /full/path/to/lib_or_binary, blessing the current
  practice.

 You are missing the new features of build-id as written earlier by
 insisting on this.

Hmm. I see very little benefit here. Firstly, to use build id,
 you have to intercept the upstream build system and add --build-id
 (and perhaps the --build-id-style) option to ld, instead of the current
 method of letting the upstream build happen and working on the produced
 objects -- this is more intrusive.  And what do we gain?

  * For the build ID method, GDB looks in the `.build-id'
 subdirectory of the global debug directory for a file named
 `NN/.debug', where NN are the first 2 hex characters of
 the build ID bit string, and  are the rest of the bit
 string.  (Real build ID strings are 32 or more hex characters, not
 10.)

So, we would still need to create   /usr/lib/debug/
 . /full/path/to/lib_or_binary/ in either case, and instead of the
 lib-or-exec name, create NN/.debug file there (which is non
 human readable, really). What have we gained? There is no potential for
 file name conflicts, since if there are file name conflicts in the
 debug symbol files, there would be file name conflicts in the
 corresponding packages, which is already a bug in Debian.

I see added complexity with no real benefit for us: it might
 have value in an environment where file conflicts are not verboten.

manoj
-- 
An apology for the devil: it must be remembered that we have heard only
one side of the case. God has written all the books.
Manoj Srivastava sriva...@debian.org http://www.debian.org/~srivasta/  
1024D/BF24424C print 4966 F272 D093 B493 410B  924B 21BA DABB BF24 424C


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Sune Vuorela]

On 2009-08-11, Manoj Srivastava sriva...@debian.org wrote:
 Hmm. I see very little benefit here. Firstly, to use build id,
  you have to intercept the upstream build system and add --build-id
  (and perhaps the --build-id-style) option to ld, instead of the current
  method of letting the upstream build happen and working on the produced
  objects -- this is more intrusive.  And what do we gain?

The plan is to make --build-id the default (As it is on many other
distributions).


   * For the build ID method, GDB looks in the `.build-id'
  subdirectory of the global debug directory for a file named
  `NN/.debug', where NN are the first 2 hex characters of
  the build ID bit string, and  are the rest of the bit
  string.  (Real build ID strings are 32 or more hex characters, not
  10.)

 So, we would still need to create   /usr/lib/debug/
  . /full/path/to/lib_or_binary/ in either case, and instead of the

no. it would be /usr/lib/debug/.build-id/NN/NN.debug

  lib-or-exec name, create NN/.debug file there (which is non
  human readable, really). What have we gained? There is no potential for
  file name conflicts, since if there are file name conflicts in the
  debug symbol files, there would be file name conflicts in the
  corresponding packages, which is already a bug in Debian.

 I see added complexity with no real benefit for us: it might
  have value in an environment where file conflicts are not verboten.

And the next step is to provide /usr/lib/debug/.build-id exported from
some internet service so that you download debugging symbols on demand,
for example thru drkonqi or bug-buddy or maybe directly with gdb.

Having a reliable way of getting from a random library version and
random executable version to get the exact corresponding debug symbols,
the build-id method is just much more reliable.

/Sune


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Roger Leigh]

On Tue, Aug 11, 2009 at 01:40:20PM +, Sune Vuorela wrote:
 On 2009-08-11, Manoj Srivastava sriva...@debian.org wrote:
  So, we would still need to create   /usr/lib/debug/
   . /full/path/to/lib_or_binary/ in either case, and instead of the
 
 no. it would be /usr/lib/debug/.build-id/NN/NN.debug
 ^
Why the need for a hidden directory in a public location?
What's wrong with /usr/lib/debug/build-id?

I don't think the unnecessary obfuscation is warranted.


Regards,
Roger

-- 
  .''`.  Roger Leigh
 : :' :  Debian GNU/Linux http://people.debian.org/~rleigh/
 `. `'   Printing on GNU/Linux?   http://gutenprint.sourceforge.net/
   `-GPG Public Key: 0x25BFB848   Please GPG sign your mail.


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Josselin Mouette]

Le mardi 11 août 2009 à 08:24 -0500, Manoj Srivastava a écrit :
 Hmm. I see very little benefit here. Firstly, to use build id,
  you have to intercept the upstream build system and add --build-id
  (and perhaps the --build-id-style) option to ld, instead of the current
  method of letting the upstream build happen and working on the produced
  objects -- this is more intrusive.  And what do we gain?

Without build IDs, GDB has no sure way to map the binary to the correct
detached symbols. Therefore it will read the whole file to compute its
CRC32 (!) and compare it to the one stored in the gnu_debuglink section
of the binary.

This sole issue is responsible for gdb taking up to several minutes to
produce a backtrace for binaries using big libraries like xulrunner. And
don’t even think of using the debugging symbols over the network in this
case.

-- 
 .''`.  Josselin Mouette
: :' :
`. `'   “I recommend you to learn English in hope that you in
  `- future understand things”  -- Jörg Schilling


signature.asc
Description: Ceci est une partie de message numériquement signée

Re: Automatic Debug Packages

[Manoj Srivastava]

On Tue, Aug 11 2009, Roger Leigh wrote:

 On Tue, Aug 11, 2009 at 01:40:20PM +, Sune Vuorela wrote:
 On 2009-08-11, Manoj Srivastava sriva...@debian.org wrote:
  So, we would still need to create   /usr/lib/debug/
   . /full/path/to/lib_or_binary/ in either case, and instead of the
 
 no. it would be /usr/lib/debug/.build-id/NN/NN.debug
  ^
 Why the need for a hidden directory in a public location?
 What's wrong with /usr/lib/debug/build-id?

 I don't think the unnecessary obfuscation is warranted.

Because that is where gdb looks for them.

manoj
-- 
Probably the earliest fly swatters were nothing more than some sort of
striking surface attached to the end of a long stick.
Manoj Srivastava sriva...@debian.org http://www.debian.org/~srivasta/  
1024D/BF24424C print 4966 F272 D093 B493 410B  924B 21BA DABB BF24 424C


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Manoj Srivastava]

On Tue, Aug 11 2009, Sune Vuorela wrote:

 On 2009-08-11, Manoj Srivastava sriva...@debian.org wrote:
 Hmm. I see very little benefit here. Firstly, to use build id,
  you have to intercept the upstream build system and add --build-id
  (and perhaps the --build-id-style) option to ld, instead of the current
  method of letting the upstream build happen and working on the produced
  objects -- this is more intrusive.  And what do we gain?

 The plan is to make --build-id the default (As it is on many other
 distributions).


   * For the build ID method, GDB looks in the `.build-id'
  subdirectory of the global debug directory for a file named
  `NN/.debug', where NN are the first 2 hex characters of
  the build ID bit string, and  are the rest of the bit
  string.  (Real build ID strings are 32 or more hex characters, not
  10.)

 So, we would still need to create   /usr/lib/debug/
  . /full/path/to/lib_or_binary/ in either case, and instead of the

 no. it would be /usr/lib/debug/.build-id/NN/NN.debug

Right. Mostly human unreadable.

  lib-or-exec name, create NN/.debug file there (which is non
  human readable, really). What have we gained? There is no potential for
  file name conflicts, since if there are file name conflicts in the
  debug symbol files, there would be file name conflicts in the
  corresponding packages, which is already a bug in Debian.

 I see added complexity with no real benefit for us: it might
  have value in an environment where file conflicts are not verboten.

 And the next step is to provide /usr/lib/debug/.build-id exported from
 some internet service so that you download debugging symbols on demand,
 for example thru drkonqi or bug-buddy or maybe directly with gdb.

You can just export /usr/lib/debug/ from the central service
 equally easily, no difference there -- you can just provide the current
 Sid/testing/stable snapshot.


 Having a reliable way of getting from a random library version and
 random executable version to get the exact corresponding debug symbols,
 the build-id method is just much more reliable.

Except you have not indicated how you (or debhelper) is going to
 intercept ld to add the requisite arguments.

manoj
-- 
Murray's Rule: Any country with democratic in the title isn't.
Manoj Srivastava sriva...@debian.org http://www.debian.org/~srivasta/  
1024D/BF24424C print 4966 F272 D093 B493 410B  924B 21BA DABB BF24 424C


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Josselin Mouette]

Le mardi 11 août 2009 à 10:11 -0500, Manoj Srivastava a écrit :
 Except you have not indicated how you (or debhelper) is going to
  intercept ld to add the requisite arguments.

http://lists.debian.org/debian-devel-changes/2009/07/msg01229.html

-- 
 .''`.  Josselin Mouette
: :' :
`. `'   “I recommend you to learn English in hope that you in
  `- future understand things”  -- Jörg Schilling


signature.asc
Description: Ceci est une partie de message numériquement signée

Re: Automatic Debug Packages

[Emilio Pozuelo Monfort]

Steve Langasek wrote:
 On Mon, Aug 10, 2009 at 09:46:49PM +0100, Roger Leigh wrote:
 Reading through this thread, I don't see a compelling reason for using
 a .ddeb extension given that they are just regular .debs, nor for
 keeping the packages separate from the main archive (if the size of the
 Packages file is an issue, can't they just go in a separate debug
 section/component?)
 
 Size of *the archive* is an issue.  They could live in a separate component,
 but as mentioned in the proposal, this component shouldn't by default be
 mirrored with the rest of the archive.

What's exactly what we will do!

Emilio



signature.asc
Description: OpenPGP digital signature

Re: Automatic Debug Packages

[Manoj Srivastava]

On Tue, Aug 11 2009, Josselin Mouette wrote:

 Le mardi 11 août 2009 à 08:24 -0500, Manoj Srivastava a écrit :
 Hmm. I see very little benefit here. Firstly, to use build id,
  you have to intercept the upstream build system and add --build-id
  (and perhaps the --build-id-style) option to ld, instead of the current
  method of letting the upstream build happen and working on the produced
  objects -- this is more intrusive.  And what do we gain?

 Without build IDs, GDB has no sure way to map the binary to the correct
 detached symbols. Therefore it will read the whole file to compute its
 CRC32 (!) and compare it to the one stored in the gnu_debuglink section
 of the binary.

 This sole issue is responsible for gdb taking up to several minutes to
 produce a backtrace for binaries using big libraries like xulrunner. And
 don’t even think of using the debugging symbols over the network in this
 case.

Yes, that would indeed be silly -- if you have managed to
 intercept ld  and added --build-id to the executable, it would be silly
 not to have the file in the location gdb will look in.

However, if you do not use the build-id mechanism, and use what
 we currently use in dh_strip and friends, objcopy --add-gnu-debuglink
 adds information that gdb looks at to figure out where the debug
 symbols live -- and no CRC check sum is ever performed. 

So a mixed mode approach would indeed be bad. But a pure debug
 link method does not have these stated drawbacks.

Given the difficulty in intercepting ld commands in upstream
 build systems, I would  be inclined to just standardize the debug link
 method, given that it produces human readable file names (so I can
 determine manually if I have debugging symbols for some library or
 not) as an added bonus. 

manoj
-- 
Work is the crab grass in the lawn of life. Schulz
Manoj Srivastava sriva...@debian.org http://www.debian.org/~srivasta/  
1024D/BF24424C print 4966 F272 D093 B493 410B  924B 21BA DABB BF24 424C


--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Emilio Pozuelo Monfort]

Josselin Mouette wrote:
 Le mardi 11 août 2009 à 10:11 -0500, Manoj Srivastava a écrit :
 Except you have not indicated how you (or debhelper) is going to
  intercept ld to add the requisite arguments.
 
 http://lists.debian.org/debian-devel-changes/2009/07/msg01229.html

Also see http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=535237 for reference.

Emilio



signature.asc
Description: OpenPGP digital signature

Re: Automatic Debug Packages

[Josselin Mouette]

Le mardi 11 août 2009 à 10:39 -0500, Manoj Srivastava a écrit :
 However, if you do not use the build-id mechanism, and use what
  we currently use in dh_strip and friends, objcopy --add-gnu-debuglink
  adds information that gdb looks at to figure out where the debug
  symbols live -- and no CRC check sum is ever performed. 

As I explained, the CRC checksum is performed unconditionally when the
gnu_debuglink mechanism is used.

 Given the difficulty in intercepting ld commands in upstream
  build systems, I would  be inclined to just standardize the debug link
  method, given that it produces human readable file names (so I can
  determine manually if I have debugging symbols for some library or
  not) as an added bonus. 

Providing a script looking at the build-id and telling whether the
debugging symbols are installed is a matter of a few lines of shell
code.

-- 
 .''`.  Josselin Mouette
: :' :
`. `'   “I recommend you to learn English in hope that you in
  `- future understand things”  -- Jörg Schilling


signature.asc
Description: Ceci est une partie de message numériquement signée

Re: Automatic Debug Packages

[Emilio Pozuelo Monfort]

Manoj Srivastava wrote:
 On Tue, Aug 11 2009, Josselin Mouette wrote:
 
 Le mardi 11 août 2009 à 08:24 -0500, Manoj Srivastava a écrit :
 Hmm. I see very little benefit here. Firstly, to use build id,
  you have to intercept the upstream build system and add --build-id
  (and perhaps the --build-id-style) option to ld, instead of the current
  method of letting the upstream build happen and working on the produced
  objects -- this is more intrusive.  And what do we gain?
 Without build IDs, GDB has no sure way to map the binary to the correct
 detached symbols. Therefore it will read the whole file to compute its
 CRC32 (!) and compare it to the one stored in the gnu_debuglink section
 of the binary.

 This sole issue is responsible for gdb taking up to several minutes to
 produce a backtrace for binaries using big libraries like xulrunner. And
 don’t even think of using the debugging symbols over the network in this
 case.
 
 Yes, that would indeed be silly -- if you have managed to
  intercept ld  and added --build-id to the executable, it would be silly
  not to have the file in the location gdb will look in.
 
 However, if you do not use the build-id mechanism, and use what
  we currently use in dh_strip and friends, objcopy --add-gnu-debuglink
  adds information that gdb looks at to figure out where the debug
  symbols live -- and no CRC check sum is ever performed. 

It still is AFAIK:

   * The executable contains a debug link that specifies the name of
 the separate debug info file.  The separate debug file's name is
 usually `EXECUTABLE.debug', where EXECUTABLE is the name of the
 corresponding executable file without leading directories (e.g.,
 `ls.debug' for `/usr/bin/ls').  In addition, the debug link
 specifies a CRC32 checksum for the debug file, which GDB uses to
 validate that the executable and the debug file came from the same
 build.

That says the debug link contains the CRC checksum. But GDB still needs to
perform it to make sure it matches that of the debug file.

Cheers,
Emilio



signature.asc
Description: OpenPGP digital signature

Re: Automatic Debug Packages

[Manoj Srivastava]

On Tue, Aug 11 2009, Josselin Mouette wrote:

 Le mardi 11 août 2009 à 10:11 -0500, Manoj Srivastava a écrit :
 Except you have not indicated how you (or debhelper) is going to
  intercept ld to add the requisite arguments.

 http://lists.debian.org/debian-devel-changes/2009/07/msg01229.html

So the proposal is to add --enable-linker-build-id to CFLAGS?

OK, I guess that would work. But you still have the advantage,
 using the current debug link mechanism, of looking to see if you have
 debug symbols for a given executable/library easily, without having to
 compute potentially 3 checksums (depending on which algorithm was
 selected at build time) and trying to match that (in multiple
 directories).

Can you  point ot me the disadvantage of continuing to use what
 dh_strip does now?

manoj
-- 
I have the simplest tastes.  I am always satisfied with the best. Oscar
Wilde
Manoj Srivastava sriva...@debian.org http://www.debian.org/~srivasta/  
1024D/BF24424C print 4966 F272 D093 B493 410B  924B 21BA DABB BF24 424C


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Manoj Srivastava]

Hi,

All right. Having been educated about the new build-id
 mechanism, I think there is not reason for policy to prohibit either
 approach, or to settle on one or the other.

To recap:
 1) packages with detached debugging symbols should be named
${package name}-${debug suffix}. As a corollary, no ordinary
packages names may end in  ${debug suffix}.
 2) These packages may just symlink
/usr/share/doc/${package name}-${debug suffix} to
/usr/share/doc/${package name}
(and of course, depend on ${package name}
 3) The detached debugging symbols should be placed in a subdirectory of
/usr/lib/debug, the exact path being determined by the mechanism
used (either build ids or debug links can and may be used)
 4) Otherwise, these packages are normal debian packages

The  ${debug suffix} nay be ddeb,  or something else to be
 decided.

Comments?

manoj
-- 
Well, it's hard for a mere man to believe that woman doesn't have equal
rights. Dwight D. Eisenhower
Manoj Srivastava sriva...@debian.org http://www.debian.org/~srivasta/  
1024D/BF24424C print 4966 F272 D093 B493 410B  924B 21BA DABB BF24 424C


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Emilio Pozuelo Monfort]

Manoj Srivastava wrote:
 On Tue, Aug 11 2009, Josselin Mouette wrote:
 
 Le mardi 11 août 2009 à 10:11 -0500, Manoj Srivastava a écrit :
 Except you have not indicated how you (or debhelper) is going to
  intercept ld to add the requisite arguments.
 http://lists.debian.org/debian-devel-changes/2009/07/msg01229.html
 
 So the proposal is to add --enable-linker-build-id to CFLAGS?

That is passed to the GCC build, so that from now on GCC will always pass
--build-id to the linker. So you don't need to do anything if your package uses
GCC (gcc/g++).

 OK, I guess that would work. But you still have the advantage,
  using the current debug link mechanism, of looking to see if you have
  debug symbols for a given executable/library easily, without having to
  compute potentially 3 checksums (depending on which algorithm was
  selected at build time) and trying to match that (in multiple
  directories).

If you have the .ddeb package installed, you will have the debugging symbols
installed :)

 Can you  point ot me the disadvantage of continuing to use what
  dh_strip does now?

It can still be used, but you will miss the advantages of using build ids.

Emilio



signature.asc
Description: OpenPGP digital signature

Re: Automatic Debug Packages

[Emilio Pozuelo Monfort]

Manoj Srivastava wrote:
 Hi,
 
 All right. Having been educated about the new build-id
  mechanism, I think there is not reason for policy to prohibit either
  approach, or to settle on one or the other.
 
 To recap:
  1) packages with detached debugging symbols should be named
 ${package name}-${debug suffix}. As a corollary, no ordinary
 packages names may end in  ${debug suffix}.

They may be automatically created. They may also be manually created (if they
are listed in debian/control, so for complex packages where they need some
manual work, it can be done).

  2) These packages may just symlink
 /usr/share/doc/${package name}-${debug suffix} to
 /usr/share/doc/${package name}
 (and of course, depend on ${package name}
  3) The detached debugging symbols should be placed in a subdirectory of
 /usr/lib/debug, the exact path being determined by the mechanism
 used (either build ids or debug links can and may be used)

Don't forget that there are other debug info, like e.g. python dbg extensions or
mono .mdb files. Those aren't placed in /usr/lib/debug, so we shouldn't restrict
the ddeb packages content to files in /usr/lib/debug.

  4) Otherwise, these packages are normal debian packages

5) There may only be one ddeb per source package (if more where needed, we could
consider it).


Cheers,
Emilio



signature.asc
Description: OpenPGP digital signature

Re: Automatic Debug Packages

[Julien Cristau]

On Tue, Aug 11, 2009 at 18:37:05 +0200, Emilio Pozuelo Monfort wrote:

 Manoj Srivastava wrote:
   2) These packages may just symlink
  /usr/share/doc/${package name}-${debug suffix} to
  /usr/share/doc/${package name}
  (and of course, depend on ${package name}
 
 5) There may only be one ddeb per source package (if more where needed, we 
 could
 consider it).
 
How do you make 2) and 5) work?  Seems to me one ddeb per binary package
makes much more sense.

Cheers,
Julien


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Bill Allombert]

On Mon, Aug 10, 2009 at 03:59:22PM -0700, Steve Langasek wrote:
 On Mon, Aug 10, 2009 at 09:46:49PM +0100, Roger Leigh wrote:
  Reading through this thread, I don't see a compelling reason for using
  a .ddeb extension given that they are just regular .debs, nor for
  keeping the packages separate from the main archive (if the size of the
  Packages file is an issue, can't they just go in a separate debug
  section/component?)
 
 Size of *the archive* is an issue.  They could live in a separate component,
 but as mentioned in the proposal, this component shouldn't by default be
 mirrored with the rest of the archive.

The size of the archive has been used to justify so much silliness that
it has become the Godwin law of Debian.

Debian mirrors are not required to have non-free/* so they should not
be required to have */debug either.

Cheers,
-- 
Bill. ballo...@debian.org

Imagine a large red swirl here. 


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Manoj Srivastava]

On Tue, Aug 11 2009, Emilio Pozuelo Monfort wrote:

 Manoj Srivastava wrote:


 OK, I guess that would work. But you still have the advantage,
  using the current debug link mechanism, of looking to see if you have
  debug symbols for a given executable/library easily, without having to
  compute potentially 3 checksums (depending on which algorithm was
  selected at build time) and trying to match that (in multiple
  directories).

 If you have the .ddeb package installed, you will have the debugging symbols
 installed :)

I guess that is true. Figure out which package the executable
 belongs to, check to see that the -ddeb package exists.


 Can you  point ot me the disadvantage of continuing to use what
  dh_strip does now?

 It can still be used, but you will miss the advantages of using build ids.

I guess I was trying to ask what the advantages were, apart from
 the CRC check overhead that is skipped on load.  I presume that the crc
 check sum has been demonstrated to be onerous. Are there any other
 advantages I am missing?

manoj
-- 
innovate, v.: To annoy people.
Manoj Srivastava sriva...@debian.org http://www.debian.org/~srivasta/  
1024D/BF24424C print 4966 F272 D093 B493 410B  924B 21BA DABB BF24 424C


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Emilio Pozuelo Monfort]

Manoj Srivastava wrote:
 On Tue, Aug 11 2009, Emilio Pozuelo Monfort wrote:
 Manoj Srivastava wrote:
 Can you  point ot me the disadvantage of continuing to use what
  dh_strip does now?
 It can still be used, but you will miss the advantages of using build ids.
 
 I guess I was trying to ask what the advantages were, apart from
  the CRC check overhead that is skipped on load.  I presume that the crc
  check sum has been demonstrated to be onerous. Are there any other
  advantages I am missing?

We plan to provide a share through the network, that you can mount to virtually
have all the debugging symbols for the archive, downloading them as needed.

With build ids, assuming we have enough disk space, we could ship debugging
symbols for more than one version at a time, and the right one would be picked.
With the debuglink method, this wouldn't be possible as the files would need to
be shipped in the same place.

Cheers,
Emilio



signature.asc
Description: OpenPGP digital signature

Re: Automatic Debug Packages

[Russ Allbery]

Emilio Pozuelo Monfort poch...@gmail.com writes:
 Manoj Srivastava wrote:

 To recap:
  1) packages with detached debugging symbols should be named
 ${package name}-${debug suffix}. As a corollary, no ordinary
 packages names may end in  ${debug suffix}.

 They may be automatically created. They may also be manually created (if
 they are listed in debian/control, so for complex packages where they
 need some manual work, it can be done).

Whether they're automatically or manually created is irrelevant for Debian
Policy.  Policy describes what the output should be, not what tools one
uses to get there.

I think the relevant question for Policy is whether these packages will be
listed in debian/control in the source package, in Binary in the *.dsc
file, and in Binary/Files/Checksums-* in the *.changes file.  And I don't
know the answer to those three questions from the discussion so far.

  3) The detached debugging symbols should be placed in a subdirectory of
 /usr/lib/debug, the exact path being determined by the mechanism
 used (either build ids or debug links can and may be used)

 Don't forget that there are other debug info, like e.g. python dbg
 extensions or mono .mdb files. Those aren't placed in /usr/lib/debug, so
 we shouldn't restrict the ddeb packages content to files in
 /usr/lib/debug.

That's a separate issue that hasn't been raised so far.  I'm surprised
that you'd want to mix this in.  I thought the whole point of this
proposal was to handle detached binary symbols in a way that's predictable
from the name of the binary package so that they could be, for instance,
automatically installed based on user request.  I thought one of the key
points of the discussion so far was that they were *not* going to take
over all of the complex variety of stuff people put into -dbg packages.

If we're also adding Python debug extensions, are we adding separate
copies of binaries built with -O0 -g?  Whole libraries built with library
debugging support?  Binaries built with more verbose trace information?
That seems like huge scope creep to me.

 5) There may only be one ddeb per source package (if more where needed,
 we could consider it).

Why would we do this?  Surely it makes more sense to have a one-to-one
correlation between debug packages and binary packages that contain the
binaries for which we have detached debugging symbols?

-- 
Russ Allbery (r...@debian.org)   http://www.eyrie.org/~eagle/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Manoj Srivastava]

On Tue, Aug 11 2009, Russ Allbery wrote:

 Emilio Pozuelo Monfort poch...@gmail.com writes:
 Manoj Srivastava wrote:

 To recap:
  1) packages with detached debugging symbols should be named
 ${package name}-${debug suffix}. As a corollary, no ordinary
 packages names may end in  ${debug suffix}.

 They may be automatically created. They may also be manually created (if
 they are listed in debian/control, so for complex packages where they
 need some manual work, it can be done).

 Whether they're automatically or manually created is irrelevant for Debian
 Policy.  Policy describes what the output should be, not what tools one
 uses to get there.

 I think the relevant question for Policy is whether these packages will be
 listed in debian/control in the source package, in Binary in the *.dsc
 file, and in Binary/Files/Checksums-* in the *.changes file.  And I don't
 know the answer to those three questions from the discussion so far.

Here is my take on this:
 a) helper packages may be extended to created debug packages by
default, whether or not they're mentioned in control. This means
that any package rebuilt the next time will get debugging packages,
even if the maintainer takes no action. Policy should not prevent
this use case, so requiring that the control file mentions them
should not be done.
 b) Some upstream packages, even if helper packages are used, might not
be readily amenable to automated generation of debug packages, and
manual help might be required. In this category I would also like to
throw in packages that do not use helper packages; since themanual
crafting of debug symbol packages is a commonality. These packages
have the debug packages in debian/control, and htey are built
normally (either through custoim scripts, or helper packages). In
this case, the helper should not automatically generate debug symbol
packages; and thus give us a mechanism to over ride, on a package by
package basis, the creation of automated debug packages.

So I think at this point it is premature for policy to decide
 one way or the other about debug symbol packages being mentioned in the
 control file (and dsc and changes).

I am also of the belief that perhaps the dsc and the changes
 file should treat them like normal .debs; and the differentiation occur
 at the archive level, when archive scripts try to determine where these
 packages go.

Another reason is that we should not be accepting any packages,
 even debug packages, in the archive unless we have a check sum match in
 a cryptographically signed file anyway.

manoj
-- 
Ten years of experience should add up to more than one year's experience
multiplied by ten.
Manoj Srivastava sriva...@debian.org http://www.debian.org/~srivasta/  
1024D/BF24424C print 4966 F272 D093 B493 410B  924B 21BA DABB BF24 424C


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Gunnar Wolf]

Russ Allbery dijo [Sat, Aug 08, 2009 at 05:51:33PM -0700]:
  You can build a .ddeb manually, yes. However for some cases
  (e.g. packages using debhelper and building ELF binaries) a .ddeb will
  be automatically created (if none is created manually) and detached
  debugging symbols will be put there. I'll try to automatize other
  languages too, so that having full archive coverage is as simpler as
  possible.
 
 Could you explain a bit more about what merits you see in creating
 something that we call a different type of package rather than just
 listing debug packages in debian/control and building them as we do now
 and handling section debug specially in the archive software?  Is it just
 the avoiding of the need to add a bunch of debian/control entries?

I would add:

• .ddebs could be autobuilt — I am not familiar with the procedure,
  but I suppose a debian/control field would indicate whether this
  package allows being built as a .ddeb (as there would be no way
  i.e. to build a Perl module as a ddeb)

• Less namespace explosion. We would get rid of all the -debug
  packages. 

-- 
Gunnar Wolf • gw...@gwolf.org • (+52-55)5623-0154 / 1451-2244


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Gunnar Wolf]

Steve Langasek dijo [Sun, Aug 09, 2009 at 05:15:39PM -0700]:
  If we are going to enshrine ddebs into policy, we might as well
   teach dpkg about ddebs.
 
 I don't have a strong opinion on whether ddebs should be documented in
 policy, but I certainly don't agree with requiring dpkg to understand them
 as a prerequisite for implementing a general purpose, public archive for
 auto-stripped debugging symbols packages.  There really is no reason for
 dpkg to treat these packages specially - a simple namespace convention
 imposed by Policy (i.e., reserving package names ending in -ddeb for use
 by this archive, which is what has been proposed) is sufficient, and
 requires no changes to dpkg, which is as it should be.
 
 I think the .ddeb extension is a red herring.  There ought not be anything
 new to teach dpkg, here; the only thing of relevance is that there not be
 namespace clashes between the ddebs and the debs in the main archive, and
 the filename is not relevant to that at all.

I understand your concern about this extension, but I do see it as a
merit. Of course, our tools must be aware of it.  And apt should know
-before updating or whatnot- that a package was installed from a ddeb,
if they are to share the base name. But I feel ddebs will allow
debugging packages creation and installation to take place in a much
more transparent, automatic fashion. I think this will be in the
interest of both users and developers.

-- 
Gunnar Wolf • gw...@gwolf.org • (+52-55)5623-0154 / 1451-2244


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Jonathan Yu]

On Tue, Aug 11, 2009 at 2:45 PM, Gunnar Wolfgw...@gwolf.org wrote:
 Russ Allbery dijo [Sat, Aug 08, 2009 at 05:51:33PM -0700]:
  You can build a .ddeb manually, yes. However for some cases
  (e.g. packages using debhelper and building ELF binaries) a .ddeb will
  be automatically created (if none is created manually) and detached
  debugging symbols will be put there. I'll try to automatize other
  languages too, so that having full archive coverage is as simpler as
  possible.
Automation is definitely the recipe for success when it comes to open source.

 Could you explain a bit more about what merits you see in creating
 something that we call a different type of package rather than just
 listing debug packages in debian/control and building them as we do now
 and handling section debug specially in the archive software?  Is it just
 the avoiding of the need to add a bunch of debian/control entries?

 I would add:

 • .ddebs could be autobuilt — I am not familiar with the procedure,
  but I suppose a debian/control field would indicate whether this
  package allows being built as a .ddeb (as there would be no way
  i.e. to build a Perl module as a ddeb)
My question then is, would it be possible to get debugging symbols for
the C/XS stuff we compile? Especially for figuring out segfaults that
would be tremendously useful, even in the context of Perl modules.

 • Less namespace explosion. We would get rid of all the -debug
  packages.
I understand what you mean, but I hope you don't intend get rid of
*all* those packages, because not all of them are what you expect them
to be. perl-debug for example is just Perl compiled with debugging
symbols enabled (you run it via debugperl rather than perl). Steve
Langasek mentioned this in a previous mail.

 --
 Gunnar Wolf • gw...@gwolf.org • (+52-55)5623-0154 / 1451-2244


 --
 To UNSUBSCRIBE, email to debian-policy-requ...@lists.debian.org
 with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org




--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Steve Langasek]

On Mon, Aug 10, 2009 at 06:06:37PM -0500, Manoj Srivastava wrote:
  So, please keep heckling from the peanut gallery to a minimum,
   please, and assume that policy editors have a modicum of sense when
   dealing with their role duties.

  If you were showing a modicum of sense, there would be no need to assume.

  For example, not referring to a fellow member of the Technical Committee,
  the constitutional authority on Debian technical policy, as the peanut
  gallery.

 The word you are looking for is not sense. It is respect.

 You expect respect based on the position you hold -- not the
  arguments you made. I reject the notion that I should make exeptions
  based on the office you hold (in other words, it is OK to call non-dd's
  on the list members of the peanut gallery, but not the
  oh-so-respectable ctte members such).

I don't think it's appropriate for you to refer to *any* participant on
debian-policy as the peanut gallery.  I additionally think that referring
to a member of the TC that way is stupid.

My original post to this thread was a technical argument in direct response
to a series of technical propositions that you advanced.  This subthread has
since degenerated into a pissing contest, apparently because I've dared
trespass the bounds of the mailing list that is your marked territory.

Thanks for that.

But if this is all the more respect you have for your fellow (TC
members|DDs|human beings), O Peerless and Saintly Policy Editor, then
perhaps the project should reconsider whether that's a position you should
hold.

 saying that policy team members should not do something that I had not
 actually advocated (I never said that dpkg implementation was a
 prerequisite to adding things to policy --- I just asked why that is
 not the reasonable thing to do first).

  If we are going to enshrine ddebs into policy, we might as well
  teach dpkg about ddebs.

  http://lists.debian.org/debian-policy/2009/08/msg00044.html

That was not a question.  But apparently, any statements you've made in
emails more than a day old are strawmen that you're not responsible for.

  Seems like if policy carves out a namespace for debug packages,
   it would serve for both automatically generated and hand crafted debug
   packages; and it is trivial for the automatic generation not to happen
   when there is an entry in debian/control for a debug package already,
   as long as there is a naming convention for debug packages.

  That's fair, but it doesn't guard against package name collisions with
  packages built from a *different* source package; so if manually-built
  packages are allowed to use the same namespace, there ought to be a policy
  in place that prevents them from being provided in a way that confuses the
  automated build process.
 
 Umm, huh? If the name space carved out is foo-debug, or even
  foo-dbg, the only collision I see is a *different* package has a native name
  of foo-dbg and thus collides with the debugging symbols from foo.

 If such packages existed, then not only would they create
  trouble with the current slew of debug packages, they can always be
  resolved by our normal disambiguation rules.

The normal disambiguation rules involve telling someone to stop building a
conflicting package.  If one of the parties is an automated build, that
doesn't work so well.  Either the namespace should be clearly reserved, or
the automatic build hooks are going to have to maintain a blacklist of
packages not to act on.

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Gunnar Wolf]

Manoj Srivastava dijo [Tue, Aug 11, 2009 at 10:12:00AM -0500]:
 On Tue, Aug 11 2009, Roger Leigh wrote:
 
  On Tue, Aug 11, 2009 at 01:40:20PM +, Sune Vuorela wrote:
  On 2009-08-11, Manoj Srivastava sriva...@debian.org wrote:
   So, we would still need to create   /usr/lib/debug/
. /full/path/to/lib_or_binary/ in either case, and instead of the
  
  no. it would be /usr/lib/debug/.build-id/NN/NN.debug
   ^
  Why the need for a hidden directory in a public location?
  What's wrong with /usr/lib/debug/build-id?
 
  I don't think the unnecessary obfuscation is warranted.
 
 Because that is where gdb looks for them.

Umh, and for human joy, adding a symlink at
/usr/lib/debug/lib_or_binary_with_slashes_substituted_by_underscores
to it would be most welcome!

-- 
Gunnar Wolf • gw...@gwolf.org • (+52-55)5623-0154 / 1451-2244


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Steve Langasek]

On Tue, Aug 11, 2009 at 01:50:21PM -0500, Gunnar Wolf wrote:

  I don't have a strong opinion on whether ddebs should be documented in
  policy, but I certainly don't agree with requiring dpkg to understand them
  as a prerequisite for implementing a general purpose, public archive for
  auto-stripped debugging symbols packages.  There really is no reason for
  dpkg to treat these packages specially - a simple namespace convention
  imposed by Policy (i.e., reserving package names ending in -ddeb for use
  by this archive, which is what has been proposed) is sufficient, and
  requires no changes to dpkg, which is as it should be.

  I think the .ddeb extension is a red herring.  There ought not be anything
  new to teach dpkg, here; the only thing of relevance is that there not be
  namespace clashes between the ddebs and the debs in the main archive, and
  the filename is not relevant to that at all.

 I understand your concern about this extension, but I do see it as a
 merit. Of course, our tools must be aware of it.

Except no one has *any intention* of making our tools be aware of this
extension.  This is a different file name convention, with no other impact.

 And apt should know -before updating or whatnot- that a package was
 installed from a ddeb, if they are to share the base name.

It was not proposed to have the packages share the base name, and doing so
implies a much more onerous implementation in the package manager than we
would otherwise need.

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Emilio Pozuelo Monfort]

Manoj Srivastava wrote:
 On Tue, Aug 11 2009, Russ Allbery wrote:
 
 Emilio Pozuelo Monfort poch...@gmail.com writes:
 Manoj Srivastava wrote:
 To recap:
  1) packages with detached debugging symbols should be named
 ${package name}-${debug suffix}. As a corollary, no ordinary
 packages names may end in  ${debug suffix}.
 They may be automatically created. They may also be manually created (if
 they are listed in debian/control, so for complex packages where they
 need some manual work, it can be done).
 Whether they're automatically or manually created is irrelevant for Debian
 Policy.  Policy describes what the output should be, not what tools one
 uses to get there.

 I think the relevant question for Policy is whether these packages will be
 listed in debian/control in the source package, in Binary in the *.dsc
 file, and in Binary/Files/Checksums-* in the *.changes file.  And I don't
 know the answer to those three questions from the discussion so far.
 
 Here is my take on this:
  a) helper packages may be extended to created debug packages by
 default, whether or not they're mentioned in control. This means
 that any package rebuilt the next time will get debugging packages,
 even if the maintainer takes no action. Policy should not prevent
 this use case, so requiring that the control file mentions them
 should not be done.
  b) Some upstream packages, even if helper packages are used, might not
 be readily amenable to automated generation of debug packages, and
 manual help might be required. In this category I would also like to
 throw in packages that do not use helper packages; since themanual
 crafting of debug symbol packages is a commonality. These packages
 have the debug packages in debian/control, and htey are built
 normally (either through custoim scripts, or helper packages). In
 this case, the helper should not automatically generate debug symbol
 packages; and thus give us a mechanism to over ride, on a package by
 package basis, the creation of automated debug packages.

ACK

 
 So I think at this point it is premature for policy to decide
  one way or the other about debug symbol packages being mentioned in the
  control file (and dsc and changes).

They should be in the changes file so they are uploaded together with it to the
archive (and so dak can process them). Not saying that should be mentioned in
policy though :)

Having them in the Binary section in the .dsc and Binary and Description in the
.changes files would mean modifying dpkg-buildpackage/dpkg-genchanges for ddebs
not listed in debian/control. However listing them in Files and Checksum-* in
the .changes requires no changes if you add the files to debian/files and use
dpkg-genchanges.

 Another reason is that we should not be accepting any packages,
  even debug packages, in the archive unless we have a check sum match in
  a cryptographically signed file anyway.

Agreed. As per my previous paragraph, having them in Files and Checksum-* would
solve this.

Regards,
Emilio



signature.asc
Description: OpenPGP digital signature

Re: Automatic Debug Packages

[Russ Allbery]

Emilio Pozuelo Monfort poch...@gmail.com writes:
 Manoj Srivastava wrote:

 So I think at this point it is premature for policy to decide
  one way or the other about debug symbol packages being mentioned in
  the control file (and dsc and changes).

 They should be in the changes file so they are uploaded together with it
 to the archive (and so dak can process them). Not saying that should be
 mentioned in policy though :)

Well, there's little point in having a standards document if we don't use
it to describe what we're doing, not that we always manage to update it
sufficiently quickly.

 Having them in the Binary section in the .dsc and Binary and Description
 in the .changes files would mean modifying
 dpkg-buildpackage/dpkg-genchanges for ddebs not listed in
 debian/control. However listing them in Files and Checksum-* in the
 .changes requires no changes if you add the files to debian/files and
 use dpkg-genchanges.

It sounds like listing them only in *.changes but not in *.dsc or
debian/control may be the easiest approach.

-- 
Russ Allbery (r...@debian.org)   http://www.eyrie.org/~eagle/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Manoj Srivastava]

On Tue, Aug 11 2009, Steve Langasek wrote:


 But if this is all the more respect you have for your fellow (TC
 members|DDs|human beings), O Peerless and Saintly Policy Editor, then
 perhaps the project should reconsider whether that's a position you should
 hold.

The -vote list is - that-away.


 saying that policy team members should not do something that I had not
 actually advocated (I never said that dpkg implementation was a
 prerequisite to adding things to policy --- I just asked why that is
 not the reasonable thing to do first).

   If we are going to enshrine ddebs into policy, we might as well
   teach dpkg about ddebs.

   http://lists.debian.org/debian-policy/2009/08/msg00044.html

 That was not a question.  But apparently, any statements you've made in
 emails more than a day old are strawmen that you're not responsible for.

It was a suggestion. It was not a dictum do this or the rule
 shall not enter policy. The correct response is to point out why there
 is no need to teach dpkg anything, or why doing so is suboptimal (the
 former applies here).


  Seems like if policy carves out a namespace for debug packages,
   it would serve for both automatically generated and hand crafted debug
   packages; and it is trivial for the automatic generation not to happen
   when there is an entry in debian/control for a debug package already,
   as long as there is a naming convention for debug packages.

  That's fair, but it doesn't guard against package name collisions with
  packages built from a *different* source package; so if manually-built
  packages are allowed to use the same namespace, there ought to be a policy
  in place that prevents them from being provided in a way that confuses the
  automated build process.
 
 Umm, huh? If the name space carved out is foo-debug, or even
  foo-dbg, the only collision I see is a *different* package has a native name
  of foo-dbg and thus collides with the debugging symbols from foo.

 If such packages existed, then not only would they create
  trouble with the current slew of debug packages, they can always be
  resolved by our normal disambiguation rules.

 The normal disambiguation rules involve telling someone to stop
 building a conflicting package.  If one of the parties is an automated
 build, that doesn't work so well.  Either the namespace should be
 clearly reserved, or the automatic build hooks are going to have to
 maintain a blacklist of packages not to act on.

The namespace for packages containing debug symbols out to be
 clearly resolved, and separated from the other non-debug-symbol
 packages, irrespective of the tools used to generate the debug symbol
 package.

The auto-tools would still have to look into debian/control, but
 that is an implementation detail.

manoj
-- 
A man wrapped up in himself makes a very small package.
Manoj Srivastava sriva...@debian.org http://www.debian.org/~srivasta/  
1024D/BF24424C print 4966 F272 D093 B493 410B  924B 21BA DABB BF24 424C


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Emilio Pozuelo Monfort]

Russ Allbery wrote:
 Having them in the Binary section in the .dsc and Binary and Description
 in the .changes files would mean modifying
 dpkg-buildpackage/dpkg-genchanges for ddebs not listed in
 debian/control. However listing them in Files and Checksum-* in the
 .changes requires no changes if you add the files to debian/files and
 use dpkg-genchanges.
 
 It sounds like listing them only in *.changes but not in *.dsc or
 debian/control may be the easiest approach.

Indeed, for the automatic-not-listed-in-debian-control ones. The others would be
listed everywhere, but that is okay.

Cheers,
Emilio



signature.asc
Description: OpenPGP digital signature

Re: Automatic Debug Packages

[Russ Allbery]

Emilio Pozuelo Monfort poch...@gmail.com writes:
 Russ Allbery wrote:

 It sounds like listing them only in *.changes but not in *.dsc or
 debian/control may be the easiest approach.

 Indeed, for the automatic-not-listed-in-debian-control ones. The others
 would be listed everywhere, but that is okay.

Yes, agreed.

Okay, to summarize what I think we've generally agreed on:

* Packages containing separate debugging symbols will have a dedicated
  package namespace, but that namespace will not be *-debug or *-dbg.
  We'll instead create a new one for this purpose.

* These packages are normal Debian packages with normal package metadata,
  but will generally have a symlink in /usr/share/doc/package pointing
  to the package for which they provide debugging information.

* They will go into a separate debug archive area, so the Section of
  such packages will be debug/foo where foo is the section of the
  corresponding binary package for which they provide debugging data.

* Those packages must be listed in *.changes like any other packages that
  are part of an upload.  They may or may not be present in debian/control
  and in *.dsc depending on the mechanism the package maintainer uses to
  generate them.

* The detached debugging symbols may use either the id or the debuglink
  mechanisms -- see Manoj's message for a summary.

Open questions:

* Can we limit this package namespace to *only* detached debugging
  symbols, not all the other sorts of debugging packages that people
  create with special compiler options or optional code features?

* What about contrib and non-free packages?  Do they just lose here?

* Can we require a one-to-one correspondance between binary package names
  and debug package names that provide symbols for that binary package?  I
  think we should; I think it would make the system more straightforward.

At some point, someone should probably open a Policy bug to track this
discussion and the resolution.

-- 
Russ Allbery (r...@debian.org)   http://www.eyrie.org/~eagle/


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org

Re: Automatic Debug Packages

[Emilio Pozuelo Monfort]

Russ Allbery wrote:
 Emilio Pozuelo Monfort poch...@gmail.com writes:
 Russ Allbery wrote:
 
 It sounds like listing them only in *.changes but not in *.dsc or
 debian/control may be the easiest approach.
 Indeed, for the automatic-not-listed-in-debian-control ones. The others
 would be listed everywhere, but that is okay.
 
 Yes, agreed.
 
 Okay, to summarize what I think we've generally agreed on:
 
 * Packages containing separate debugging symbols will have a dedicated
   package namespace, but that namespace will not be *-debug or *-dbg.
   We'll instead create a new one for this purpose.
 
 * These packages are normal Debian packages with normal package metadata,
   but will generally have a symlink in /usr/share/doc/package pointing
   to the package for which they provide debugging information.

We haven't agreed on whether there should be one ddeb per source or per binary
package, so I would leave this still opened.

 * They will go into a separate debug archive area, so the Section of
   such packages will be debug/foo where foo is the section of the
   corresponding binary package for which they provide debugging data.

Not sure about the archive area bit. What I talked with the ftpmasters was that
they would be in a totally different archive, so instead of
ftp.debian.org/debian unstable main, you would have something like
ftp.debian.org/debian-debug unstable main. I don't think that's an archive
area in Policy terms.

 * Those packages must be listed in *.changes like any other packages that
   are part of an upload.  They may or may not be present in debian/control
   and in *.dsc depending on the mechanism the package maintainer uses to
   generate them.

I guess that doesn't imply they need to be listed in Binary and Description, but
that Files and Checksum-* are enough? If so, perfect.

 * The detached debugging symbols may use either the id or the debuglink
   mechanisms -- see Manoj's message for a summary.
 
 Open questions:
 
 * Can we limit this package namespace to *only* detached debugging
   symbols, not all the other sorts of debugging packages that people
   create with special compiler options or optional code features?

Why should we limit it? There currently are about 85 python -dbg packages. A lot
more could be added. Why limit .ddebs to ELF binaries? Same for mono, I've added
a (trivial) patch to dh_clistrip to support ddebs. We would gain support for
many packages with exactly zero changes (or just a change to remove the -dbg
where they exist). What are the benefits of such a limitation?

 * What about contrib and non-free packages?  Do they just lose here?

If it's legal to ship debugging symbols for them, I can't see why we couldn't
support them normally.

 * Can we require a one-to-one correspondance between binary package names
   and debug package names that provide symbols for that binary package?  I
   think we should; I think it would make the system more straightforward.

I guess the Packages file could grow a Has-DDeb: yes line (or the Sources file
if we go for one ddeb per source package). Or we could have a different file for
this similar to the Maintainers one, I guess, since the trend seems to be to
split those files nowadays.

 At some point, someone should probably open a Policy bug to track this
 discussion and the resolution.

I would be happy to do that. I guess it can wait a bit until we have a bit more
consensus, or should I do it now and update it with whatever conclusions we 
reach?

Best,
Emilio



signature.asc
Description: OpenPGP digital signature