Re: Fwd: [Bug 2060275] [NEW] pmproxy crash at startup in libpcp_web.so.1
Hi Martin,
On Tue, Apr 9, 2024 at 6:09 PM Martin Pitt wrote:
>
> Nathan Scott [2024-04-09 17:30 +1000]:
> > > It's not really unknown, it's "just" a file conflict:
> >
> > Yeah - the unknown bit for me is "why tho" - I cannot see conflicting
> > files in those packages that would have any debug symbols (there's
> > some common directories... but no binaries shared AFAICS).
> >
> > > | dpkg: error processing archive
> > > build/deb/pcp-pmda-infiniband-dbgsym_6.2.1-0.20240409.f312285_amd64.deb
> > > (--install):
> > > | trying to overwrite
> > > '/usr/lib/debug/.build-id/57/02df011cfaf166b948e1fefde236eaf3a6ee65.debug',
> > > which is also in package pcp-dbgsym 6.2.1-0.20240409.f312285
> > > |
> > > | dpkg: error processing archive
> > > build/deb/pcp-testsuite-dbgsym_6.2.1-0.20240409.f312285_amd64.deb
> > > (--install):
> > > | trying to overwrite
> > > '/usr/lib/debug/.build-id/17/6edc7e590f766a2ea87b5decaeb994d7c48d24.debug',
> > > which is also in package pcp-dbgsym 6.2.1-0.20240409.f312285
> > >
> > > I.e. these are shipped in two different packages.
> >
> > "these"?
>
> These two files, i.e.
> /usr/lib/debug/.build-id/57/02df011cfaf166b948e1fefde236eaf3a6ee65.debug
> exists
> both in pcp-pmda-infiniband-dbgsym and pcp-dbgsym. Presumably they shouldn't
> be
> in the latter.
Yep, understood - but again, I'm not understanding why. AFAICS, there
are no files with the same names or contents between those packages.
> > OK ... so that's pointing towards v3 archives a little bit, good.
> >
> > > > The limited stack we have suggests we're in pmproxy log discovery
> > > > code, in an inotify/libuv event, which does have v3-specific code.
> > > >
> > > > For those who can reproduce this, it'd be worth experimenting and
> > > > setting the following field back to 2 ... (requires pmlogger restart).
> > > >
> > > > $ grep PCP_ARCHIVE_VERSION /etc/pcp.conf
> > > > PCP_ARCHIVE_VERSION=3
>
> I created https://github.com/cockpit-project/cockpit/pull/20275 with an x120
> test amplification, and intererestinly there the overwhelming majority of test
> runs actually crashes there. So with that I have a fairly high confidence in
> the significance of test results when trying a change.
>
> I tested with
>
> sed -i 's/PCP_ARCHIVE_VERSION=3/PCP_ARCHIVE_VERSION=2/' /etc/pcp.conf
>
> This runs on image preparation, i.e. clean /var/log and no daemons running.
> The
> VM is freshly booted for each test, so no running pmlogger. There is no
> observed change, it still crashes the same way and with the same frequency
> ("almost every time").
OK, so it's not related to the recent v3 archive changes then.
I have a fairly recent Debian VM locally - I tried reproducing the problem
there but had no luck, it always starts fine and runs fine. Tried running it
under valgrind too just in case, but again nothing.
We also don't see this issue on Fedora, CentOS or RHEL and SuSE are
also not reporting this. Since (I think) all Debian versions are fine (can we
confirm?) it might be a Ubuntu-specific issue. Are there patches or some
global compiler option(s) unique to Ubuntu versions where this is failing?
Otherwise, I'm fresh out of ideas.
I can't really justify time chasing this any further - I think we'll need some
assistance from Ubuntu folk to help us gain additional insights.
cheers.
--
Nathan
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2060275
Title:
pmproxy crash at startup in libpcp_web.so.1
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/pcp/+bug/2060275/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: Fwd: [Bug 2060275] [NEW] pmproxy crash at startup in libpcp_web.so.1
Nathan Scott [2024-04-09 17:30 +1000]:
> > It's not really unknown, it's "just" a file conflict:
>
> Yeah - the unknown bit for me is "why tho" - I cannot see conflicting
> files in those packages that would have any debug symbols (there's
> some common directories... but no binaries shared AFAICS).
>
> > | dpkg: error processing archive
> > build/deb/pcp-pmda-infiniband-dbgsym_6.2.1-0.20240409.f312285_amd64.deb
> > (--install):
> > | trying to overwrite
> > '/usr/lib/debug/.build-id/57/02df011cfaf166b948e1fefde236eaf3a6ee65.debug',
> > which is also in package pcp-dbgsym 6.2.1-0.20240409.f312285
> > |
> > | dpkg: error processing archive
> > build/deb/pcp-testsuite-dbgsym_6.2.1-0.20240409.f312285_amd64.deb
> > (--install):
> > | trying to overwrite
> > '/usr/lib/debug/.build-id/17/6edc7e590f766a2ea87b5decaeb994d7c48d24.debug',
> > which is also in package pcp-dbgsym 6.2.1-0.20240409.f312285
> >
> > I.e. these are shipped in two different packages.
>
> "these"?
These two files, i.e.
/usr/lib/debug/.build-id/57/02df011cfaf166b948e1fefde236eaf3a6ee65.debug exists
both in pcp-pmda-infiniband-dbgsym and pcp-dbgsym. Presumably they shouldn't be
in the latter.
(I'm out of this for many years, so I'm afraid I don't know what a good
solution is, i.e. how much control you have over dbgsym generation).
> OK ... so that's pointing towards v3 archives a little bit, good.
>
> > > The limited stack we have suggests we're in pmproxy log discovery
> > > code, in an inotify/libuv event, which does have v3-specific code.
> > >
> > > For those who can reproduce this, it'd be worth experimenting and
> > > setting the following field back to 2 ... (requires pmlogger restart).
> > >
> > > $ grep PCP_ARCHIVE_VERSION /etc/pcp.conf
> > > PCP_ARCHIVE_VERSION=3
I created https://github.com/cockpit-project/cockpit/pull/20275 with an x120
test amplification, and intererestinly there the overwhelming majority of test
runs actually crashes there. So with that I have a fairly high confidence in
the significance of test results when trying a change.
I tested with
sed -i 's/PCP_ARCHIVE_VERSION=3/PCP_ARCHIVE_VERSION=2/' /etc/pcp.conf
This runs on image preparation, i.e. clean /var/log and no daemons running. The
VM is freshly booted for each test, so no running pmlogger. There is no
observed change, it still crashes the same way and with the same frequency
("almost every time").
Note that I can easily pull in a PPA or even a binary with curl for
testing.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2060275
Title:
pmproxy crash at startup in libpcp_web.so.1
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/pcp/+bug/2060275/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: Fwd: [Bug 2060275] [NEW] pmproxy crash at startup in libpcp_web.so.1
Hello Nathan, Nathan Scott [2024-04-09 16:19 +1000]: > Is any of this getting through... ? Just checked the Ubuntu tracker > URL, and looks like every response Ken or I sent has been dropped on > the ground. Right, I didn't get any response either (not a surprise, as it's *first* Launchpad receiving replies, and then it sends out notifications). I did do bug replies via email in the past, but these days it may get caught in some spam prevention measures? Probably better to just post them on the web ui? I CC my reply to the LP bug if you don't mind -- first of all to test this, and also to keep a more permanent record of our discussion. > Long and short of it is, we've not been able to reproduce and debian > dbgsym on sub-packages is still broken for unknown reasons... > https://github.com/performancecopilot/pcp/pull/1948 It's not really unknown, it's "just" a file conflict: | dpkg: error processing archive build/deb/pcp-pmda-infiniband-dbgsym_6.2.1-0.20240409.f312285_amd64.deb (--install): | trying to overwrite '/usr/lib/debug/.build-id/57/02df011cfaf166b948e1fefde236eaf3a6ee65.debug', which is also in package pcp-dbgsym 6.2.1-0.20240409.f312285 | | dpkg: error processing archive build/deb/pcp-testsuite-dbgsym_6.2.1-0.20240409.f312285_amd64.deb (--install): | trying to overwrite '/usr/lib/debug/.build-id/17/6edc7e590f766a2ea87b5decaeb994d7c48d24.debug', which is also in package pcp-dbgsym 6.2.1-0.20240409.f312285 I.e. these are shipped in two different packages. [1] https://github.com/performancecopilot/pcp/actions/runs/8610492722/job/23596103839?pr=1948#step:9:149 > This is not a known bug - do you know if this is specific to pcp-6.2.0 > (latest PCP) or are earlier versions affected? One change that may > be related here is we enabled v3 PCP archives by default in 6.2.0. We see this only in noble (i.e. upcoming 24.04). I.e. 6.0.5 in 23.10 was still ok, and 6.2.0 occasionally crashes. > The limited stack we have suggests we're in pmproxy log discovery > code, in an inotify/libuv event, which does have v3-specific code. > > For those who can reproduce this, it'd be worth experimenting and > setting the following field back to 2 ... (requires pmlogger restart). > > $ grep PCP_ARCHIVE_VERSION /etc/pcp.conf > PCP_ARCHIVE_VERSION=3 > > If that clears the issue, it'll help us triangulate on a possible cause. OK -- I'll do some experimentation and report back here. Thanks! Martin -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2060275 Title: pmproxy crash at startup in libpcp_web.so.1 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/pcp/+bug/2060275/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
