Re: sentry evaluation

[Harald Sitter]

The problem you are describing is because of the evaluative nature of
the current setup, where I was asked to force the entire system to
have limited exposure. In full rollout every report that the user
manually writes in drkonqi will also end up on sentry. Every
additional crash also, so long as the user enabled auto-submission
(that's the current plan anyway). i.e. Sentry would be the superset of
all crashes. We can look into tighter linking between bugzilla and
sentry but, again, that hasn't happened because of the evaluative
nature of things.

HS

On Thu, Jun 1, 2023 at 10:44 PM Nate Graham  wrote:
>
> To be honest, I haven't found Sentry to be that useful in its current
> implementation. The primary issue is that it represents a second source
> of truth for where crash reports live. As a result, developers who
> already struggle to notice Bugzilla-based crash reports have to look in
> a second place, further diving their scarce attention. I haven't seen
> evidence of people regularly interacting with it or looking at its
> crashes beyond the excitement of the initial rollout, so now it's
> largely just a new graveyard of issues being ignored due to insufficient
> developer time.
>
> I think Sentry could make sense to keep if it was implemented for us
> more as a kind of automatic symbolication service that can take a bad
> backtrace, add debug symbols, retrace it, and then pass *that* off to
> DrKonqi so that the resulting Bugzilla ticket is guaranteed to have a
> *good* backtrace in it. That's a real problem we currently have that
> could benefit from being solved in a targeted way.
>
> If we can't or don't want to do that, then Sentry might make sense if it
> were possible for us to bypass the "multiple sources of crash truth"
> issue by completely disabling Bugzilla for crash reports and migrating
> old Bugzilla crashes into sentry. But Then we'd run into the new issue
> that Sentry doesn't permit discussion with the person experiencing the
> crash to ask for more details if needed. This isn't always needed, but
> it sometimes is. Sentry also isn't integrated into our issue priority
> tracking system in Bugzilla, but that's a more minor issue.
>
> Nate
>
>
>
> On 6/1/23 04:35, Harald Sitter wrote:
> > Hey,
> >
> > We've had almost a year, albeit in a super limited capacity for git
> > builds, with sentry (https://errors-eval.kde.org) and we should
> > probably render a final verdict on the evaluation.
> >
> > Did you like it?
> > What could be improved?
> > Should we move ahead with a more permanent setup?
> >
> > The overall game plan would be to have drkonqi ask the user to opt
> > into automatic crash submission when they open drkonqi so we get close
> > to all crashes (the ones caught by kcrash) automatically filed into
> > sentry from then on out. To increase exposure of this feature I'd also
> > like to glue it into the feedback KCM but I'm not yet sure if it
> > should be a separate setting or linked to the regular feedback
> > categories, the former sounds more accessible. The current bugzilla
> > based workflow would still be available for when the user actually
> > wants to write a description.
> >
> > (previous discussion: https://markmail.org/thread/6y5paczdposz3aoj)
> >
> > HS

Re: sentry evaluation

[Harald Sitter]

On Mon, Jun 12, 2023 at 11:39 PM David Edmundson
 wrote:
> I'm not sure the teams and groups really work out right now. Do we
> need to request access to groups and have that approved?

That is how it used to be, yes. I've just sprinkled some code at the
instance that should allow us to forgo the approval system though,
only KDE developers can now log in (which is what the approval was
based on anyway).

The way this works in general is that "issues" get sent to a "project"
and that project can be part of multiple teams but to get access to
the data the developer has to opt into a team with access to a given
project first. Alas, I don't think we'll get around the opt-in.
Someone who's only interested in e.g. kwin would get annoyed by/ignore
the noise caused by e.g. ktuberling. That said, projects can be part
of multiple teams and we can have as many teams as we feel like.

> There was also a minor issue that a plasma bug might have a frameworks
> or Qt fix, but I couldn't reliably say "fixed in next version", it's
> not a case which sentry had a good infra for. (bugzilla also isn't
> amazing at that)

Indeed. It's a complicated problem in particular because we are not
the binary distributor for regular linuxy builds. I don't really have
a good answer here unfortunately.

> >Should we move ahead with a more permanent setup?
>
> 100% has my vote. Can we do things on a per-project basis? We can
> surface things in the Plasma UI right now, and there's still a while
> before release to pivot if needed.

I suppose we could do things per-project. What advantage do you think
we gain from this? Everything routes through drkonqi anyway and at
that point it makes no difference if we are inspecting kate or
kinfocenter.

HS

Re: sentry evaluation

[David Edmundson]

>Did you like it?

When we had an initial influx of reports after you first blogged it
was amazing. We were getting reports in places I hadn't seen before on
bugzilla.
It did a really good job of implicitly showing which things were the
most relevant needing fixing, and what's just noise we can just
ignore, and whether a crash has just gone away on it's own.

Now it's dead, but that's a chicken and egg situation.

>What could be improved?

I'm not sure the teams and groups really work out right now. Do we
need to request access to groups and have that approved? I don't want
people to have an excuse to not fix crashes!
There was also a minor issue that a plasma bug might have a frameworks
or Qt fix, but I couldn't reliably say "fixed in next version", it's
not a case which sentry had a good infra for. (bugzilla also isn't
amazing at that)

>Should we move ahead with a more permanent setup?

100% has my vote. Can we do things on a per-project basis? We can
surface things in the Plasma UI right now, and there's still a while
before release to pivot if needed.

David

Re: sentry evaluation

[Harald Sitter]

On Sun, Jun 11, 2023 at 7:13 PM Albert Vaca Cintora
 wrote:
>
> I didn't know we had Sentry! How can we get crash reports from KDE
> Connect through it?

Because this isn't widely rolled out yet, due to evaluation period,
there's only limited data for kdeconnect. Here's one:
https://errors-eval.kde.org/organizations/kde/issues/119/

(currently kdeconnect goes into the fallthrough project, which is
because of insufficient configuration on my part)

HS

Re: sentry evaluation

[Albert Vaca Cintora]

I didn't know we had Sentry! How can we get crash reports from KDE
Connect through it? That would be super useful. I've used Sentry in
the past for other projects and I see a big value in it.

Actually, in the case of the KDE Connect Android app, we already get
crash reports aggregated by popularity by default via the Play Store.
Same with the Windows version published in the Windows Store. It's
very useful to fix the most common crashes, and it would be great to
have something like that for Linux as well.

About the alternative of using bugzilla for crashes: We can't know how
common a crash is with the crashes that seldomly get uploaded to
bugzilla. So +1 for keeping Sentry.

Re: sentry evaluation

[Harald Sitter]

On Mon, Jun 5, 2023 at 12:20 AM Albert Astals Cid  wrote:
>
> El dijous, 1 de juny de 2023, a les 12:35:41 (CEST), Harald Sitter va
> escriure:
> > Hey,
> >
> > We've had almost a year, albeit in a super limited capacity for git
> > builds, with sentry (https://errors-eval.kde.org) and we should
> > probably render a final verdict on the evaluation.
> >
> > Did you like it?
>
> Did I miss the email announcing this?

There may have been no email. I've announced it at akademy and blogged about it
https://apachelog.wordpress.com/2022/10/13/kde-crash-tracking-system-%f0%9f%92%a3/

> By clicking on https://errors-eval.kde.org i can't really seem to see anything
> to evaluate either, all it deos is it asks me to join a team.

Yep. To get access to projects you need to be member of a team with
access to that project. E.g. if you join the #plasma team you get
access to the plasma projects. #community has access to everything.

HS

Re: sentry evaluation

[Albert Astals Cid]

El dijous, 1 de juny de 2023, a les 12:35:41 (CEST), Harald Sitter va 
escriure:
> Hey,
> 
> We've had almost a year, albeit in a super limited capacity for git
> builds, with sentry (https://errors-eval.kde.org) and we should
> probably render a final verdict on the evaluation.
> 
> Did you like it?

Did I miss the email announcing this?

I can only see a "Towards Excellent Defect Management" sent 2 years ago that 
describes a plan, but i can't see any kind of "You can join sentry to do this 
and that".

By clicking on https://errors-eval.kde.org i can't really seem to see anything 
to evaluate either, all it deos is it asks me to join a team.

Cheers,
  Albert

> What could be improved?
> Should we move ahead with a more permanent setup?
> 
> The overall game plan would be to have drkonqi ask the user to opt
> into automatic crash submission when they open drkonqi so we get close
> to all crashes (the ones caught by kcrash) automatically filed into
> sentry from then on out. To increase exposure of this feature I'd also
> like to glue it into the feedback KCM but I'm not yet sure if it
> should be a separate setting or linked to the regular feedback
> categories, the former sounds more accessible. The current bugzilla
> based workflow would still be available for when the user actually
> wants to write a description.
> 
> (previous discussion: https://markmail.org/thread/6y5paczdposz3aoj)
> 
> HS

Re: sentry evaluation

[Aleix Pol]

I think it has been a good tool. There's some problems that are simply
too hard to report using bugzilla and these can be analyzed
accordingly. In the end, they are all problems that exist and users
face. I understand it's a problem that we don't really get the context
in these but maybe it's just a price to pay.

I'm sure there's still room for improvement, but I'm sure there's
crashes we don't have now because our devs have seem there that they
appear and addressed them, even if from a theoretical perspective.

Aleix

On Thu, Jun 1, 2023 at 12:36 PM Harald Sitter  wrote:
>
> Hey,
>
> We've had almost a year, albeit in a super limited capacity for git
> builds, with sentry (https://errors-eval.kde.org) and we should
> probably render a final verdict on the evaluation.
>
> Did you like it?
> What could be improved?
> Should we move ahead with a more permanent setup?
>
> The overall game plan would be to have drkonqi ask the user to opt
> into automatic crash submission when they open drkonqi so we get close
> to all crashes (the ones caught by kcrash) automatically filed into
> sentry from then on out. To increase exposure of this feature I'd also
> like to glue it into the feedback KCM but I'm not yet sure if it
> should be a separate setting or linked to the regular feedback
> categories, the former sounds more accessible. The current bugzilla
> based workflow would still be available for when the user actually
> wants to write a description.
>
> (previous discussion: https://markmail.org/thread/6y5paczdposz3aoj)
>
> HS

Re: sentry evaluation

[Sharaf Zaman]

Hi!

Nate Graham  writes:

> To be honest, I haven’t found Sentry to be that useful in its current
> implementation. The primary issue is that it represents a second source of 
> truth
> for where crash reports live. As a result, developers who already struggle to
> notice Bugzilla-based crash reports have to look in a second place, further
> diving their scarce attention. I haven’t seen evidence of people regularly
> interacting with it or looking at its crashes beyond the excitement of the
> initial rollout, so now it’s largely just a new graveyard of issues being
> ignored due to insufficient developer time.
>

I think looking at Sentry as just another tool to get reports is a wrong way to
look at it (: It should be more like a tool to look at reports which *most*
people are facing, in this case crashes. Sentry isn’t currently implemented in
Krita, but on Android we have similar reporting, thanks to google play and it
has helped me a lot to detect crashes in our beta (most people never report
them!). Similarly it has helped me ignore the obscure error that 0.001% of the
user space gets on their out of ordinary device (and you have no idea a crash is
obscure from bugzilla). In that sense sentry is something very similar and could
be a great metric to direct your attention at issues rather than a distraction 
:)

> I think Sentry could make sense to keep if it was implemented for us more as a
> kind of automatic symbolication service that can take a bad backtrace, add 
> debug
> symbols, retrace it, and then pass *that* off to DrKonqi so that the resulting
> Bugzilla ticket is guaranteed to have a *good* backtrace in it. That’s a real
> problem we currently have that could benefit from being solved in a targeted
> way.

I would absolutely love if sentry could also symbolicate from binary factory
artifacts, but I know it can be difficult!

>
> If we can’t or don’t want to do that, then Sentry might make sense if it were
> possible for us to bypass the “multiple sources of crash truth” issue by
> completely disabling Bugzilla for crash reports and migrating old Bugzilla
> crashes into sentry. But Then we’d run into the new issue that Sentry doesn’t
> permit discussion with the person experiencing the crash to ask for more 
> details
> if needed. This isn’t always needed, but it sometimes is. Sentry also isn’t
> integrated into our issue priority tracking system in Bugzilla, but that’s a
> more minor issue.
>
> Nate
>
>
>
> On 6/1/23 04:35, Harald Sitter wrote:
>> Hey,
>> We’ve had almost a year, albeit in a super limited capacity for git
>> builds, with sentry () and we should
>> probably render a final verdict on the evaluation.
>> Did you like it?
>> What could be improved?
>> Should we move ahead with a more permanent setup?
>> The overall game plan would be to have drkonqi ask the user to opt
>> into automatic crash submission when they open drkonqi so we get close
>> to all crashes (the ones caught by kcrash) automatically filed into
>> sentry from then on out. To increase exposure of this feature I’d also
>> like to glue it into the feedback KCM but I’m not yet sure if it
>> should be a separate setting or linked to the regular feedback
>> categories, the former sounds more accessible. The current bugzilla
>> based workflow would still be available for when the user actually
>> wants to write a description.
>> (previous discussion: )
>> HS

Re: sentry evaluation

[Nate Graham]

To be honest, I haven't found Sentry to be that useful in its current 
implementation. The primary issue is that it represents a second source 
of truth for where crash reports live. As a result, developers who 
already struggle to notice Bugzilla-based crash reports have to look in 
a second place, further diving their scarce attention. I haven't seen 
evidence of people regularly interacting with it or looking at its 
crashes beyond the excitement of the initial rollout, so now it's 
largely just a new graveyard of issues being ignored due to insufficient 
developer time.


I think Sentry could make sense to keep if it was implemented for us 
more as a kind of automatic symbolication service that can take a bad 
backtrace, add debug symbols, retrace it, and then pass *that* off to 
DrKonqi so that the resulting Bugzilla ticket is guaranteed to have a 
*good* backtrace in it. That's a real problem we currently have that 
could benefit from being solved in a targeted way.


If we can't or don't want to do that, then Sentry might make sense if it 
were possible for us to bypass the "multiple sources of crash truth" 
issue by completely disabling Bugzilla for crash reports and migrating 
old Bugzilla crashes into sentry. But Then we'd run into the new issue 
that Sentry doesn't permit discussion with the person experiencing the 
crash to ask for more details if needed. This isn't always needed, but 
it sometimes is. Sentry also isn't integrated into our issue priority 
tracking system in Bugzilla, but that's a more minor issue.


Nate



On 6/1/23 04:35, Harald Sitter wrote:

Hey,

We've had almost a year, albeit in a super limited capacity for git
builds, with sentry (https://errors-eval.kde.org) and we should
probably render a final verdict on the evaluation.

Did you like it?
What could be improved?
Should we move ahead with a more permanent setup?

The overall game plan would be to have drkonqi ask the user to opt
into automatic crash submission when they open drkonqi so we get close
to all crashes (the ones caught by kcrash) automatically filed into
sentry from then on out. To increase exposure of this feature I'd also
like to glue it into the feedback KCM but I'm not yet sure if it
should be a separate setting or linked to the regular feedback
categories, the former sounds more accessible. The current bugzilla
based workflow would still be available for when the user actually
wants to write a description.

(previous discussion: https://markmail.org/thread/6y5paczdposz3aoj)

HS