Hi,
So the changes issue I was referring to doesn't seem to be fixed (I
checked the latest Buildbot code) and looked back over my changes. The
problem that I identified is here
https://github.com/buildbot/buildbot/blob/9f1cac1d3bb61baa0b6c836cc18812a64cfa9c2b/master/buildbot/data/resultspec.py#L320
because is some cases there would be an unmatched_filter or
unmatched_order. *So if this is indeed the case, you should see the
warning in the buildbot log file:*
"Warning: limited data api query is not backed by db because of
following
filters..." as defined here
https://github.com/buildbot/buildbot/blob/9f1cac1d3bb61baa0b6c836cc18812a64cfa9c2b/master/buildbot/data/resultspec.py#L322
There are two reasons why this happens. One, an incomplete definition of
the fieldMapping for the data/changes.py and this would be the fix:
+++ b/master/buildbot/data/changes.py
@@ -42,6 +42,19 @@ class FixerMixin:
return change
fieldMapping = {
'changeid': 'changes.id',
+ 'author': 'changes.author',
+ 'committer': 'changes.committer',
+ 'comments': 'changes.comments',
+ 'branch': 'changes.branch',
+ 'revision': 'changes.revision',
+ 'revlink': 'changes.revlink',
+ 'when_timestamp': 'changes.when_timestamp',
+ 'category': 'changes.category',
+ 'repository': 'changes.repository',
+ 'codebase': 'changes.codebase',
+ 'project': 'changes.project',
+ 'sourcestampid': 'changes.sourcestampid',
+ 'parent_changeids': 'changes.parent_changeids',
}
and two not having all the columns in the select statement in order to
be
able to check for matched/unmatched filtering or ordering as it is done
here
https://github.com/buildbot/buildbot/blob/9f1cac1d3bb61baa0b6c836cc18812a64cfa9c2b/master/buildbot/data/resultspec.py#L269.
My solution was to get all the fields from the changes table from the
database (don't know if this is the best approach) as you can see here:
+++ b/master/buildbot/db/changes.py
@@ -249,7 +249,7 @@ class
ChangesConnectorComponent(base.DBConnectorComponent):
def thd(conn):
# get the changeids from the 'changes' table
changes_tbl = self.db.model.changes
- q = sa.select([changes_tbl.c.changeid])
+ q = changes_tbl.select()
However, since only the changeid would be needed, don't sure if this
would
be the right approach. Also, I would suspect some missing tests because
I
would expect this to be a pretty common use case.
Regarding the profiler, I also had some problems running it for a longer
time, so if you find a fix, please share it with me.
Cheers,
Vlad
On Wed, Jan 13, 2021 at 1:33 PM Pierre Tardy <[email protected]> wrote:
I am not sure why this wouldn't work. I vaguely recall there was an
issue
there, but can't figure it out staring at the code.
As this is quite ancient, I am not sure the JS part will even build
anymore :-/
You can change the default values at that line in the python
https://github.com/tardyp/buildbot_profiler/blob/master/buildbot_profiler/api.py#L193
I think you will be able to force them by editing this file inside your
virtualenv..
Regards
Pierre
Le mer. 13 janv. 2021 à 11:46, Yngve N. Pettersen <[email protected]> a
écrit :
> I insist on the buildbot profiler. What I was saying before is that
you
> need to hit the record button before the problem appears, and put a
large
> enough record time to be sure to catch a spike.
> Then, you will be able to zoom to the cpu spike and catch the issue
> precisely.
>
> If the spike is in the order of minutes like you said, you can
configure
> it
> like this and get enough samples to get enough evidence to where the
code
> is actually spending time:
>
> ProfilerService(frequency=500, gatherperiod=60 * 60, mode='virtual',
> basepath=None, wantBuilds=100
I tried configuring this with the settings dropdown in the WebGUI
plugin,
but AFAICT it is not working, it only gathers info for 30 seconds.
I guess I must be holding it incorrectly.
> This will record for one hour, and mitigate the memory used if you
worry
> about it.
--
Sincerely,
Yngve N. Pettersen
Vivaldi Technologies AS
--
Vlad
