Re: [Patchwork-maintainers] patchwork.ozlabs.org downtime for maintenance - 15/16 August

[Jeremy Kerr]

Hi Ilya,

> Jeremy, could you, please, try to unban the robot?

Sure thing, done!

Let me know how it goes on your side, I'll monitor things here.

> Note:  Just to clarify, all the UAs should have '(pw-ci)' prefix
> now with additional identifications for the project/actions they
> are related to.

OK, good to know - as long as there's a common identifier present.

Cheers,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [Patchwork-maintainers] patchwork.ozlabs.org downtime for maintenance - 15/16 August

[Jeremy Kerr]

Hi Ilya,

> Ugh.  Yeah, the checks/ requests are definitely something we can
> improve. Aaron is working on removing vast majority of this type of
> requests as we speak.  Hopefully, that will be done soon.

OK, sounds good!

> Do you think it'll be fine to unban the robot once it doesn't run that
> many requests on the checks/ API in particular?  (I expect the number
> of requests to be less than a 100-ish per day after the fix.)

Yes, definitely. I'm okay with unbanning it sooner, if it's doing useful
stuff, and we can contain the load somewhat, and/or we need to test
against production data.

(more that I wasn't sure if it was a legitimate use of the API
earlier, hence reducing the load via the nft rule)

> That might be useful, thanks!

OK, I will send that separately.

Cheers,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [Patchwork-maintainers] patchwork.ozlabs.org downtime for maintenance - 15/16 August

[Jeremy Kerr]

Hi Ilya,

> We have a robot running in the RH network that pushes checks and
> updates statuses for openvswitch and ovn projects, but it shouldn't
> really make any "global patches view" types of requests.  All the
> patches it looks at supposed to be only from these two projects.  And
> it should not make more than one concurrent request.

OK, that looks like it then; this isn't one of the spiders crawling
through *all* patches (and does seem to be contained to OVS & OVN) but
it's certainly a major contributor to load.

It seems to be re-requesting the same view hundreds of times. From one
day's worth of log, the top 10 URLs from that IP:
   
   399 /api/patches/1887072/checks/
   285 /api/patches/1888116/checks/
   285 /api/patches/1888115/checks/
   285 /api/patches/1888111/checks/
   228 /api/patches/1888114/checks/
   228 /api/patches/1888112/checks/
   228 /api/patches/1887464/checks/
   228 /api/patches/1887463/checks/
   228 /api/patches/1884952/checks/
   228 /api/patches/1884950/checks/

- totalling 43,786 requests for that day.

> Could you provide some examples of requests that are heavy (maybe
> off-list), so we can take a look?

I can send you a log over a day if that's helpful.

Cheers,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [Patchwork-maintainers] patchwork.ozlabs.org downtime for maintenance - 15/16 August

[Jeremy Kerr]

Hi all,
> I'll try and get to that over the weekend.

Looks like the heaviest database load is due to API request to the
global patches view, which is a bit of an odd use-case; that all
appears to be mostly spider traffic.

Konstantin: I'm not sure your new index would help in that case, we're
not looking up delegates for those views.

Looking through the access logs, there seem to be three clients that
are causing around 40-50% of patchwork load:

 - one IP from an "Alibaba Cloud HK" AS, various UAs
 - one IP from a Red Hat AS, curl/7.61.1 UA
 - the Bytedance "Bytespider" UA

All three seem to be scraping the patchwork site.

I have blocked all three for now, but it would be worthwhile setting up
a more fair robots.txt and/or a reasonable ratelimit for the latter
case.

If anyone knows what might be up with that Red Hat crawler, please get
in touch with me.

I'll keep an eye on things here; there's still likely a bunch of
potential configuration optimisation we can do too. Let me know if your
observations change though.

Cheers,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [Patchwork-maintainers] patchwork.ozlabs.org downtime for maintenance - 15/16 August

[Jeremy Kerr]

Hi Rob,

(adding Ilya, who has done some very handy logging too)

> I have a job that runs every 20min checking PW. I added timestamps
> and
> here's all the 503 failures in the last 22 hours:
> 
> Wed Jan 17 10:12:29 AM CST 2024
> Wed Jan 17 12:16:38 PM CST 2024
> Wed Jan 17 01:37:01 PM CST 2024
> Wed Jan 17 03:00:23 PM CST 2024
> Wed Jan 17 03:43:04 PM CST 2024
> Wed Jan 17 11:52:18 PM CST 2024
> Thu Jan 18 12:32:38 AM CST 2024
> Thu Jan 18 02:33:54 AM CST 2024
> Thu Jan 18 03:34:50 AM CST 2024
> Thu Jan 18 06:20:40 AM CST 2024
> Thu Jan 18 06:40:57 AM CST 2024
> 
> Looks pretty much spread out except 4PM-11PM my time didn't have any
> failures.

Thanks for that - good to know!

Ilya: your timestamps - are they in UTC or local?

>  Must be the Europeans causing problems. ;)

I blame you all being north of the equator :D

Your data seems pretty consistent with what Ilya had found - spread out
through the day rather than just during backups.

It looks like the request load has increased enough to need tuning of
the application server's backlog queue. I'll do some adjustments to
that, but it does need to be a little careful in that we don't just
punt the problem somewhere else.

I'll try and get to that over the weekend.

Cheers,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [Patchwork-maintainers] patchwork.ozlabs.org downtime for maintenance - 15/16 August

[Jeremy Kerr]

Hi Rob,

> When was recent? The 503s have been happening for probably a month or
> 2 now, but seem to be increasing in frequency to the point of PW
> being unusable now.

That was early Jan, so you've been seeing those since before the
upgrade.

I do see quite a bit of spider traffic now though, so might have some
options to limit load there too.

> > So far it does seem time of day related, so possibly conflicting
> > with load from overnight backups.
> 
> When and how long do those run? I'm pretty much working your
> overnight.

I'd have to defer to Stephen on that, but I had been assuming 4am /
17:00 UTC.

Cheers,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [Patchwork-maintainers] patchwork.ozlabs.org downtime for maintenance - 15/16 August

[Jeremy Kerr]

Hi Rob & Konstantin,

> 
> > I'm still seeing this slowness and now to add to that intermittent
> > 503 errors.

OK, I've heard the same yesterday from the ovn folks. Sounds like this
is continuing after a recent hardware upgrade too.

So far it does seem time of day related, so possibly conflicting with
load from overnight backups.

> In case it helps, and just for the record, the following index really
> helped to improve some of the patch list views when delegates were
> used:
> 
>     ALTER TABLE patchwork_patch ADD INDEX
> patchwork_patch_delegate_id_state_id_archived (delegate_id, state_id,
> archived);

Neat, good to know! I'll check out the query log and see if we have
overlap on the 'expensive' queries issued, and add the index if it
looks like that would help.

Cheers,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [Patchwork-maintainers] patchwork.ozlabs.org downtime for maintenance - 15/16 August

[Jeremy Kerr]

Hi Rob,

> > I'll do a bit of experimentation here to see what's up, will keep
> > you
> > posted.
> 
> Actually, it is worse than just slow. New patches aren't showing up
> since before the move.

Looks like I missed a dependency for the custom filter for the DT feed;
apologies for that. I've updated the filter, and incoming patches
should be parsed from here on.

I've also had a look at the speed issues, which seem particularly bad
for the DT list. We've bumped up the system memory, and tweaked the
indexes a little; things look a little faster now, but keep me posted
on performance there too.

Cheers,


Jeremy

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [Patchwork-maintainers] patchwork.ozlabs.org downtime for maintenance - 15/16 August

[Jeremy Kerr]

Hi Rob,


> Since the move, the web interface seems slow for some operations.
> Loading the list view takes ~30s. Loading a patch seems fine, and I'm
> not seeing any slowness with the CLI.

OK, thanks for letting me know. It looks like this is reliably
reproducible, but only affects some of the patch lists; DT is
particularly slow, but linuxppc seems fine.


I'll do a bit of experimentation here to see what's up, will keep you
posted.

Cheers,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

patchwork.ozlabs.org downtime for maintenance - 15/16 August

[Jeremy Kerr]

Hi patchworkers,

Stephen and I will be moving the patchwork.ozlabs.org instance to a new
server this week. For this, we'll need about two hours of downtime,
starting at:

 Australia East: midday Tuesday 16 Aug
 Australia West: 10am Tuesday 16 Aug
 UTC: 2am Tuesday 16 Aug
 US West: 7pm Monday 15 Aug
 US East: 10pm Monday 15 Aug

Once back up, there should be no change to the software, just where
it's hosted.

Let me know if you have any queries.

Cheers,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Downloading archives no longer possible

[Jeremy Kerr]

Hi Stephen (F),

> Our documentation [1] suggests people initialize their development
> environments by download the archives of the Patchwork mailing lists
> from [2] and load them with the 'parsearchive' command. However, it's
> been pointed out to me that at request to [1] now return only a handful
> of recent messages. Is this change intentional?

No, no intentional change there; that's just something exported by
mailman, I'm not sure how it's managed.

sfr: has anything changed on the mailman side there?

Cheers,


Jeremy

> 
> Stephen
> 
> [1] 
> https://patchwork.readthedocs.io/en/latest/development/installation/#import-mailing-list-archives
> [2] https://lists.ozlabs.org/private/patchwork.mbox/patchwork.mbox
> 
> 

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

[PATCH 4/5] parser: don't trigger database IntegrityErrors on duplicate comments

[Jeremy Kerr]

As we've done for the Patch model, this change prevents database errors
from duplicate Comments.

Signed-off-by: Jeremy Kerr 
---
 patchwork/parser.py|  6 +++---
 patchwork/tests/test_parser.py | 12 
 2 files changed, 15 insertions(+), 3 deletions(-)

diff --git a/patchwork/parser.py b/patchwork/parser.py
index e03634a..406c916 100644
--- a/patchwork/parser.py
+++ b/patchwork/parser.py
@@ -1247,7 +1247,9 @@ def parse_mail(mail, list_id=None):
 
 author = get_or_create_author(mail, project)
 
-try:
+with transaction.atomic():
+if Comment.objects.filter(submission=submission, msgid=msgid):
+raise DuplicateMailError(msgid=msgid)
 comment = Comment.objects.create(
 submission=submission,
 msgid=msgid,
@@ -1255,8 +1257,6 @@ def parse_mail(mail, list_id=None):
 headers=headers,
 submitter=author,
 content=message)
-except IntegrityError:
-raise DuplicateMailError(msgid=msgid)
 
 logger.debug('Comment saved')
 
diff --git a/patchwork/tests/test_parser.py b/patchwork/tests/test_parser.py
index c7c918a..d1a9a21 100644
--- a/patchwork/tests/test_parser.py
+++ b/patchwork/tests/test_parser.py
@@ -1122,3 +1122,15 @@ class DuplicateMailTest(TestCase):
 self._test_duplicate_mail(m)
 
 self.assertEqual(Patch.objects.count(), 1)
+
+def test_duplicate_comment(self):
+diff = read_patch('0001-add-line.patch')
+m1 = create_email(diff, listid=self.listid, msgid='1...@example.com')
+_parse_mail(m1)
+
+m2 = create_email('test', listid=self.listid, msgid='2...@example.com',
+  in_reply_to='1...@example.com')
+self._test_duplicate_mail(m2)
+
+self.assertEqual(Patch.objects.count(), 1)
+self.assertEqual(Comment.objects.count(), 1)
-- 
2.17.1

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

[PATCH 1/5] tests: Add duplicate mail test

[Jeremy Kerr]

Test that we get the correct DuplicateMailError from parsing the same
mail twice.

Signed-off-by: Jeremy Kerr 
---
 patchwork/tests/test_parser.py | 27 +++
 1 file changed, 27 insertions(+)

diff --git a/patchwork/tests/test_parser.py b/patchwork/tests/test_parser.py
index f5631be..3a41000 100644
--- a/patchwork/tests/test_parser.py
+++ b/patchwork/tests/test_parser.py
@@ -14,6 +14,7 @@ import unittest
 
 from django.test import TestCase
 from django.test import TransactionTestCase
+from django.db.transaction import atomic
 
 from patchwork.models import Comment
 from patchwork.models import Patch
@@ -31,6 +32,7 @@ from patchwork.parser import parse_series_marker
 from patchwork.parser import parse_version
 from patchwork.parser import split_prefixes
 from patchwork.parser import subject_check
+from patchwork.parser import DuplicateMailError
 from patchwork.tests import TEST_MAIL_DIR
 from patchwork.tests import TEST_FUZZ_DIR
 from patchwork.tests.utils import create_project
@@ -1080,3 +1082,28 @@ class WeirdMailTest(TransactionTestCase):
 
 def test_x_face(self):
 self._test_patch('x-face.mbox')
+
+
+class DuplicateMailTest(TestCase):
+def setUp(self):
+self.listid = 'patchwork.ozlabs.org'
+create_project(listid=self.listid)
+create_state()
+
+def _test_duplicate_mail(self, mail):
+_parse_mail(mail)
+with self.assertRaises(DuplicateMailError):
+# If we see any database errors from the duplicate insert
+# (typically an IntegrityError), the insert will abort the current
+# transaction. This atomic() ensures that we can recover, and
+# perform subsequent queries.
+with atomic():
+_parse_mail(mail)
+
+def test_duplicate_patch(self):
+diff = read_patch('0001-add-line.patch')
+m = create_email(diff, listid=self.listid, msgid='1...@example.com')
+
+self._test_duplicate_mail(m)
+
+self.assertEqual(Patch.objects.count(), 1)
-- 
2.17.1

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

[PATCH 2/5] tests: ensure we don't see database errors during duplicate insert

[Jeremy Kerr]

Currently, the parser causes IntegrityErrors while inserting duplicate
patches; these tend to pollute database logs.

This change adds a check, which currently fails, to ensure we do not
cause errors during a duplicate patch parse.

Signed-off-by: Jeremy Kerr 
---
 patchwork/tests/test_parser.py | 27 +--
 1 file changed, 21 insertions(+), 6 deletions(-)

diff --git a/patchwork/tests/test_parser.py b/patchwork/tests/test_parser.py
index 3a41000..c7c918a 100644
--- a/patchwork/tests/test_parser.py
+++ b/patchwork/tests/test_parser.py
@@ -15,6 +15,7 @@ import unittest
 from django.test import TestCase
 from django.test import TransactionTestCase
 from django.db.transaction import atomic
+from django.db import connection
 
 from patchwork.models import Comment
 from patchwork.models import Patch
@@ -1091,14 +1092,28 @@ class DuplicateMailTest(TestCase):
 create_state()
 
 def _test_duplicate_mail(self, mail):
+errors = []
+
+def log_query_errors(execute, sql, params, many, context):
+try:
+result = execute(sql, params, many, context)
+except Exception as e:
+errors.append(e)
+raise
+return result
+
 _parse_mail(mail)
+
 with self.assertRaises(DuplicateMailError):
-# If we see any database errors from the duplicate insert
-# (typically an IntegrityError), the insert will abort the current
-# transaction. This atomic() ensures that we can recover, and
-# perform subsequent queries.
-with atomic():
-_parse_mail(mail)
+with connection.execute_wrapper(log_query_errors):
+# If we see any database errors from the duplicate insert
+# (typically an IntegrityError), the insert will abort the
+# current transaction. This atomic() ensures that we can
+# recover, and perform subsequent queries.
+with atomic():
+_parse_mail(mail)
+
+self.assertEqual(errors, [])
 
 def test_duplicate_patch(self):
 diff = read_patch('0001-add-line.patch')
-- 
2.17.1

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

[PATCH 3/5] parser: prevent IntegrityErrors

[Jeremy Kerr]

Currently, the parser relies on causing (and catching) IntegrityErrors
on patch insert to catch duplicate (msgid,project) mails.

This change performs an atomic select -> insert instead.

Signed-off-by: Jeremy Kerr 
---
 patchwork/parser.py | 7 ---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/patchwork/parser.py b/patchwork/parser.py
index a09fd75..e03634a 100644
--- a/patchwork/parser.py
+++ b/patchwork/parser.py
@@ -1062,7 +1062,10 @@ def parse_mail(mail, list_id=None):
 filenames = find_filenames(diff)
 delegate = find_delegate_by_filename(project, filenames)
 
-try:
+with transaction.atomic():
+if Patch.objects.filter(project=project, msgid=msgid):
+raise DuplicateMailError(msgid=msgid)
+
 patch = Patch.objects.create(
 msgid=msgid,
 project=project,
@@ -1077,8 +1080,6 @@ def parse_mail(mail, list_id=None):
 delegate=delegate,
 state=find_state(mail))
 logger.debug('Patch saved')
-except IntegrityError:
-raise DuplicateMailError(msgid=msgid)
 
 for attempt in range(1, 11):  # arbitrary retry count
 try:
-- 
2.17.1

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

[PATCH 0/5] Prevent database error logs on duplicate mail

[Jeremy Kerr]

On patchwork.ozlabs.org, we see a lot of noise in the postgres logs, due
to INSERTs with duplicate values for (project,msgid) keys. Patchwork's
strategy for catching the resulting IntegrityError means that each of
these is logged.

Instead, this change moves to an atomic select -> insert for the Patch,
Comment and Coverletter parsing.

Jeremy Kerr (5):
  tests: Add duplicate mail test
  tests: ensure we don't see database errors during duplicate insert
  parser: prevent IntegrityErrors
  parser: don't trigger database IntegrityErrors on duplicate comments
  parser: don't trigger database IntegrityErrors on duplicate
coverletters

 patchwork/parser.py| 20 ++-
 patchwork/tests/test_parser.py | 64 ++
 2 files changed, 75 insertions(+), 9 deletions(-)

-- 
2.17.1

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

[PATCH 5/5] parser: don't trigger database IntegrityErrors on duplicate coverletters

[Jeremy Kerr]

As we've done for the Patch and Comment models, this change prevents
database errors from duplicate CoverLetters.

Signed-off-by: Jeremy Kerr 
---
 patchwork/parser.py|  7 ---
 patchwork/tests/test_parser.py | 10 ++
 2 files changed, 14 insertions(+), 3 deletions(-)

diff --git a/patchwork/parser.py b/patchwork/parser.py
index 406c916..b9eb88c 100644
--- a/patchwork/parser.py
+++ b/patchwork/parser.py
@@ -1220,7 +1220,10 @@ def parse_mail(mail, list_id=None):
 SeriesReference.objects.create(
 msgid=msgid, project=project, series=series)
 
-try:
+with transaction.atomic():
+if CoverLetter.objects.filter(project=project, msgid=msgid):
+raise DuplicateMailError(msgid=msgid)
+
 cover_letter = CoverLetter.objects.create(
 msgid=msgid,
 project=project,
@@ -1229,8 +1232,6 @@ def parse_mail(mail, list_id=None):
 headers=headers,
 submitter=author,
 content=message)
-except IntegrityError:
-raise DuplicateMailError(msgid=msgid)
 
 logger.debug('Cover letter saved')
 
diff --git a/patchwork/tests/test_parser.py b/patchwork/tests/test_parser.py
index d1a9a21..cdf299c 100644
--- a/patchwork/tests/test_parser.py
+++ b/patchwork/tests/test_parser.py
@@ -21,6 +21,7 @@ from patchwork.models import Comment
 from patchwork.models import Patch
 from patchwork.models import Person
 from patchwork.models import State
+from patchwork.models import CoverLetter
 from patchwork.parser import clean_subject
 from patchwork.parser import get_or_create_author
 from patchwork.parser import find_patch_content as find_content
@@ -1134,3 +1135,12 @@ class DuplicateMailTest(TestCase):
 
 self.assertEqual(Patch.objects.count(), 1)
 self.assertEqual(Comment.objects.count(), 1)
+
+def test_duplicate_coverletter(self):
+m = create_email('test', listid=self.listid, msgid='1...@example.com')
+del m['Subject']
+m['Subject'] = '[PATCH 0/1] test cover letter'
+
+self._test_duplicate_mail(m)
+
+self.assertEqual(CoverLetter.objects.count(), 1)
-- 
2.17.1

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH 1/2] api: do not fetch every patch in a patch detail view 404

[Jeremy Kerr]

Hi Daniel,

> mpe and jk and sfr found that the OzLabs server was melting due
> to some queries downloading every patch.
> 
> Turns out if you 404 the patch detail view in the API, d-r-f attempts
> to render a listbox with every single patch to fill in the 'related'
> field.

... and the query for that includes all patch content and headers, so
transferring gigabytes of data per access.

>  The bundle API also has a similar field.
> 
> Replace the multiple selection box with a text field. You can still
> (AIUI) populate the relevant patch IDs manually.

Much better, the patchwork server is no longer on fire!

Thanks for that.

Tested-by: Jeremy Kerr 
Server-no-longer-on-fire-by: Jeremy Kerr 

Cheers,


Jeremy

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH 2/2] api: allow filtering patches and covers by msgid

[Jeremy Kerr]

Hi Daniel,

> In the process of fixing the previous bug, I realised that:
> 
>  a) /api/patches/msgid is a perfectly reasonable thing to attempt
>  b) We have no way of finding a patch by message id in the API
> 
> We can't actualy make /api/patches/msgid work because it may not
> be unique, but we can add a filter.

LGTM: 
https://patchwork.ozlabs.org/api/patches/?msgid=20200414062102.6798-3-...@axtens.net

Tested-by: Jeremy Kerr 

Cheers,


Jeremy

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

[PATCH] lib/sql: Update grant script for recent schema changes

[Jeremy Kerr]

This change fixes a few omissions in the grant scripts:

- patchrelation is missing from both mysql and postgres scripts; it's
  only needed for web user access.

- event is missing from the web grants on postgres, and the mail grants
  on mysql.

Tested on postgres only.

Fixes: 27c2acf56c ("models, templates: Add patch relations")
Fixes: 34e3c9c493 ("sql: Update 'grant-all.mysql' script with missing tables")
Fixes: 234bc7c316 ("lib/sql: fix permissions for v2.0.0 on postgres")
Signed-off-by: Jeremy Kerr 
---
 lib/sql/grant-all.mysql.sql| 2 ++
 lib/sql/grant-all.postgres.sql | 4 
 2 files changed, 6 insertions(+)

diff --git a/lib/sql/grant-all.mysql.sql b/lib/sql/grant-all.mysql.sql
index 0277077..100cd38 100644
--- a/lib/sql/grant-all.mysql.sql
+++ b/lib/sql/grant-all.mysql.sql
@@ -23,6 +23,7 @@ GRANT SELECT, UPDATE, INSERT, DELETE ON patchwork_emailoptout 
TO 'www-data'@loca
 GRANT SELECT, UPDATE, INSERT, DELETE ON patchwork_event TO 
'www-data'@localhost;
 GRANT SELECT, UPDATE, INSERT, DELETE ON patchwork_patch TO 
'www-data'@localhost;
 GRANT SELECT, UPDATE, INSERT, DELETE ON patchwork_patchchangenotification TO 
'www-data'@localhost;
+GRANT SELECT, UPDATE, INSERT, DELETE ON patchwork_patchrelation TO 
'www-data'@localhost;
 GRANT SELECT, UPDATE, INSERT, DELETE ON patchwork_patchtag TO 
'www-data'@localhost;
 GRANT SELECT, UPDATE, INSERT, DELETE ON patchwork_person TO 
'www-data'@localhost;
 GRANT SELECT, UPDATE, INSERT, DELETE ON patchwork_project TO 
'www-data'@localhost;
@@ -38,6 +39,7 @@ GRANT SELECT, UPDATE, INSERT, DELETE ON 
patchwork_userprofile_maintainer_project
 -- cover letters) and series
 GRANT INSERT, SELECT ON patchwork_comment TO 'nobody'@localhost;
 GRANT INSERT, SELECT ON patchwork_coverletter TO 'nobody'@localhost;
+GRANT INSERT, SELECT ON patchwork_event TO 'nobody'@localhost;
 GRANT INSERT, SELECT ON patchwork_patch TO 'nobody'@localhost;
 GRANT INSERT, SELECT ON patchwork_person TO 'nobody'@localhost;
 GRANT INSERT, SELECT ON patchwork_series TO 'nobody'@localhost;
diff --git a/lib/sql/grant-all.postgres.sql b/lib/sql/grant-all.postgres.sql
index 10ec8d2..56a2486 100644
--- a/lib/sql/grant-all.postgres.sql
+++ b/lib/sql/grant-all.postgres.sql
@@ -21,8 +21,10 @@ GRANT SELECT, UPDATE, INSERT, DELETE ON
patchwork_delegationrule,
patchwork_emailconfirmation,
patchwork_emailoptout,
+   patchwork_event,
patchwork_patch,
patchwork_patchchangenotification,
+   patchwork_patchrelation,
patchwork_patchtag,
patchwork_person,
patchwork_project,
@@ -50,7 +52,9 @@ GRANT SELECT, UPDATE ON
patchwork_comment_id_seq,
patchwork_delegationrule_id_seq,
patchwork_emailconfirmation_id_seq,
+   patchwork_event_id_seq,
patchwork_patch_id_seq,
+   patchwork_patchrelation_id_seq,
patchwork_patchtag_id_seq,
patchwork_person_id_seq,
patchwork_project_id_seq,
-- 
2.17.1

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: patchwork.ozlabs.org offline for upgrade - Mon 13 April

[Jeremy Kerr]

Hi all,

> I see that there are a couple of large migrations in the 2.2.0 release,
> but hopefully they won't take too long. I'll send an update when we're
> done.

OK, we're back up and running on v2.2.0. Let me know if you see
anything odd post-upgrade.

A couple of notes:

 - by far the longest migration task was executing these queries:


https://github.com/getpatchwork/patchwork/blob/v2.2.0/patchwork/migrations/0032_migrate_data_from_series_patch_to_patch.py#L18

   looks like that would be a good contender for an UPDATE .. FROM, 
   rather than the ~2 million subqueries

 - the database disk usage has increased by about 14%; I assume this is
   the new indexes.

That said, all went pretty smoothly, given the python & django upgrades
required too. Thanks patchworkers, and thanks to sfr for the help!

Cheers,


Jeremy


___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

patchwork.ozlabs.org offline for upgrade - Mon 13 April

[Jeremy Kerr]

Hi all,

Given a lot of folks are taking a bit of time off work over Easter, I
will be upgrading the patchwork instance on ozlabs.org in a couple of
minutes, which requires a bit of downtime in order to upgrade the
database and run migrations.

I see that there are a couple of large migrations in the 2.2.0 release,
but hopefully they won't take too long. I'll send an update when we're
done.

Cheers,


Jeremy

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Deduplication of patchwork mail content?

[Jeremy Kerr]

Hi Daniel,

> While I'm at it, it occurred to me that for both the ozlabs and
> kernel.org instances, there are a lot of mails that are sent across
> multiple projects. ATM the entire contents of the mail - content,
> headers, diff, what have you, will be stored in full for each project.

The headers will be different, as they've gone through different lists.
This may not be too relevant to the actual purpose of patchwork though.

The comments (apart from the first) may diverge, depending on whether
responders keep both lists on CC.

The diffs will be the same, so we could deduplicate those, if it's worth
your trouble:

   patchwork=# select sum(dup_size) from (select octet_length(diff) *
   (n-1) as dup_size, a.msgid, n from (select msgid, count(msgid) as n,
   min(id) as id from patchwork_submission group by msgid having
   count(msgid) > 1) as a inner join patchwork_patch on
   patchwork_patch.submission_ptr_id = a.id) as b;
   sum
   ---
221334709
   (1 row)

and:

   patchwork=# select sum(octet_length(diff)) from patchwork_patch;
   sum 
   
6261083055
   (1 row)


So 221MB out of 6.2GB is duplicate; around 3.5%.

Cheers,


Jeremy

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [U-Boot-Custodians] Patchwork bug?

[Jeremy Kerr]

Hi Simon,

> > Since the bug was in the parser, the patches received before that
> > was
> > applied would have been affected. Anything parsed from then onwards
> > should be fine.
> 
> OK that is good news, thanks. Has this version rolled out?

Yes, I'd updated the patchwork.ozlabs.org instance right after that fix.

Cheers,


Jeremy

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

[PATCH] notifications: fix notification expiry when no user is associated

[Jeremy Kerr]

It's possible that an EmailConfirmation object will have no associated
user (eg, for email opt-out, which does not require a user object). In
this case, we will see a NULL value for EmailConfirmation.user_id.

However, having a NULL value appear in a SQL 'IN' clause will match
every value. This means that once one of these null-user
EmailConfirmations is present, we will never expire any non-active user
accounts.

This change adds a filter for a valid user_id when we query for active
EmailConfirmation objects. This means we'll have a valid values set to
use in the pending_confs set.

Signed-off-by: Jeremy Kerr 
---
 patchwork/notifications.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/patchwork/notifications.py b/patchwork/notifications.py
index a5f6423..571cff7 100644
--- a/patchwork/notifications.py
+++ b/patchwork/notifications.py
@@ -109,7 +109,8 @@ def expire_notifications():
 EmailConfirmation.objects.filter(q).delete()
 
 # remove inactive users with no pending confirmation
-pending_confs = EmailConfirmation.objects.values('user')
+pending_confs = (EmailConfirmation.objects
+.filter(user__isnull=False).values('user'))
 users = User.objects.filter(is_active=False).exclude(id__in=pending_confs)
 
 # delete users
-- 
2.17.1

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Planned patchwork.ozlabs.org upgrades (and downtime) - 27 April

[Jeremy Kerr]

Hi all,


After that, we should be up and running at v2.1.1.


OK, all done! patchwork.ozlabs.org is now at 2.1.1.

The upgrade was fairly painless, aside from a few failing test cases. 
Those all look to be due to differences from the "standard" test 
environment, but I'll be taking a closer look over the week. The 
database migration was much quicker than expected too.


Let me know if anything looks amiss after the upgrade.

Cheers,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Planned patchwork.ozlabs.org upgrades (and downtime) - 27 April

[Jeremy Kerr]

Hello patchworkers,

A growing number of folks using patchwork.ozlabs.org have asked when 
we're planning to update the patchwork version there. So:


I'm planning an upgrade of the patchwork instance at 
patchwork.ozlabs.org on the 27th of April, around 10am UTC+8. I expect 
that updates to the database schema may take some time, so it's likely 
that patchwork will be unavailable for a few hours.


After that, we should be up and running at v2.1.1.

Let me know if you have any queries.

Cheers,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

patchwork.ozlabs.org database upgrade: 22/23 November

[Jeremy Kerr]

Hi all,

We have a couple of database upgrades pending for patchwork.ozlabs.org,
and would like to get these applied soon.

This will require a short period of unavailability, so we're planning to
use the quiet(er) period over US Thanksgiving to perform perform the
upgrade. This will be happening at 2am UTC on the 23rd of November,
which is:

 10am Friday   Australia West
  1pm Friday   Australia East
  6pm Thursday US West
  9pm Thursday US East

This is just a change to the database backend, so there should be no
changes to any of the patchwork interfaces.

Let me know if you have any queries.

Cheers,


Jeremy



___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: patchwork.ozlabs.org down, and e-mails not recorded

[Jeremy Kerr]

Hi all,

Good news: with the logging we had in place recently (and a fix to
correct error reporting), it looks like we've found the problem with
parsing emails from a series. This issue was recently addressed by
Daniel, so I've updated patchwork.ozlabs.org to run the current
stable/2.0 branch, which has fixes for these. Hopefully this resolves
the issue with dropped patches.

The update should also make the mail parser a little more robust against
transient failures too.

Let me know how things go over the next few days.

Thomas and Dave: please keep using the custom addresses you've
configured for the incoming mail feed; this will allow me to keep an eye
on things on my side too. Assuming everything goes fine, we can revert
to the usual address next week.

Cheers,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: patchwork.ozlabs.org down, and e-mails not recorded

[Jeremy Kerr]

Hi Daniel,

> We have quite a few of these sitting in our queue :-(

After a bit of time with some extra debug enabled for the buildroot mail
feed, I see this for the failed parsemail invocations where we've
returned a temp-fail:

  No handlers could be found for logger 
"patchwork.management.commands.parsemail"

If you recall, we had to alter the logging configuration for the ozlabs
instance:

  --- a/patchwork/settings/production.py
  +++ b/patchwork/settings/production.py
  @@ -70,7 +70,7 @@
   STATIC_URL = '/static/'
   
   
  -LOGGING = {
  +_LOGGING = {
   'version': 1,
   'disable_existing_loggers': False,
   'formatters': {

[this was your suggested fix an issue with parsemail failing with the
shipped LOGGING configuration, but I don't have the specifics of that..]

So, it appears that this workaround needs to be reworked?

Regards,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: patchwork.ozlabs.org down, and e-mails not recorded

[Jeremy Kerr]

Hi Andrew,


jk, any idea whether there's something particular about the ozlabs.org
instance that could be causing it to drop these patches?


We're running plain upstream with regards to the parsing code; so if it
parses we should be fine there.

Assume the parsing is OK, there are a couple of reasons for potential
drops:

 - the database server was down at the time of parse. This is quite
   infrequent, but can happen during database upgrades. The last one of
   those was at 2018-01-27 05:08:30 UTC, so doesn't look like the case
   here

 - after the update to 2.x, database load has significantly increased;
   there have been occasions where db connections are rejected due to
   this. The recent patchwork update should address some issues there,
   but I'm not sure whether we're back to optimal db behaviour now...

Cheers,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Patchwork API

[Jeremy Kerr]

Hi Stijn,

> I am trying to use git-pw, which uses the new API in patchwork 2.0, but
> as soon as add a project to the filter, the list of patches is empty.
> Any idea what might be wrong here?
> https://patchwork.ozlabs.org/api/patches/ vs
> https://patchwork.ozlabs.org/api/patches/?project=lede1==

I'm no expert on the API (I wasn't involved in writing it), but it looks
like you need to reference the project by id, not shortname:

  [jk@pudge ~]$ wget -qO - 
'https://patchwork.ozlabs.org/api/patches/?project=lede' | json_pp
  []
  [jk@pudge ~]$ wget -qO - 
'https://patchwork.ozlabs.org/api/patches/?project=54' | json_pp
  [
 {
"url" : "https://patchwork.ozlabs.org/api/patches/618204/;,
"name" : "generic: Colorize the command line prompt",
"hash" : "917cfbb306fe87f84ad1acccb0e9797d798d13e1",
"commit_ref" : null,
"archived" : false,
"state" : "changes-requested",
"checks" : "https://patchwork.ozlabs.org/api/patches/618204/checks/;,
"tags" : {},
"project" : {
   "web_url" : "http://lede-project.org/;,
   "scm_url" : "",
   "list_email" : "lede-...@lists.infradead.org",
   "id" : 54,
   "webscm_url" : "http://git.lede-project.org/;,
   "link_name" : "lede",
   "url" : "https://patchwork.ozlabs.org/api/projects/54/;,
   "name" : "LEDE development",
   "list_id" : "lede-dev.lists.infradead.org"
},
"date" : "2016-05-04T01:33:50",
"pull_url" : null,
"mbox" : "https://patchwork.ozlabs.org/patch/618204/mbox/;,
"id" : 618204,
"series" : [],
"delegate" : null,
"msgid" : "<1462325630-18332-1-git-send-email-ldpin...@gmail.com>",
"check" : "pending",
"submitter" : {
   "name" : "L. D. Pinney",
   "url" : "https://patchwork.ozlabs.org/api/people/65512/;,
   "id" : 65512,
   "email" : "ldpin...@gmail.com"
}
 },
 [...]

The /api/projects/ interface gives you the project IDs.

I've CCed the patchwork list in case I've completely missed something :)

Cheers,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH] REST: Filter projects by 'linkname', not 'name'

[Jeremy Kerr]

Hi Dnaiel,

> I was just reading though this before applying it when it occurred to
> me:
>>  except ValueError:
>> -filters = {'name__iexact': ' '.join(value.split('-'))}
>> +filters = {'linkname__iexact': ' '.join(value.split('-'))}
> 
> Surely a linkname shouldn't have spaces in it? It'd be of the form
> foo.bar.org, right?

patchwork=# select linkname from patchwork_project limit 10;
 linkname
---
 cbe-oss-dev
 linux-mtd
 linuxppc-embedded
 linux-ext4
 netdev
 sparclinux
 linux-ide
 qemu-devel
 ubuntu-kernel
 yaboot

So yes, we wouldn't expect spaces, and we tend to use dashes instead of
dots. This is essentially used as a path component in URLs.

Cheers,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH] views: Fix "Add to bundle" dropdown on patch list view

[Jeremy Kerr]

Hi Andrew,

> jk: if you could apply this to ozlabs.org would be much appreciated

Sure thing, done.

Cheers,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Patchwork v2.0.0 Available

[Jeremy Kerr]

Hi Andrew,

> Ticket on github, after mpe and paulus complained about it:
> https://github.com/getpatchwork/patchwork/issues/116
> 
> I'll try and take a look at it today - jk, if I get a fix to you do you
> mind applying it on ozlabs.org?

Can do!

Cheers,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

[PATCH 2/2] lib/sql: fix permissions for v2.0.0 on postgres

[Jeremy Kerr]

Some tables are no longer present, and others that are used by the web
interface and mail parser need access permissions added.

This change was required to get patchwork going on patchwork.ozlabs.org;
there may be other permissions required, that we haven't hit yet. So,
some review would be good here.

Also: it's unlikely that we need DELETE for the mail parser, but I'm not
confident enough to remove that at the moment.

Signed-off-by: Jeremy Kerr <j...@ozlabs.org>
---
 lib/sql/grant-all.postgres.sql | 17 -
 1 file changed, 8 insertions(+), 9 deletions(-)

diff --git a/lib/sql/grant-all.postgres.sql b/lib/sql/grant-all.postgres.sql
index 405ba44..c709866 100644
--- a/lib/sql/grant-all.postgres.sql
+++ b/lib/sql/grant-all.postgres.sql
@@ -12,6 +12,7 @@ GRANT SELECT, UPDATE, INSERT, DELETE ON
auth_group,
auth_user_user_permissions,
auth_permission,
+   authtoken_token,
patchwork_emailconfirmation,
patchwork_state,
patchwork_comment,
@@ -47,9 +48,7 @@ GRANT SELECT, UPDATE ON
patchwork_bundle_id_seq,
patchwork_bundlepatch_id_seq,
patchwork_comment_id_seq,
-   patchwork_submission_id_seq,
patchwork_patch_id_seq,
-   patchwork_coverletter_id_seq,
patchwork_series_id_seq,
patchwork_seriespatch_id_seq,
patchwork_seriesreference_id_seq,
@@ -69,16 +68,17 @@ TO "www-data";
 -- cover letters) and series
 GRANT INSERT, SELECT ON
patchwork_submission,
-   patchwork_patch,
patchwork_coverletter,
-   patchwork_series,
patchwork_seriespatch,
patchwork_seriesreference,
patchwork_comment,
-   patchwork_person
+   patchwork_event
 TO "nobody";
 GRANT INSERT, SELECT, UPDATE, DELETE ON
-   patchwork_patchtag
+   patchwork_patchtag,
+   patchwork_patch,
+   patchwork_series,
+   patchwork_person
 TO "nobody";
 GRANT SELECT ON
patchwork_project,
@@ -87,15 +87,14 @@ GRANT SELECT ON
patchwork_delegationrule
 TO "nobody";
 GRANT UPDATE, SELECT ON
-   patchwork_submission_id_seq,
patchwork_patch_id_seq,
-   patchwork_coverletter_id_seq,
patchwork_series_id_seq,
patchwork_seriespatch_id_seq,
patchwork_seriesreference_id_seq,
patchwork_person_id_seq,
patchwork_comment_id_seq,
-   patchwork_patchtag_id_seq
+   patchwork_patchtag_id_seq,
+   patchwork_event_id_seq
 TO "nobody";
 
 COMMIT;
-- 
2.7.4

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

[PATCH 1/2] tests: Run FuzzTest within a transaction

[Jeremy Kerr]

Currently, the FuzzTests fail for me with:

/backends/base/base.py", line 428, in validate_no_broken_transaction
"An error occurred in the current transaction. You can't "
TransactionManagementError: An error occurred in the current transaction. You 
can't execute queries until the end of the 'atomic' block.

- because the SQL inserts can fail, during an active transaction (the
first failure I see is attempting to insert \0 chars in
codec-null.mbox); this causes the setup for the next test case to fail.

Instead, run each test in its own transaction.

Signed-off-by: Jeremy Kerr <j...@ozlabs.org>
---
 patchwork/tests/test_parser.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/patchwork/tests/test_parser.py b/patchwork/tests/test_parser.py
index 583dfcc..9389f28 100644
--- a/patchwork/tests/test_parser.py
+++ b/patchwork/tests/test_parser.py
@@ -25,7 +25,7 @@ from email.utils import make_msgid
 import os
 import unittest
 
-from django.test import TestCase
+from django.test import TestCase, TransactionTestCase
 from django.utils import six
 
 from patchwork.models import Comment
@@ -865,7 +865,7 @@ class SubjectTest(TestCase):
 self.assertEqual(parse_version('Hello, world (V6)', []), 6)
 
 
-class FuzzTest(TestCase):
+class FuzzTest(TransactionTestCase):
 """Test fuzzed patches."""
 def setUp(self):
 create_project(listid='patchwork.ozlabs.org')
-- 
2.7.4

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Patchwork v2.0.0 Available

[Jeremy Kerr]

Hi David,


Please look into this soon, thank you.


OK, sorted. Permissions problem with the patchwork_event table. Let me
know if anything else is breaking.

Daniel: I also seem to be getting zero (email) notification of
server-side errors, I assume from the new LOGGING configuration in
base.py :(

Cheers,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Patchwork v2.0.0 Available

[Jeremy Kerr]

Hi Andrew,


I'm seeing a 500 immediately upon login at
https://patchwork.ozlabs.org/user/...


OK, sorted. Looks like the permissions scripts are quite out-of-date
with the changes to the db.

Cheers,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Patchwork v2.0.0 Available

[Jeremy Kerr]

Hi all,


Just an update: the database migration for v2.0.0 is taking a *long*
time, so we'll be out for more than that one hour. I'll get things
back up and running as soon as possible, but we're dependent on this
migration completing first.


OK, we're back up now. Please let me know if anything looks amiss.

Cheers,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Patchwork v2.0.0 Available

[Jeremy Kerr]

Hi all,


The total timeframe somewhat depends on how smoothly the upgrade goes,
but one hour would be a conservative estimate.


Just an update: the database migration for v2.0.0 is taking a *long*
time, so we'll be out for more than that one hour. I'll get things
back up and running as soon as possible, but we're dependent on this
migration completing first.

Cheers,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Patchwork v2.0.0 Available

[Jeremy Kerr]

Hi all,

> We're delighted to announce the release of Patchwork v2.0.0 ("Dazzle"):

Awesome!

I'm planning to update patchwork.ozlabs.org this weekend, at around
Sunday, 1pm (Australia/Perth timezone - UTC-8). Consequently, patchwork
will be unavailable during the upgrade.

The total timeframe somewhat depends on how smoothly the upgrade goes,
but one hour would be a conservative estimate.

Cheers,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Moving pwclient into its own project

[Jeremy Kerr]

Hi Stephen,

> I'd like to move pwclient into its own project/repo/possibly mailing
> list. There are a couple of reasons for doing this:

I have no specific objections to this (aside from splitting the mailing
list - I think it'd be a mistake to split the potential community
there), but:

> - I want to be able to package this up for PyPi to allow installation
>   with pip

Makes sense; I have very little experience with PyPi, but does that mean
we need separate repos?

> - I want to be able to release new pwclient features to everyone
>   without waiting for their patchwork admin to bump the Patchwork
>   version

I don't see why that needs an update? There should be separate channels
for each of these.

> - I want to be able to provide more comprehensive documentation for
>   pwclient, as it slowly morphs into a VCS-independent, general-purpose
>   Patchwork CLI tool

That sounds awesome, but doesn't mean we need a separate repo.

> - (stretch goal) I'd like to create deb/rpm packages for Fedora,
>   Ubuntu, etc.

Distros should have no issue with producing multiple (installable)
binary packages from one source package.

Overall, I'm not convinced that this is a good idea, but I'm certainly
not the deciding factor in this :)

Cheers,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: captcha type support for user registration?

[Jeremy Kerr]

Hi Andy,

>> My ITS recently noticed that some people are using the registration
>> form on our patchwork servers to send out confirmation emails to
>> random addresses. I was wondering if anyone else had experienced that
>> or had implemented some type of protection against this abuse?
> 
> I don't operate a Patchwork instance so I can't say about the
> regularity of such abuse. Jeremy might be able to offer better advice,
> however.

Yep, we see the same on patchwork.ozlabs.org - it seems that there's
bots that fill out random web forms with random values. I see about
20-40 per day.

I probably wouldn't bother with a captcha for our instance though; it's
a fairly trivial amount of email traffic for those confirmations, and
the non-confirmed accounts should get purged form the db after the
expiry.

So yes, if you do propose a patch for this, please make it optional via
a django setting :)

Cheers,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

patchwork.ozlabs.org wasn't consuming new mail yesterday

[Jeremy Kerr]

Hi all,

Yesterday, we updated the django framework on patchwork.ozlabs.org,
which seems to have caused an issue with the patch parser. The
consequence of this is that it was no longer adding incoming mail to the
database.

I've since fixed the issue, but there will still be a set of patches
missing, from new email from 3:00 UTC yesterday, until around 00:00 UTC
today.

If you'd like me to add any missing mail, send me a .mbox file and I'll
add manually. Patchwork will de-duplicate by message ID, so there's no
issues with overlapping those times.

Apologies for the inconvenience.

Cheers,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

[PATCH] templates: Move comments to above patch

[Jeremy Kerr]

It's much more likely that existing comments will affect the user's
actions, rather than the patch content itself, so put the comments
first.

Signed-off-by: Jeremy Kerr <j...@ozlabs.org>
---
 patchwork/templates/patchwork/patch.html | 32 
 1 file changed, 16 insertions(+), 16 deletions(-)

diff --git a/patchwork/templates/patchwork/patch.html 
b/patchwork/templates/patchwork/patch.html
index a05f00d..8b6d1fe 100644
--- a/patchwork/templates/patchwork/patch.html
+++ b/patchwork/templates/patchwork/patch.html
@@ -202,6 +202,22 @@ function toggle_headers(link_id, headers_id)
 
 
 
+{% for item in patch.comments.all %}
+{% if forloop.first %}
+Comments
+{% endif %}
+
+
+
+ {{ item.submitter|personify:project }}
+ {{ item.date }}
+
+
+{{ item|commentsyntax }}
+
+
+{% endfor %}
+
 {% if patch.diff %}
 
  Patch
@@ -220,20 +236,4 @@ function toggle_headers(link_id, headers_id)
 
 {% endif %}
 
-{% for item in patch.comments.all %}
-{% if forloop.first %}
-Comments
-{% endif %}
-
-
-
- {{ item.submitter|personify:project }}
- {{ item.date }}
-
-
-{{ item|commentsyntax }}
-
-
-{% endfor %}
-
 {% endblock %}
-- 
2.5.0

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

[PATCH] parsemail: Fix return value for find_content error case

[Jeremy Kerr]

If we fail to decode a message payload, we'll fail with the following:

  Traceback (most recent call last):
File "./patchwork/bin/parsemail.py", line 563, in 
  sys.exit(main(sys.argv))
File "./patchwork/bin/parsemail.py", line 553, in main
  return parse_mail(mail, args['list_id'])
File "./patchwork/bin/parsemail.py", line 464, in parse_mail
  (patch, comment, filenames) = find_content(project, mail)
  ValueError: need more than 2 values to unpack

- as the error condition for find_content only returns a 2-tuple. This
change fixes the error case to the 3-tuple return type.

Signed-off-by: Jeremy Kerr <j...@ozlabs.org>
---
 patchwork/bin/parsemail.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/patchwork/bin/parsemail.py b/patchwork/bin/parsemail.py
index f369fe4..3147431 100755
--- a/patchwork/bin/parsemail.py
+++ b/patchwork/bin/parsemail.py
@@ -231,7 +231,7 @@ def find_content(project, mail):
 
 # Could not find a valid decoded payload.  Fail.
 if payload is None:
-return (None, None)
+return (None, None, None)
 
 if subtype in ['x-patch', 'x-diff']:
 patchbuf = payload
-- 
2.5.0

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

[PATCH 2/4] parsemail: Fix default value of verbosity argument

[Jeremy Kerr]

With the current parsemail.py, the default value for the verbosity
argument does not exist in the VERBOSITY_LEVELS dict, so we get:

  Traceback (most recent call last):
File "./patchwork/bin/parsemail.py", line 569, in 
  sys.exit(main(sys.argv))
File "./patchwork/bin/parsemail.py", line 555, in main
  logging.basicConfig(level=VERBOSITY_LEVELS[args['verbosity']])
  KeyError: 20

This change uses an actual key instead.

Signed-off-by: Jeremy Kerr <j...@ozlabs.org>
---
 patchwork/bin/parsemail.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/patchwork/bin/parsemail.py b/patchwork/bin/parsemail.py
index 9640ff3..f369fe4 100755
--- a/patchwork/bin/parsemail.py
+++ b/patchwork/bin/parsemail.py
@@ -542,7 +542,7 @@ def main(args):
 group.add_argument('--list-id', help='mailing list ID. If not supplied '
'this will be extracted from the mail headers.')
 group.add_argument('--verbosity', choices=list_logging_levels(),
-   help='debug level', default=logging.INFO)
+   help='debug level', default='info')
 
 args = vars(parser.parse_args())
 
-- 
2.5.0

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

[PATCH 4/4] templates: don't emit tags for empty URLs, allow both web_url & webscm_url

[Jeremy Kerr]

Rather than always outputting one of web_url or webscm_url, only output
when these are present. This prevents us from emitting empty links if
both are missing.

Signed-off-by: Jeremy Kerr <j...@ozlabs.org>
---
 patchwork/templates/patchwork/projects.html | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/patchwork/templates/patchwork/projects.html 
b/patchwork/templates/patchwork/projects.html
index afaf6ab..0db1227 100644
--- a/patchwork/templates/patchwork/projects.html
+++ b/patchwork/templates/patchwork/projects.html
@@ -30,7 +30,8 @@
 
 {% if p.web_url %}
   {{p.web_url}}
-{% else %}
+{% endif %}
+{% if p.webscm_url %}
   {{p.webscm_url}}
 {% endif %}
  
-- 
2.5.0

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: getting meta-data out of patchwork

[Jeremy Kerr]

Hi Stephen,

> Is there anyway to get statistics out of patchwork about how long
> patches sit around, reject/accept rates, and patch sizes?

There's no exposed interface for stats at present, but if it's a
once-off on patchwork.ozlabs.org I'm happy to do a query. For more
general stuff, we can certainly looking at adding a report to the
patchwork UI.

Note that we don't (currently) track 'log' data (ie, when updates
actually occurred), so there's no way to tell (for example) if a patch
was accepted on the day it came in, or three months later.

Cheers,


Jeremy

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Changing person/author names

[Jeremy Kerr]

Hi Alexandre,

> I have a feature request. I had a quick look at the code and I have to
> admit that I'm not quite sure how to achieve it :)
> 
> Patches applied using pwclient will always use the first name that has
> been seen for a particular email address. While this is usually fine, It
> actually causes issues for people changing name without changing email
> addresses or people changing the capitalization of their name (less
> important though).

... and for folks that change names (I've had a few requests for that).

Yes, I think this is worthwhile.

> So I would prefer .save() to be called even if the email already exists
> in the database.

- provided that the Person does not have a patchwork account where the
name has previously been manually set; we'd probably need a boolean on
the profile model for that.

Regards,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Patchwork update on patchwork.ozlabs.org

[Jeremy Kerr]

Hi Patchworkers,

It's time I updated the patchwork instance on patchwork.ozlabs.org to
the latest 1.1 release.

This will involve some downtime while the database schema is updated.
We've done a test-run with the current data, so that *should* go
smoothly.

I plan to do this *next weekend* - most likely Saturday (26 March)
morning (UTC+8), but I'm flexible with the times if we need to shift
that a little.

Once it's complete, we'll look more like the more-recent instance at:

  https://patchwork.freedesktop.org/

Let me know if you have any feedback.

Cheers,


Jeremy

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH] management: Create 'rehash' command

[Jeremy Kerr]

Hi Stephen,

>> The rehash script, though undocumented and possibly unused at the
>> moment, likely has some value to some users. Howver, it makes more
>> sense to provide this command as a management command like 'retag'.
>> Do this.
>>
>> Signed-off-by: Stephen Finucane 
> 
> I'm still not sure if this should be converted to a command or just
> deleted. Does anyone use it? It seems like the hash could never really
> get out of date without manual DB modification (in which case you have
> more than one problem) thanks to the 'save' method.

This was useful when the hashes were first introduced, but wouldn't be
used much now. If we did need to change the hashing algorithm, (eg, move
to SHA1, or change which parts of a patch are included in the hashed
text), then it'd be required for that.

I'd say keep it in, since you've already done the work converting it :)

Cheers,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: ozlabs.org web server down?

[Jeremy Kerr]

Hi Brian,

> I can ping ozlabs.org, but pwclient/xmlrpc and HTTP to
> http://patchwork.ozlabs.org/ just wait indefinitely.

Looks like all web services are out there; I'll have a chat to Stephen.
Thanks for letting me know!

Regards,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH 2/2] project: Provide a setting to only consider git send-email patches

[Jeremy Kerr]

Hi Stephen,

> For the setup on patchwork.ozlabs.org, a flag for parsemail means I'd
>> need project-specific configuration outside of the admin system, which
>> seems a bit clunky.
> 
> So you're suggesting...not including this option anywhere?

No, just that it should be a project setting (ie, part of the Project
model), rather than something we pass as an argument to parsemail.

Regards,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH 2/2] project: Provide a setting to only consider git send-email patches

[Jeremy Kerr]

Hi Stephen,

>> One goal I have is to lower patchwork maintenance (ie limit the
>> number of fixes we need to do by hand). That's a small improvement
>> towards that direction.
>> 
>> Signed-off-by: Damien Lespiau 
> 
> I like this idea a lot: I don't think this needs to be a per-project
> setting, however. There's only one "entry point" for submitting
> patches - parsemail.py - so a flag on this application would be more
> than adequate for limiting access. This will remove the need for the
> database migration. What are your thoughts on adding this flag in
> place of the project setting?

This means that patchwork will only work for projects (and
contributors!) that use git though. I'd prefer we keep patchwork fairly
VCS-agnostic.

For the setup on patchwork.ozlabs.org, a flag for parsemail means I'd
need project-specific configuration outside of the admin system, which
seems a bit clunky.

Regards,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Patchwork git repo now on github

[Jeremy Kerr]

Hi all,

With Stephen and Damien working on patchwork maintenance now, we need to
have a repo that allows commits from multiple maintainers. To this end,
we're shifting the "official" git repo from ozlabs.org to:

 https://github.com/getpatchwork/patchwork

which can be cloned over git:

 git clone git://github.com/getpatchwork/patchwork

I'll be keeping the ozlabs repo around for a little while, but will
remove it in a couple of weeks' time.

Cheers,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: patchwork configuration issues

[Jeremy Kerr]

Hi,

>   File "/srv/patchwork/patchwork/bin/parsemail.py", line 410, in main
> django.setup()
> AttributeError: 'module' object has no attribute 'setup'

Could you include the version of django that you're using there?

Thanks,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Patchwork updated, request for lost patches

[Jeremy Kerr]

Hi all,

The good news: I've updated the patchwork git tree - with the main set 
of changes from Brian Norris  Stephen Finucane (full shortlog below), 
and some further updates for django 1.7 onwards. The patch-tagging 
(Acked-by/Reviewed-by/Tested-by) code is now in the main repo, and the 
database schema for this is now done.


The bad news: when updating the instance on patchwork.ozlabs.org, I was 
missing a fix for django-1.7, for the patch parser code. This means that 
we missed about 22 hours worth of patches.


The parser is fixed, so we should be receiving new patches again. If 
you're maintaining a project in patchwork, and are missing patches, just 
send me a .mbox of mail between:


  May 28 01:50 UTC and
  May 29 00:33 UTC

(times are be conservative; duplicate messages will get filtered out 
fine), and I'll feed that directly into patchwork.


I've already done this for the skiboot, linuxppc and buildroot projects. 
If your list is hosted on lists.ozlabs.org too, I can take patches from 
mailman archive directly, but please still send me an email.


Sorry about the hassle, but keep me posted on how things are going.

Cheers,


Jeremy


Brian Norris (9):
  xmlrpc: add 'archived' status to patch summary
  xmlrpc: include UNIX mbox 'From ' header in patch_get_mbox()
  TODO: remove completed items
  INSTALL: fix mysql syntax
  pwclient: add project option to single-patch commands
  pwclient: exit on first patch which fails to apply
  xmlrpc: support filtering by 'archived'
  pwclient: drop unused variables
  pwclient: support 'archived' filtering and updating

Bryce Harrington (2):
  INSTALL: Fix some minor typos
  INSTALL: Fix indentation (whitespace only change) to 4 spaces

Damien Lespiau (1):
  Always use #!/usr/bin/env python

Jeremy Kerr (14):
  Move to a more recent django project structure
  lib/apache2: use django-1.7-compatible wsgi application
  lib/apache2: Update sample configuration files for new
  staticfiles application
  docs: Fix documentation for new settings infrastructure
  bin: Fix references to old settings module location
  Update documentation and default settings to suit patchwork
  deployment model
  docs: Add collectstatic step to installation instructions
  Add patch tag infrastructure
  doc: Add NEWS file for recent updates
  patchwork/utils: Remove unnecessary MultiplePatchForm import
  cron: Move patchwork-cron script to a management command
  tests: Remove old-style test runner module
  parsemail: run django.setup to initialise Models
  parsemail: Don't catch all exceptions when a Project isn't found

Johannes Berg (1):
  views/xmlrpc: fix xmlrpc delegate filtering

Michael Ellerman (2):
  parser: Fix parsing of patches with a trailing no-newline marker
  Make the submitter name link to a query for that submitter

Stephen Finucane (8):
  Resolve removed 'AUTH_PROFILE_MODULE' setting
  views: Replace 'mimetype' with 'content_type'
  settings: Resolve all but one Django 1.7 warning
  docs: Use 'prod' and 'dev' requirements files
  settings: Restructure settings file
  Integrate 'django.contrib.staticfiles'
  settings: Split 'settings.py' into multiple files
  tox: Add tox.ini file

aldot (1):
  pwclient: honor PAGER for view

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Looking for the source code for patchwork.ozlabs.org

[Jeremy Kerr]

Hi James,


The specific piece I'm looking for is to Support a workflow in SCSI
where a patch is considered ready to apply once it has a pre-determined
number of acked-by's, tested-by's or reviewed-by's.  The patchwork
instance at Ozlabs is counting these in the A R T columns of every
project.  However, when I look at what is supposed to be the source
repository: git://ozlabs.org/home/jk/git/patchwork, the code for doing
this is missing.  Is there another more up to date repository which does
have the A R T counting?


The patchwork instance at patchwork.ozlabs.org has a few in-development 
changes, of which the A/R/T counts are the main delta.


Because it's still a little rusty, I haven't merged to the main repo - 
it needs a bit of a cleanup, which will involve a schema change. So, I'd 
be reluctant to advise running this code as it is as the moment. If 
you're still keen though, I'll put it in a separate branch in the 
patchwork repo.


Cheers,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH 1/6] xmlrpc: add 'archived' status to patch summary

[Jeremy Kerr]

Hi Brian,


This will now show up in 'pwclient info'.

Signed-off-by: Brian Norris computersforpe...@gmail.com


Thanks for the contributions, I've applied this series.

Cheers,


Jeremy

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [RFC/PATCH] fix xmlrpc delegate filtering

[Jeremy Kerr]

Hi Johannes,


Trying to use
   pwclient list -d 'johan...@sipsolutions.net'

doesn't result in any patches listed - it seems that the filter is
constructed wrongly on the xmlrpc server side (going by how the
submitter filter is done.)


Thanks, applied.


Jeremy

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Delegate Vs Reviewer

[Jeremy Kerr]

Hi all,

 On Mon, Nov 24, 2014 at 05:35:11PM +, Damien Lespiau wrote:
 I'd like to be able to assign reviewers to patches. That seems very
 close to what a delegate is today. Would anyone have objections if I
 just use Delegate as Reviewer?
 
 I'm honestly not sure what Delegate is intended for. It seems like its
 purpose is open to your interpretation.

One example: on the powerpc list, we have an overall maintainer, and
several sub-maintainers, who are generally responsible for their own
subsystem or set of hardware.

If a patch for one of the sub-maintiners' areas appears, it can be
delegated to the appropriate person to review, test, and/or merge. The
change is then included in the next merge-request from the
sub-maintainer to the main maintainer.

In this case, patch review is only one of the tasks of the delegate.

Cheers,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH] COPYING: Add the text of the GPLv2 license

[Jeremy Kerr]

Hi Damien,


I missed having that file when I first looked at the git repository to
figure out the license of the project. So add one.


Thanks! Applied.

Cheers,


Jeremy

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH] docs: Add the Debian package name for virtualenv/mysql

[Jeremy Kerr]

Hi Damien,


Signed-off-by: Damien Lespiau damien.lesp...@intel.com
---
  docs/HACKING | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)


Thanks, applied.

Cheers,


Jeremy

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH] parsemail: Honour existing PYTHONPATH

[Jeremy Kerr]

Hi Damien,


For setups where the patchwork dependencies are not part of the system
packages, one needs to indicate where to find them. parsemail.sh was
overriding PYTHONPATH without leaving us a chance to join in.


OK, makes sense. Applied.

Cheers,


Jeremy

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH] jquery: Fix jqeury typo

[Jeremy Kerr]

Hi Damien,


@@ -1,7 +1,7 @@
  This directory contains the jQuery Javascript library, and jQuery
  Table Drag  Drop plugin. These are available from:

-  http://jqeury.com/
+  http://jquery.com/


That seems a pretty common typo for me. Thanks! Applied.


Jeremy

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH 1/2] docs: Add a note about how to run tests

[Jeremy Kerr]

Hi Damien,


Given that hacking on patchwork is spaced in time,


You too huh? :)

Thanks, applied.


Jeremy

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH 00/46] A re-styling of Patchwork

[Jeremy Kerr]

Hi Damien,


I'm part of the i915 driver team and we'd like to use patchwork. However, there
are a few things getting in the way: the bigger item is that patchwork doesn't
scale very well above a certain threshold of patches.

We have a number of ideas to improve the situation, mainly being able to
manipulate a higher level object, the series, instead of patches.


That's something that I'd definitely like to see. Being able to work 
with an entire series would make things a lot more manageable in this 
situation.




But I'm getting ahead of myself here. This is mostly a spare time project, and
this re-styling series is only a step on the road.

Belén, a designer working in my office on Yocto (www.yoctoproject.org), has
been kind enough to spend some of her spare time doing a pass on the patchwork
HTML/CSS and I've sprinkled a few ideas of my own as well. We've tried to
respect what was there and offer a slight refresh for the first pass. It's
possible to see the result at:

   http://patchwork.lespiau.name/


Wow, looks great!

Since you don't touch the models there, I'd like to set up a beta site, 
(which would use the same database as the regular site) and use your 
changes on that. Once we've got some feedback from core patchwork users, 
we can merge these to the main site.


So, I'll get the reworked A/R/T code whipped into shape, apply your 
regesign series on top of that, and we can go from there.


Thanks for the contributions!

Cheers,


Jeremy

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH 40/46] patch: Single out the commit message

[Jeremy Kerr]

Hi Damien,


All 'Comments' are stored the same way in the db, but I believe it's
worth making the distinction between introducing what the patch does and
eventual review comments.

Signed-off-by: Damien Lespiau damien.lesp...@intel.com
---
  apps/patchwork/models.py   | 11 ++-
  templates/patchwork/patch.html | 18 +-
  2 files changed, 27 insertions(+), 2 deletions(-)

diff --git a/apps/patchwork/models.py b/apps/patchwork/models.py
index 4bed9b8..b9c7794 100644
--- a/apps/patchwork/models.py
+++ b/apps/patchwork/models.py
@@ -188,7 +188,16 @@ class Patch(models.Model):
  return self.name

  def comments(self):
-return Comment.objects.filter(patch = self)
+comments = Comment.objects.filter(patch = self)
+# len() will trigger a queryset evaluation. That's what we want as
+# we need the full list of comments. That the list is then sliced into
+# (commit message, list of followup comments) won't hit the db twice.
+n_comments = len(comments)
+if n_comments  2:
+return (comments[0], comments[1:])
+if n_comments == 1:
+return (comments[0], ())
+return (None, ())


This doesn't guarantee that you'll pick the correct comment for the 
commit message - you need to find the comment that has the same 
message-id as the patch.


Also, could we have this as a separate function? I'd like to 
Patch.comments() for the complete set as a single list. Your template 
might be a bit cleaner if have separate functions for 
commit-message-comments and follow-up-comments too.


Cheers,


Jeremy

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH 25/46] filters: Rewrite the submitter autocompletion code

[Jeremy Kerr]

Hi Damien,


We now have a nice(r) autocompletion widget that will popup the list of
options directly underneath the input fild.


woot!


A slight change in behaviour is that we now don't allow free form
submitter search, ie we need a valid completion to search with that
specific person (through its primary key). I didn't remove the backend
logic that allows the free form mechanism, but maybe we should.


Originally, I've included that for the case where we have no client-side 
javascript, and hence no completion. It looks like that'll still work 
with this change though - is that right?


Cheers,


Jeremy

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH 26/46] tests: Fix the submitter completion tests

[Jeremy Kerr]

Hi Damien,


Because we've changed the format of the json file we return to be a
proper object rather than the db dump django does by default,
there's no 'fields' sub-object anymore.


If you're doing a v2, could you roll this into the original change?

Cheers,


Jeremy

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH 0/4] Add git-am -s, --signoff

[Jeremy Kerr]

Hi Bernhard,

 git-am was lacking a way to pass -s
 I lightly tested a few commands but since this revamps the whole
 option-handling testers are welcome :) Please holler if i broke
 something!
 
 Please consider merging.

All looks good - I've applied the series (including 5/n and 6/n) and
pushed. Thanks for contributing!

I've made one change:

  http://git.ozlabs.org/?p=patchwork;a=commitdiff;h=3d0da317

Because I needed to git-am a set of patches, and needed to preserve the
argument order. Is there any potential problem (other not filtering
duplicate patches) with this?

Cheers,


Jeremy

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH 1/2] INSTALL: Update the database configuration instructions

[Jeremy Kerr]

Hi Damien,

 That's the new (django 1.5+) way of defining databases, strictly
 following the instructions wasn't working. This should save the next
 user a bit of time.

Ah, excellent. Thanks for the contribution, I've applied and pushed.

Cheers,


Jeremy

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH 0/4] Using virtual env with patchwork

[Jeremy Kerr]

Hi Damien,

 This series introduces a few different configuration definitions, ie. a
 certain combination of dependencies people may be interested in. virtualenv
 allows to easily setup those configurations and provide isolation from the
 system-wide python packages.

Looks good, applied.

 I'm not sure how important it is to test with older versions of django
 considering how easy it is to run a server with virtualenv to select a 
 specific
 django version. I, for one, would like to hard-depend on 1.7 to start using 
 the
 new DB migration facility.

Not everyone is using virtualenv though; and I'd prefer not to add more
hurdles to those who want to develop patchwork.

Cheers,


Jeremy


___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH 0/4] Add git-am -s, --signoff

[Jeremy Kerr]

Hi Bernhard,

 No, thats fine if you don't mind dups.

I'd prefer no dups too, but I think the ordering is more important. We
could always add a warning if necessary, but I think it's okay for now.

Cheers,


Jeremy

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Fwd: [Users] Postgresql update

[Jeremy Kerr]

Hi Patchworkers,

An upcoming outage - patchwork.ozlabs.org will be down for 30-60 mins,
at 01:00 UTC.

Cheers,


Jeremy


 Forwarded Message 
Subject: [Users] Postgresql update
Date: Tue, 12 Aug 2014 09:42:49 +1000
From: Stephen Rothwell


Hi all,

I am going to update Postgresql to version 9.3 (from 9.1) today at
about 11:00 (Canberra time - in about an 80 minutes).  This will mean
that the databases are unavailable for some amount of time as they will
all be dumped and reloaded.  This will take an unspecified amount of
time since the patchwork database is quite large (!).  Other main
things affected will be Roundcube and kisskb.

Sorry for the inconvenience.
-- 
Cheers,
Stephen Rothwells...@canb.auug.org.au



___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH] templates/patch-list: patches table has 6 columns

[Jeremy Kerr]

Hi Andreas,


I'd like to ping this also. But I think it is outdated since the 'A/R/T'
column was added. Unfortunately I can't find those patches in git
currently. Could you please push those changes too?


The A/R/T stuff needs a bit of a rework before I push it to the git repo 
- mostly from feedback on projects using the instance on ozlabs.org. 
Working on that now though :)


Cheers,


Jeremy

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH 1/4] INSTALL: adopt PYTHONPATH

[Jeremy Kerr]

Hi Andreas,


ping?


Thanks! All applied.

Sorry for the delay, been a little flat-out lately, If only I had some 
way of tracking with patches were still outstanding...


Cheers,


Jeremy


___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH] Fallback to common charsets when charset is None or x-unknown

[Jeremy Kerr]

Hi Siddesh,

 We recently encountered a case in our glibc patchwork instance on
 sourceware, where a patch was dropped because it had x-unknown
 charset.  I used the following patch to fix this in our instance.  The
 fix I used was to fall back on a set of encodings (instead of just
 utf-8) when the charset is not mentioned or if it is set as x-unknown.

I tried this patch with a testcase from the failing patch you forwarded
me, and get the following:

ERROR: testPatch (patchwork.tests.test_patchparser.CharsetFallbackPatchTest)
--
Traceback (most recent call last):
  File
/home/jk/devel/patchwork/apps/patchwork/tests/test_patchparser.py,
line 432, in testPatch
(patch, comment) = find_content(self.project, self.mail)
  File /home/jk/devel/patchwork/apps/patchwork/bin/parsemail.py, line
180, in find_content
decoded_payload = try_decode(payload, cset)
  File /home/jk/devel/patchwork/apps/patchwork/bin/parsemail.py, line
152, in try_decode
payload = unicode(payload, charset)
LookupError: unknown encoding: none

In this case, it looks like the encoding is actually ascii, but the
none charset is causing trouble:

  Content-Type: text/plain; charset=none
  Content-Transfer-Encoding: QUOTED-PRINTABLE

So the failure there is due to the none, and not the charset of the
patch content. I've put together an updated change (will send it
shortly), could you let me know if it looks OK to you?

Regards,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

[PATCH] parsemail: Fallback to common charsets when charset is None or x-unknown

[Jeremy Kerr]

From: Siddhesh Poyarekar siddh...@redhat.com

We recently encountered a case in our glibc patchwork instance on
sourceware, where a patch was dropped because it had x-unknown
charset.

This change adds a fallback on a set of encodings (instead of just
utf-8) when the charset is not mentioned or if it is set as x-unknown.

Minor changes and testcase by Jeremy Kerr j...@ozlabs.org

Signed-off-by: Siddhesh Poyarekar siddh...@redhat.com
Signed-off-by: Jeremy Kerr j...@ozlabs.org

---
 apps/patchwork/bin/parsemail.py |   40 -
 apps/patchwork/tests/mail/0010-invalid-charset.mbox |   91 
 apps/patchwork/tests/test_patchparser.py|   11 +
 3 files changed, 136 insertions(+), 6 deletions(-)

diff --git a/apps/patchwork/bin/parsemail.py b/apps/patchwork/bin/parsemail.py
index b6eb97a..2a4866f 100755
--- a/apps/patchwork/bin/parsemail.py
+++ b/apps/patchwork/bin/parsemail.py
@@ -24,6 +24,7 @@ import re
 import datetime
 import time
 import operator
+import codecs
 from email import message_from_file
 try:
 from email.header import Header, decode_header
@@ -147,6 +148,13 @@ def find_pull_request(content):
 return match.group(1)
 return None
 
+def try_decode(payload, charset):
+try:
+payload = unicode(payload, charset)
+except UnicodeDecodeError:
+return None
+return payload
+
 def find_content(project, mail):
 patchbuf = None
 commentbuf = ''
@@ -157,15 +165,35 @@ def find_content(project, mail):
 continue
 
 payload = part.get_payload(decode=True)
-charset = part.get_content_charset()
 subtype = part.get_content_subtype()
 
-# if we don't have a charset, assume utf-8
-if charset is None:
-charset = 'utf-8'
-
 if not isinstance(payload, unicode):
-payload = unicode(payload, charset)
+charset = part.get_content_charset()
+
+# Check that we have a charset that we understand. Otherwise,
+# ignore it and fallback to our standard set.
+if charset is not None:
+try:
+codec = codecs.lookup(charset)
+except LookupError:
+charset = None
+
+# If there is no charset or if it is unknown, then try some common
+# charsets before we fail.
+if charset is None:
+try_charsets = ['utf-8', 'windows-1252', 'iso-8859-1']
+else:
+try_charsets = [charset]
+
+for cset in try_charsets:
+decoded_payload = try_decode(payload, cset)
+if decoded_payload is not None:
+break
+payload = decoded_payload
+
+# Could not find a valid decoded payload.  Fail.
+if payload is None:
+return (None, None)
 
 if subtype in ['x-patch', 'x-diff']:
 patchbuf = payload
diff --git a/apps/patchwork/tests/mail/0010-invalid-charset.mbox 
b/apps/patchwork/tests/mail/0010-invalid-charset.mbox
new file mode 100644
index 000..a8614ef
--- /dev/null
+++ b/apps/patchwork/tests/mail/0010-invalid-charset.mbox
@@ -0,0 +1,91 @@
+From libc-alpha-return-50517-siddhesh=redhat@sourceware.org Thu Jun  5 
10:36:33 2014
+Received: (qmail 11948 invoked by alias); 4 Jun 2014 17:51:01 -
+Mailing-List: contact libc-alpha-h...@sourceware.org; run by ezmlm
+List-Id: libc-alpha.sourceware.org
+Sender: libc-alpha-ow...@sourceware.org
+Date: Wed, 4 Jun 2014 17:50:46 +
+From: Joseph S. Myers jos...@codesourcery.com
+To: libc-al...@sourceware.org
+Subject: Fix pow overflow in non-default rounding modes (bug 16315)
+Message-ID: pine.lnx.4.64.1406041749420.3...@digraph.polyomino.org.uk
+MIME-Version: 1.0
+Content-Type: multipart/mixed;
+   boundary=-1152306461-1522705971-1401904246=:3719
+Content-Length: 24171
+
+---1152306461-1522705971-1401904246=:3719
+Content-Type: text/plain; charset=none
+Content-Transfer-Encoding: QUOTED-PRINTABLE
+
+This patch, relative to a tree with
+https://sourceware.org/ml/libc-alpha/2014-06/msg00076.html applied,
+fixes bug 16315, bad pow handling of overflow/underflow in non-default
+rounding modes.  Tests of pow are duly converted to ALL_RM_TEST to run
+all tests in all rounding modes.
+
+There are two main issues here.  First, various implementations
+compute a negative result by negating a positive result, but this
+yields inappropriate overflow / underflow values for directed
+rounding, so either overflow / underflow results need recomputing in
+the correct sign, or the relevant overflowing / underflowing operation
+needs to be made to have a result of the correct sign.  Second, the
+dbl-64 implementation sets FE_TONEAREST internally; in the overflow /
+underflow case, the result needs recomputing in the original rounding
+mode.
+
+Tested x86_64 and x86 and ulps updated accordingly.
+
+(auto-libm-test-out diffs omitted below.)
+
+2014

Re: [PATCH] post-receive: exclude commits from the patch update step

[Jeremy Kerr]

Hi Brian,

 Actually, I think the rev-parse serves as a little extra parameter
 checking. Unless someone can pin a good reason one way or the other,
 I'll stick with my original patch as I sent it. And I've been testing
 that version for a few weeks now.

OK, just did some testing with that: it looks like the rev-parse will
barf to stderr if EXCLUDE is empty. Should we do this conditionally?

Or, given that we're essentially doing an exclude on ${1} there,
how about we just add any excludes to the rev-list instead? Something
like:

  EXCLUDES=(refs/heads/upstream)

  [...]

  update_patches()
  {
local cnt; cnt=0
include=${2}
excludes=(${1} ${EXCLUDES[@]})
revs=$include ${excludes[@]/#/^}
for rev in $(git rev-list --no-merges --reverse $revs); do
  [...]


Cheers,


Jeremy

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH] Fallback to common charsets when charset is None or x-unknown

[Jeremy Kerr]

Hi Siddesh,


We recently encountered a case in our glibc patchwork instance on
sourceware, where a patch was dropped because it had x-unknown
charset.  I used the following patch to fix this in our instance.  The
fix I used was to fall back on a set of encodings (instead of just
utf-8) when the charset is not mentioned or if it is set as x-unknown.

I hope this is useful.  I'd love to know if you all think there is a
better way to fix this so that I can implement that in our instance
instead of my hack.


Just one thing I noticed when applying this:


+
+# If there is no charset or if it is unknown, then try some common
+# charsets before we fail.
+if charset is None or charset == 'x-unknown':
+try_charsets = ['utf-8', 'windows-1252', 'ascii', 'iso-8859-1']


Is there any point including ascii there? If it didn't parse as utf-8, I 
don't think it'll parse as ascii either.


Also, could you send me a Signed-off-by line for this patch too?

Regards,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH] post-receive: exclude commits from the patch update step

[Jeremy Kerr]

Hi Brian,


When merging upstream work related to other projects into your own
project repository, you probably don't want to check for (and try to
update) the status on every change-set in the merge. So add a list of
references (branches, tags, commits, etc.) whose commits should be
ignored in the patch update step.

This could be used, for example, to set:

 EXCLUDE=refs/heads/upstream

Then when you're ready to merge in new upstream code, you first update
the 'upstream' branch before pushing your own.


This looks good to me, but I'd like to get Martin's OK, as he's the 
author of this script.


Martin - how does this look?



Signed-off-by: Brian Norris computersforpe...@gmail.com
---
  tools/post-receive.hook | 9 -
  1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/tools/post-receive.hook b/tools/post-receive.hook
index 4fb741d3ea98..a38522e22f35 100755
--- a/tools/post-receive.hook
+++ b/tools/post-receive.hook
@@ -8,6 +8,12 @@ set -eu

  #TODO: the state map should really live in the repo's git-config
  STATE_MAP=refs/heads/master:Accepted
+#
+# ignore all commits already present in these refs
+# e.g.,
+#   EXCLUDE=refs/heads/upstream refs/heads/other-project
+#
+EXCLUDE=

  PWDIR=/srv/patchwork/apps/patchwork

@@ -39,7 +45,8 @@ set_patch_state()
  update_patches()
  {
local cnt; cnt=0
-  for rev in $(git rev-list --no-merges --reverse ${1}..${2}); do
+  for rev in $(git rev-parse --not ${EXCLUDE} |
+   git rev-list --stdin --no-merges --reverse ${1}..${2}); do
  if [ $do_exit = 1 ]; then
echo I: exiting... 2
break



Cheers,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [pull request v2] Pull request for branch yem/pwclient

[Jeremy Kerr]

Hi Yann,

 On 2014-07-01 20:14 +0200, Yann E. MORIN spake thusly:
 Here is an attempt to change the format of ~/.pwclientrc so it is easier
 to manage such scatered-around projects.
 
 Does this new series address your concerns about the previous one?
 Do you have additional comments?

This looks great, applied.

Thanks!


Jeremy

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [pull request] Pull request for branch yem/pwclient

[Jeremy Kerr]

Hi Yann,

 As I previosuly explained [0], it is not really convenient to use pwclient
 when dealing with different projects hosted on different servers.
 
 Here is an attempt to change the format of pwclientrc so it is easier to
 manage such scatered-around projects.

Oh awesome, I've always wanted to do this. My only concern is
compatibility with older config files. How difficult would it be to
automatically migrate?

Cheers,


Jeremy

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Make Acked/Reviewd/Tested-by tags visible

[Jeremy Kerr]

Hi Yann,

 How and when are the tags parsed?

They should be parsed when the original patch is received, and when any
follow-ups are appended to the patch. However:

 I noticed that (in the Buildroot patchwork [0]) some tags were not
 accounted for:
 http://patchwork.ozlabs.org/patch/354654/  (missing reviewed-by)
 http://patchwork.ozlabs.org/patch/354590/  (missing acked-by)
 [...]

.. it looks like I may be missing a case there. I'll take a look.


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Make Acked/Reviewd/Tested-by tags visible

[Jeremy Kerr]

Hi Yann,

 How's this look?

   http://ozlabs.org/~jk/tmp/patchwork-ART.png
 
 Just a little reminder on this...
 
 When are you planning on deploying that?
 Will it also show up in the output of pwclient?

I've just rolled out the initial code to patchwork.ozlabs.org. The tag
counts are all at zero to start with; I've triggered a re-parse of all
of the existing comments, so the counts should reflect reality when that
process finishes.

The filter UI and pwclient interface are still yet to be done though;
they're next on my todo list :)

Cheers,


Jeremy

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Make Acked/Reviewd/Tested-by tags visible

[Jeremy Kerr]

Hi all,

 I've just rolled out the initial code to patchwork.ozlabs.org. The tag
 counts are all at zero to start with; I've triggered a re-parse of all
 of the existing comments, so the counts should reflect reality when that
 process finishes.

On second thoughts, I've disabled the display of the counts until the
update is complete, so we don't have false info in the UI in the
meantime. I'll re-enable once that's done.

Cheers,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Patches spontaneously changing ownership

[Jeremy Kerr]

Hi Ralf,

 Earlier today I submitted a patch to my own patchwork which did show
 up as submitted by user root - but my email address.  Turned out
 the root account which I'm only using for django and my normal
 patchwork account ralf where both using the same email address
 and I think that eventually lead patchwork into confusion.

Hm, interesting. Looks like I've got the (admin) user relation is being
mixed up with the person relation somewhere.

 Something as user friendly as update patchwork_patch set submitter_id = 6814
 where submitter_id = 4; resolved the issue again but that should probably
 be fixed permanently.

OK, could you send me:

 SELECT * FROM patchwork_person
   WHERE user_id IN (4, 6814)
 OR email ILIKE '%r...@linux-mips.org%';


(off-list if you prefer).

Thanks!


Jeremy

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Prevent anonymous users from viewing patchwork

[Jeremy Kerr]

Hi Joon,

 We are currently testing patchwork for internal company use. We would
 like anonymous users not to be able to view the patchwork site at all.
 However, I cannot seem to find how to do this. Any help would be
 appreciated.

There's no facility in the patchwork code itself to do this, but I'd
suggest simply configuring your HTTP server to only allow HTTP
authenticated connections. If you have a company-internal directory
service (such as LDAP), you could hook your web server into that.

Regards,


Jeremy

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Using From to start commit message

[Jeremy Kerr]

Hi Thierry,

 I've noticed something strange happening when a commit message starts
 with the word From. When downloading the commit as mbox, I see that
 the line is escaped using a '' character, which causes git am to
 ignore the line.
 
 This can be seen in the following patch for example:
 
   http://patchwork.ozlabs.org/patch/330330/
 
 I'm not familiar with the internals of patchwork, but I would expect
 this to be perfectly legal in a commit message. I could imagine that
 to be special treatment for From lines in mailboxes, but that should
 not apply to the text in commit messages.

Hm, interesting. This escaping is generally required for the file to be
a valid mbox; the 'Fromspace' sequence defines the beginning of a new
message (hence the escaping with ''), but it looks like this isn't what
git am expects.

We could generate the message without the escaping (and drop the initial
From_ line too), but that may break other uses of the mailbox files. I'm
not sure how many other consumers there are though...

Carl - I see you've done some work on git am's parser for From_ syntax;
do you know what's happening here?

Regards,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Bug tracker?

[Jeremy Kerr]

Hi Albert,

   git clone git://ozlabs.org/~jk/patchwork
 
 This command won't work. I believe that the right git clone command
 should use another URI:

Sorry about that! I've fixed the issue, the original URL should work now.

Cheers,


Jeremy

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Bug tracker?

[Jeremy Kerr]

Hi Gary,

 Does patchwork have a bug tracker somewhere?

This mailing list has been sufficient so far; we don't have the bug
report volume to justify a bug tracker at this stage.

  I'm experimenting
 with it and I found what is probably a bug (in that bundle URLs
 don't work if a bundle name contains spaces, there is probably
 a missing urlencode/urldecode somewhere).  I'd mail you a patch
 but I'm borrowing someone else's patchwork instance at present
 so I don't have access beyond the web interface to investigate.

Ah, thanks for letting us know. I'll take a look.

If you're keen to try it out locally, it's possible to set up a
development patchwork instance, using django's in-built web server:

 git clone git://ozlabs.org/~jk/patchwork
 cd patchwork/apps
 touch local_settings.py
 # using postgres as a database, using the default name of 'patchwork':
 createdb patchwork
 ./manage.py syncdb
 ./manage.py runserver

- this will run a local instance of patchwork on http://127.0.0.1:8000/

Hope this helps, and let me know if you need a hand with anything.

Cheers,


Jeremy



___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: ordering by submitter, then date

[Jeremy Kerr]

Hi Bjorn,

 I use the order-by-submitter view, e.g.,
 http://patchwork.ozlabs.org/project/linux-pci/list/?order=submitter,
 to weed out superseded patches.
 
 This would be easier if I could use two sort keys: first by submitter,
 then by date.  Right now, they are ordered by submitter, but patches
 from a single submitter are ordered in a way I haven't figured out
 yet.
 
 Is there a way to do this already?  If not, how hard would it be to add?

There's no way to specify two orderings at present. However, I've just
sketched up a change to apply the date ordering as a secondary whenever
a non-default order is selected. So the ordering you'll get with that
view is by submitter, then by date within each submitter. The same
applies to any other non-default ordering.

Would that sort out (pun intended) this problem for you? :)

Cheers,


Jeremy

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [RFC] parsemail.py: Don't search for patches in replies

[Jeremy Kerr]

Hi Markus,

 Make sure we don't attempt to search for a patch in a reply e-mail.
 There are MUAs out there who leave the quoted e-mail intact without
 prepending quote characters such as  at the beginning of each line.

D'oh!

 When that happens, parse_patch() thinks the quoted patch is new. The
 result are multiple database entries containing the same patch (one for
 each such reply) when one would really expect a consolidated thread
 containing the entire discussion and only one copy of the patch.

OK, sounds like something that we might be able to address, but I don't
think this is the correct approach (as others have mentioned too). We
don't want to drop follow-ups, as they may contain patches that we need
to track.

Perhaps there's some other way we could identify these replies that
shouldn't be parsed as patches? Could you send an unmodified (mbox-form)
mail that we can take a look at?

Cheers,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH 1/3] Fix post-receive hook to not update the previously pushed commit

[Jeremy Kerr]

Hi Carl,


Previously, the post-receive hook would always examine one commit that
had been previously pushed, (when the purpose of the hook is only to
example newly-pushed commits). We fix this by simply dropping the '^'
in the commit-range specification.


Thanks for the series; I've applied it all, plus the fix for the 
post-receive hook.


Are you okay with me adding your signed-off-by line to all four patches?

Cheers,

Jeremy

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [U-Boot] patchwork does not pick my patch

[Jeremy Kerr]

Hi all,

 On 11/21/2013 12:14 PM, Masahiro Yamada wrote:
 Hi.

 I posted a patch to change a file permission.
 This:
 http://lists.denx.de/pipermail/u-boot/2013-November/167573.html

 and again:
 http://lists.denx.de/pipermail/u-boot/2013-November/167608.html


 But my patch would not appear on patchwork.
 (Maybe because there is no diff line.)
 
 I think so too. There is some handling for renames without patch in
 patchwork, but I think there is no handling for just file permission
 changes. That should be fixed in patchwork then.

Yep, that's correct - there's no support for these permission changes at
present. I'll add that!

Cheers,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: Make Acked/Reviewd/Tested-by tags visible

[Jeremy Kerr]

Hi Yann,

 We would like to suggest that the web GUI and the pwclient CLI both
 display such tags besides each patch, a bit like (hypotetical output of
 pwclient):
 
 ID Tags State   Subject
 123456 ART  New Wonderfull patch to apply quickly

I've been looking to implement this for a while, but didn't have any
good ideas about how to present this in the UI. I love your proposal
here with the flags, it's simple and I think would work well for the
majority of cases.

 Since patchwork already inserts those tags to the commit log (eg.
 pwclient view PATCH-ID), I hope it would be easy enough to also display
 it in the patch list.

It's easy enough to pull these out of the comment at patch-display time,
but a little more tricky with list views (without some horrible SQL
joining). I'll work out a nice way to to this; I think it'll involve
applying these flags when the follow-up comment is parsed.

Other projects have different policies for their tagging (some require 3
acks, for example), which will introduce a little complexity there too,
but that's not insurmountable either.

Cheers,


Jeremy
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: filtering email?

[Jeremy Kerr]

Hi Kumar,

 Yes, that would be great.

OK, I'll get it set up. It might be next week before I can get it done
though.

 Where is the arm setup hosted ?

The IMX folks have their patchwork instance at:

 http://patchwork.ozlabs.org/project/linux-imx/list/

Cheers,


Jeremy

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork