Re: [PATCH 10/10] [RFC] Fuzzing harness

[Andrew Donnellan]

On 30/06/17 15:37, Daniel Axtens wrote:

Ah yes, sorry, you do.

I just downloaded it into the source dir and set PATH so py-afl-fuzz
could find it.


apt-get install --no-install-recommends afl works too :)

--
Andrew Donnellan  OzLabs, ADL Canberra
andrew.donnel...@au1.ibm.com  IBM Australia Limited

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH 10/10] [RFC] Fuzzing harness

[Daniel Axtens]

>
> On 28/06/17 17:48, Daniel Axtens wrote:
>
>> diff --git a/tools/docker/Dockerfile b/tools/docker/Dockerfile
>> index ff05707a6049..266603e3bdcf 100644
>> --- a/tools/docker/Dockerfile
>> +++ b/tools/docker/Dockerfile
>> @@ -48,6 +48,8 @@ RUN cat /tmp/bashrc >> /home/patchwork/.bashrc
>>
>>  COPY tools/docker/entrypoint.sh /usr/local/bin/entrypoint.sh
>>
>> +RUN apt-get install -y cython cython3; pip3 install python-afl; pip
>> install python-afl
>> +
>>
>
> Do you need afl-fuzz itself as well?
>
> Ah yes, sorry, you do.

I just downloaded it into the source dir and set PATH so py-afl-fuzz could
find it.

Regards,
Daniel


> --
> Andrew Donnellan  OzLabs, ADL Canberra
> andrew.donnel...@au1.ibm.com  IBM Australia Limited
>
>
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH 10/10] [RFC] Fuzzing harness

[Andrew Donnellan]

On 28/06/17 17:48, Daniel Axtens wrote:

diff --git a/tools/docker/Dockerfile b/tools/docker/Dockerfile
index ff05707a6049..266603e3bdcf 100644
--- a/tools/docker/Dockerfile
+++ b/tools/docker/Dockerfile
@@ -48,6 +48,8 @@ RUN cat /tmp/bashrc >> /home/patchwork/.bashrc

 COPY tools/docker/entrypoint.sh /usr/local/bin/entrypoint.sh

+RUN apt-get install -y cython cython3; pip3 install python-afl; pip install 
python-afl
+


Do you need afl-fuzz itself as well?

--
Andrew Donnellan  OzLabs, ADL Canberra
andrew.donnel...@au1.ibm.com  IBM Australia Limited

___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

Re: [PATCH 10/10] [RFC] Fuzzing harness

[Stephen Finucane]

On Wed, 2017-06-28 at 17:48 +1000, Daniel Axtens wrote:
>  - install python-afl in Docker (py2 doesn't seem to work)
> 
>  - change parser to return BrokenEmailException. This allows
>    us to catch other sorts of ValueError.
> 
>  - fuzz management command to be used in py-afl-fuzz
> 
> Signed-off-by: Daniel Axtens 

FYI I haven't reviewed this, given that it's an RFC and non-critical, per your
comments. Will come back and revisit post 2.0.

Stephen
___
Patchwork mailing list
Patchwork@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/patchwork

[PATCH 10/10] [RFC] Fuzzing harness

[Daniel Axtens]

 - install python-afl in Docker (py2 doesn't seem to work)

 - change parser to return BrokenEmailException. This allows
   us to catch other sorts of ValueError.

 - fuzz management command to be used in py-afl-fuzz

Signed-off-by: Daniel Axtens 
---
 patchwork/management/commands/fuzz.py | 88 +++
 patchwork/parser.py   | 18 ---
 patchwork/tests/test_parser.py|  5 +-
 tools/docker/Dockerfile   |  2 +
 tools/fuzzer_dict | 52 +
 5 files changed, 156 insertions(+), 9 deletions(-)
 create mode 100644 patchwork/management/commands/fuzz.py
 create mode 100644 tools/fuzzer_dict

diff --git a/patchwork/management/commands/fuzz.py 
b/patchwork/management/commands/fuzz.py
new file mode 100644
index ..c2c08bcfbec2
--- /dev/null
+++ b/patchwork/management/commands/fuzz.py
@@ -0,0 +1,88 @@
+# Patchwork - automated patch tracking system
+# Copyright (C) 2016 Stephen Finucane 
+#
+# This file is part of the Patchwork package.
+#
+# Patchwork is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# Patchwork is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Patchwork; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+
+import email
+import logging
+
+from django.core.management import base
+from django.utils import six
+
+from patchwork.models import Person
+from patchwork.models import Patch
+from patchwork.models import Series
+from patchwork.models import CoverLetter
+from patchwork.models import Comment
+from patchwork.models import SeriesReference
+from patchwork.parser import parse_mail
+from patchwork.parser import BrokenEmailException
+
+import afl
+afl.init()
+
+logger = logging.getLogger(__name__)
+
+
+class Command(base.BaseCommand):
+help = 'Parse an mbox file and store any patch/comment found.'
+
+def add_arguments(self, parser):
+parser.add_argument(
+'infile',
+nargs=1,
+type=str,
+help='input mbox file')
+parser.add_argument(
+'--list-id',
+help='mailing list ID. If not supplied, this will be '
+'extracted from the mail headers.')
+
+def cleanup(self):
+Series.objects.all().delete()
+SeriesReference.objects.all().delete()
+Patch.objects.all().delete()
+Comment.objects.all().delete()
+CoverLetter.objects.all().delete()
+Person.objects.all().delete()
+
+def handle(self, *args, **options):
+infile = options['infile'][0]
+
+logger.info('Parsing mail loaded by filename')
+try:
+if six.PY3:
+with open(infile, 'rb') as file_:
+mail = email.message_from_binary_file(file_)
+else:
+with open(infile) as file_:
+mail = email.message_from_file(file_)
+except AttributeError:
+logger.warning("Broken email ignored")
+return
+
+try:
+parse_mail(mail, options['list_id'])
+self.cleanup()
+except BrokenEmailException:
+logger.warning("Broken email ignored")
+self.cleanup()
+except Exception as E:
+logger.exception('Error when parsing incoming email',
+ extra={'mail': mail.as_string()})
+self.cleanup()
+raise E
diff --git a/patchwork/parser.py b/patchwork/parser.py
index 46e6ca161574..eaeafa6f 100644
--- a/patchwork/parser.py
+++ b/patchwork/parser.py
@@ -54,6 +54,10 @@ SERIES_DELAY_INTERVAL = 10
 logger = logging.getLogger(__name__)
 
 
+class BrokenEmailException(Exception):
+pass
+
+
 def normalise_space(value):
 whitespace_re = re.compile(r'\s+')
 return whitespace_re.sub(' ', value).strip()
@@ -293,7 +297,7 @@ def find_author(mail):
 from_header = clean_header(mail.get('From'))
 
 if not from_header:
-raise ValueError("Invalid 'From' header")
+raise BrokenEmailException("Invalid 'From' header")
 
 name, email = (None, None)
 
@@ -324,7 +328,7 @@ def find_author(mail):
 break
 
 if not email:
-raise ValueError("Invalid 'From' header")
+raise BrokenEmailException("Invalid 'From' header")
 
 email = email.strip()
 if name is not None:
@@ -627,7 +631,7 @@ def clean_subject(subject, drop_prefixes=None):
 subject =