Re: [gentoo-portage-dev] [PATCH] repoman: add HOMEPAGE.missingurischeme check

[Zac Medico]

On 01/09/2017 02:07 AM, Wim Muskee wrote:
> thx Zac, I would never have figured that out myself
> 
> the updated patch:
> 
> From c703875bebd82271921d7e94b7cd61936b87d32b Mon Sep 17 00:00:00 2001
> From: Wim Muskee >
> Date: Sat, 7 Jan 2017 12:02:41 +0100
> Subject: [PATCH] repoman: add HOMEPAGE.missingurischeme check

Thanks, merged:

https://gitweb.gentoo.org/proj/portage.git/commit/?id=55dedaa865334543e51693838700dcf5e72754d2
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [PATCH] repoman: add HOMEPAGE.missingurischeme check

[Wim Muskee]

thx Zac, I would never have figured that out myself

the updated patch:

>From c703875bebd82271921d7e94b7cd61936b87d32b Mon Sep 17 00:00:00 2001
From: Wim Muskee 
Date: Sat, 7 Jan 2017 12:02:41 +0100
Subject: [PATCH] repoman: add HOMEPAGE.missingurischeme check

---
 .../pym/repoman/modules/scan/metadata/ebuild_metadata.py| 13 +++--
 repoman/pym/repoman/qa_data.py  |  2 ++
 2 files changed, 13 insertions(+), 2 deletions(-)

diff --git a/repoman/pym/repoman/modules/scan/metadata/ebuild_metadata.py
b/repoman/pym/repoman/modules/scan/metadata/ebuild_metadata.py
index e991a30..0a29d02 100644
--- a/repoman/pym/repoman/modules/scan/metadata/ebuild_metadata.py
+++ b/repoman/pym/repoman/modules/scan/metadata/ebuild_metadata.py
@@ -10,9 +10,10 @@

 from repoman.modules.scan.scanbase import ScanBase
 from repoman.qa_data import missingvars
+from repoman,_portage import portage

 NON_ASCII_RE = re.compile(r'[^\x00-\x7f]')
-
+URISCHEME_RE = re.compile(r'^[a-z][0-9a-z\-\.\+]+://')

 class EbuildMetadata(ScanBase):

@@ -62,10 +63,18 @@ def virtual(self, **kwargs):
self.qatracker.add_error(myqakey, 
ebuild.relative_path)
return False

+   def homepage_urischeme(self, **kwargs):
+   ebuild = kwargs.get('ebuild').get()
+   if kwargs.get('catdir') != "virtual":
+   for homepage in
portage.dep.use_reduce(ebuild.metadata["HOMEPAGE"],
matchall=True,flat=True):
+   if URISCHEME_RE.match(homepage) is None:
+   
self.qatracker.add_error("HOMEPAGE.missingurischeme",
ebuild.relative_path )
+   return False
+
@property
def runInPkgs(self):
return (False, [])

@property
def runInEbuilds(self):
-   return (True, [self.invalidchar, self.missing, self.old_virtual,
self.virtual])
+   return (True, [self.invalidchar, self.missing, self.old_virtual,
self.virtual, self.homepage_urischeme])
diff --git a/repoman/pym/repoman/qa_data.py b/repoman/pym/repoman/qa_data.py
index c3f4207..29a95ab 100644
--- a/repoman/pym/repoman/qa_data.py
+++ b/repoman/pym/repoman/qa_data.py
@@ -115,6 +115,8 @@
"Ebuilds that have a missing or empty HOMEPAGE variable"),
"HOMEPAGE.virtual": (
"Virtuals that have a non-empty HOMEPAGE variable"),
+   "HOMEPAGE.missingurischeme": (
+   "HOMEPAGE is missing an URI scheme"),
"PDEPEND.suspect": (
"PDEPEND contains a package that usually only belongs in 
DEPEND."),
"LICENSE.syntax": (



On Mon, Jan 9, 2017 at 1:04 AM, Zac Medico  wrote:

> On 01/07/2017 05:32 AM, Michael Orlitzky wrote:
> > On 01/07/2017 06:08 AM, Wim Muskee wrote:
> >>
> >> URISCHEME_RE = re.compile(r'^[a-z\-]+://')
> >>
> >> ...
> >>
> >> URISCHEME_RE.match(ebuild.metadata.get("HOMEPAGE")) is None:
> >>
> >
> > The PMS allows some weird stuff in HOMEPAGE:
> >
> >   https://dev.gentoo.org/~ulm/pms/head/pms.html#x1-760008
> >
> > Specifically,
> >
> >   In addition, SRC_URI, HOMEPAGE, RESTRICT, PROPERTIES, LICENSE and
> >   REQUIRED_USE use dependency-style specifications to specify their
> >   values.
> >
> > That means that something like,
> >
> >   HOMEPAGE="branding? ( https://www.mozilla.org/ )
> >!branding? ( https://www.gentoo.org/ )"
> >
> > would be valid. It's a little crazy, but there it is.
> >
> > If you can figure out a way to parse a dependency spec (this has to
> > exist somewhere in repoman/portage), then you can run your check against
> > the URLs at the leaf nodes. At that point, it should be relatively easy
> > to update the regex to match the RFC =)
> >
> >   https://tools.ietf.org/html/rfc3986#section-3.1
>
> This will return a flat list:
>
> portage.dep.use_reduce(ebuild.metadata["HOMEPAGE"], matchall=True,
> flat=True)
> --
> Thanks,
> Zac
>
>

Re: [gentoo-portage-dev] [PATCH] repoman: add HOMEPAGE.missingurischeme check

[Zac Medico]

On 01/07/2017 05:32 AM, Michael Orlitzky wrote:
> On 01/07/2017 06:08 AM, Wim Muskee wrote:
>>
>> URISCHEME_RE = re.compile(r'^[a-z\-]+://')
>>
>> ...
>>
>> URISCHEME_RE.match(ebuild.metadata.get("HOMEPAGE")) is None:
>>
> 
> The PMS allows some weird stuff in HOMEPAGE:
> 
>   https://dev.gentoo.org/~ulm/pms/head/pms.html#x1-760008
> 
> Specifically,
> 
>   In addition, SRC_URI, HOMEPAGE, RESTRICT, PROPERTIES, LICENSE and
>   REQUIRED_USE use dependency-style specifications to specify their
>   values.
> 
> That means that something like,
> 
>   HOMEPAGE="branding? ( https://www.mozilla.org/ )
>!branding? ( https://www.gentoo.org/ )"
> 
> would be valid. It's a little crazy, but there it is.
> 
> If you can figure out a way to parse a dependency spec (this has to
> exist somewhere in repoman/portage), then you can run your check against
> the URLs at the leaf nodes. At that point, it should be relatively easy
> to update the regex to match the RFC =)
> 
>   https://tools.ietf.org/html/rfc3986#section-3.1

This will return a flat list:

portage.dep.use_reduce(ebuild.metadata["HOMEPAGE"], matchall=True,
flat=True)
-- 
Thanks,
Zac

[gentoo-portage-dev] [PATCH] repoman: add HOMEPAGE.missingurischeme check

[Wim Muskee]

am I doing this right?

X-Gentoo-Bug: 533554
X-Gentoo-Bug-URL: https://bugs.gentoo.org/533554
---
>From 6765f3ac1e77ce743d44f47c5dda79a573b8629c Mon Sep 17 00:00:00 2001
From: Wim Muskee 
Date: Sat, 7 Jan 2017 12:02:41 +0100
Subject: [PATCH] repoman: add HOMEPAGE.missingurischeme check

---
 repoman/pym/repoman/modules/scan/metadata/ebuild_metadata.py | 11 +--
 repoman/pym/repoman/qa_data.py   |  2 ++
 2 files changed, 11 insertions(+), 2 deletions(-)

diff --git a/repoman/pym/repoman/modules/scan/metadata/ebuild_metadata.py
b/repoman/pym/repoman/modules/scan/metadata/ebuild_metadata.py
index e991a30..3e2cea8 100644
--- a/repoman/pym/repoman/modules/scan/metadata/ebuild_metadata.py
+++ b/repoman/pym/repoman/modules/scan/metadata/ebuild_metadata.py
@@ -12,7 +12,7 @@
 from repoman.qa_data import missingvars

 NON_ASCII_RE = re.compile(r'[^\x00-\x7f]')
-
+URISCHEME_RE = re.compile(r'^[a-z\-]+://')

 class EbuildMetadata(ScanBase):

@@ -62,10 +62,17 @@ def virtual(self, **kwargs):
self.qatracker.add_error(myqakey, 
ebuild.relative_path)
return False

+   def homepage_urischeme(self, **kwargs):
+   ebuild = kwargs.get('ebuild').get()
+   if kwargs.get('catdir') != "virtual" and \
+   URISCHEME_RE.match(ebuild.metadata.get("HOMEPAGE")) is 
None:
+   
self.qatracker.add_error("HOMEPAGE.missingurischeme",
ebuild.relative_path )
+   return False
+
@property
def runInPkgs(self):
return (False, [])

@property
def runInEbuilds(self):
-   return (True, [self.invalidchar, self.missing, self.old_virtual,
self.virtual])
+   return (True, [self.invalidchar, self.missing, self.old_virtual,
self.virtual, self.homepage_urischeme])
diff --git a/repoman/pym/repoman/qa_data.py b/repoman/pym/repoman/qa_data.py
index c3f4207..29a95ab 100644
--- a/repoman/pym/repoman/qa_data.py
+++ b/repoman/pym/repoman/qa_data.py
@@ -115,6 +115,8 @@
"Ebuilds that have a missing or empty HOMEPAGE variable"),
"HOMEPAGE.virtual": (
"Virtuals that have a non-empty HOMEPAGE variable"),
+   "HOMEPAGE.missingurischeme": (
+   "HOMEPAGE is missing an URI scheme"),
"PDEPEND.suspect": (
"PDEPEND contains a package that usually only belongs in 
DEPEND."),
"LICENSE.syntax": (