On Tue, Feb 13, 2018 at 12:17 AM, SULLIVAN, BRYAN L (BRYAN L) < bryan.sulli...@research.att.com> wrote:
> Comments etc inline > > > > Thanks, > > Bryan Sullivan | AT&T > > > > *From:* Luke Hinds [mailto:lhi...@redhat.com] > *Sent:* Monday, February 12, 2018 9:04 AM > *To:* SULLIVAN, BRYAN L (BRYAN L) <bryan.sulli...@research.att.com> > *Cc:* opnfv-tech-discuss@lists.opnfv.org; degirmenci, fatih < > fatih.degirme...@ericsson.com>; Raymond Paik <rp...@linuxfoundation.org> > *Subject:* Re: [opnfv-tech-discuss] Anteater status and link issue > > > > > > > > On Tue, Feb 6, 2018 at 2:32 PM, SULLIVAN, BRYAN L (BRYAN L) < > bryan.sulli...@research.att.com> wrote: > > Hi all, > > I’m wondering where the Anteater program is – and want to note a broken > link: build jobs with Anteater violations reference “Please visit: > https://wiki.opnfv.org/x/5oey > <https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.opnfv.org_x_5oey&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=ML-JPRZQOfToJjMwlJLPlcWimAEwMA5DZGNIrk-cgy0&m=rxS4DJ0v-VW97BzYjY1_yRCmM1znHlxObcXiTIP6RBI&s=s4zQHMsxrgVhlTs-Sw4-uGIsKYDMsnIQuvx0TehUoSk&e=> > ”, which is the wiki page https://wiki.opnfv.org/pages/v > iewpage.action?pageId=11700198 > <https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.opnfv.org_pages_viewpage.action-3FpageId-3D11700198&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=ML-JPRZQOfToJjMwlJLPlcWimAEwMA5DZGNIrk-cgy0&m=rxS4DJ0v-VW97BzYjY1_yRCmM1znHlxObcXiTIP6RBI&s=burTDZjfgUSG9lAKW4MjRDZULxleQEsKGknHvhdqzbA&e=>, > which says “Project specific exceptions can be added for file_name, > file_contents and binaries, by using the name of the repository within the > anteater/exceptions/ directory of the releng-anteater > <https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.opnfv.org_gerrit.opnfv.org-3A29418_releng-2Danteater.git&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=ML-JPRZQOfToJjMwlJLPlcWimAEwMA5DZGNIrk-cgy0&m=rxS4DJ0v-VW97BzYjY1_yRCmM1znHlxObcXiTIP6RBI&s=LrWykp0HOa_RUbxOEJDo7sojbPgNgsVsrlV6jmwMVx4&e=> > repository.” – but that link (releng-anteater > <https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.opnfv.org_gerrit.opnfv.org-3A29418_releng-2Danteater.git&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=ML-JPRZQOfToJjMwlJLPlcWimAEwMA5DZGNIrk-cgy0&m=rxS4DJ0v-VW97BzYjY1_yRCmM1znHlxObcXiTIP6RBI&s=LrWykp0HOa_RUbxOEJDo7sojbPgNgsVsrlV6jmwMVx4&e=>) > is broken. > > I want to start adding the exceptions for Models etc as an example for the > LF IT team that is setting up the Acumos project gerrit/CI/CD process, and > in general to help optimize the Anteater overhead for projects. I think we > need to get some analysis of the types of exceptions that are typical, and > establish a process for vetting those exceptions that goes beyond a simple > review by a releng committer. > > Further, we need to bring in other scan tools (e.g. security > vulnerability, virus, or malicious code scans) into the Anteater process. > This is in response to concerns about the security of the governance > process for open source (e.g. upstream, but also direct contribution in > projects) that is used to build production-oriented systems. We need to > demonstrate that OPNFV and other LF projects are addressing these concerns > through their infra toolsets. > > > > Sorry Bryan, I missed a few of these emails thanks (or rather no thanks) > to a bad mail filter rule. > > I am working on the following now which we will see soon: > > Much better documentation: http://anteater.readthedocs.io/en/latest/ > <https://urldefense.proofpoint.com/v2/url?u=http-3A__anteater.readthedocs.io_en_latest_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=ML-JPRZQOfToJjMwlJLPlcWimAEwMA5DZGNIrk-cgy0&m=rxS4DJ0v-VW97BzYjY1_yRCmM1znHlxObcXiTIP6RBI&s=AdeEyIqajKWRGD1zz3MXcKrWoAWYR6mXmQDgVVzp1Zo&e=> > > [bryan] Are you going to start hosting these docs at docs.opnfv.org? > We can do yes, although I guess it make sense to have the main body of the documentation around the tool upstream (once the github re-homing happens), and then have everything OPNFV developers need to know about how anteater is used in OPNFV at docs.opnfv.org - this way there won't be materials in docs.opnfv.org around using Travis CI (which would confuse people). > Virus total integration: > > * Any binaries will be scanned using the virus total API, unless a > sha256 waiver is already present e.g. https://github.com/opnfv/relen > g-anteater/blob/master/exceptions/calipso.yaml#L9 > <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_opnfv_releng-2Danteater_blob_master_exceptions_calipso.yaml-23L9&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=ML-JPRZQOfToJjMwlJLPlcWimAEwMA5DZGNIrk-cgy0&m=rxS4DJ0v-VW97BzYjY1_yRCmM1znHlxObcXiTIP6RBI&s=WNetEYMktH0pxwVzSJXZyDFVnJr6lIDBhM6laGBrbjs&e=> > > * Any IP addresses / domain name / URL will be scanned (again using the > Virus Total API) for known malware and other nastiness. > > [bryan] VirusTotal looks like a useful service. Are there any stats for > its effectiveness at detecting threats, including new threats and delay in > supporting them? > Its pretty much the epicentre of community based threat collaboration . It aggregates 40 virus / malware scanners to asses files, and domains / IP addresses are assessed against 70 URL/domain blacklisting services: https://support.virustotal.com/hc/en-us/articles/115002126889-How-it-works It also fits well into anteater: Currently anteater will generate a sha256sum of any blobs it finds, and will report them, *unless* a sha256 is entered into the exception files. I will extend this, so that if a blob is found an *no* sha256sum exception exists, we send the file hash to Virustotal to see if its registered as nefarious. If it is we will fail the job and alarm the finding. If not, an exception can be entered and it will be ignored from there on in and we won't trouble the VT API again for that particular file - unless someone at a later point changes the file (which would change the checkum) and then a scan is made again - this way we can be sure that an infected file is not checked into a project and we are not aware as it has the same name as before. > I also have a load of new strings to add to dig out and report anything of > a more recent finding (for example a javascript based bitcoin miner). > > [bryan] I would like to see how we can improve the contextual > effectiveness of the pattern matching approach. Any bar (or port in a > storm) may seem to be better than none, and can at least catch newbie > mistakes and anti-patterns, but most of the strings I’ve included in > https://github.com/opnfv/models/blob/master/tools/anteater-exceptions.yaml > relate to IMO innocuous (if admittedly sometimes cheap or anti-patterned) > use of prohibited words. Others, I clearly need to fix. > So I am very open to switching off the more noisy regexs that emit false positives and also open to new approaches. I am sure I can fine tune them much better as well. Likewise open to any feature recommendations etc. > The project is also hopefully going to move into github (once agreed with > LF) to encourage wider contributions and allow it to be more easily > consumed else. > > [bryan] Anything that broadens contribution and consumption makes sense to > me. Are there any other open source projects in this same space that you > are considering leveraging, to avoid re-developing features unnecessarily? > We plan to discuss wider LFN adoption , one example being OpenDayLight where I manage security. I also plan to get more eyes on the tool for smaller projects to utilise. An OpenStack project is also considering the tool, but more for finding depreciated key directives and release tags. > Once the above is in place, docs will be clearer to follow, project will > be more presentable, with more coverage in finding vulns will be wider. > > > > [bryan] We probably need more docs re the process for getting exceptions > approved, and how the community can track its effectiveness in the mission > represented by this toolset, through the types of approved exception > patterns, as they grow (or shrink… it would be good to see the community > improving through reduction in the need to maintain exceptions, and partly > because the tool is getting smarter). > Very much agree, it would be great to see people add to the master exception / ignore list and feedback on where the tool works well / is annoying etc. I also agree on the docs and enlarging upon the process for getting exceptions approved. I plan to have all this done before ONS so we can see it in place for then. Thanks, > > Bryan Sullivan | AT&T > > > > > _______________________________________________ > opnfv-tech-discuss mailing list > opnfv-tech-discuss@lists.opnfv.org > https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss > <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.opnfv.org_mailman_listinfo_opnfv-2Dtech-2Ddiscuss&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=ML-JPRZQOfToJjMwlJLPlcWimAEwMA5DZGNIrk-cgy0&m=rxS4DJ0v-VW97BzYjY1_yRCmM1znHlxObcXiTIP6RBI&s=8NPFgQFDZsv688HirOlM8HW1u0X9QVVgUfsN6B5PP_s&e=> > >
_______________________________________________ opnfv-tech-discuss mailing list opnfv-tech-discuss@lists.opnfv.org https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
