commit 128531b9bc1557d5d36580651bac9caaeee50a95
Author: Arturo Filastò <art...@filasto.net>
Date:   Tue Nov 22 12:37:23 2016 +0000

    First pass at freshening up the architecture document (#580)
---
 docs/source/architecture.rst | 351 ++++++++++++++++++++++++-------------------
 docs/source/conf.py          |   2 +-
 2 files changed, 201 insertions(+), 152 deletions(-)

diff --git a/docs/source/architecture.rst b/docs/source/architecture.rst
index 5e7d253..921fcbb 100644
--- a/docs/source/architecture.rst
+++ b/docs/source/architecture.rst
@@ -1,32 +1,119 @@
 Architecture
 ============
 
-The goal of this document is provide an overview of how ooni works, what are
-it's pieces and how they interact with one another.
+Last Updated: 2016-08-01
 
-Keep in mind that this is the *big picture* and not all of the features and
-compontent detailed here are implemented.
-To get an idea of what is implemented and with what sort of quality see the
-`Implementation status`_ section of this page.
+The purpose of this goal is to illustrate the design goals of the various
+components part of the OONI ecosystem, how they work and what is the
+relationship between each other.
 
-The two main components of ooni are `oonib`_ and `ooniprobe`_.
+The following diagram gives you an idea of how the various OONI components
+are related to each other.
 
-.. image:: _static/images/ooniprobe-architecture.png
-    :width: 700px
+.. graphviz::
 
-ooniprobe
----------
+    digraph Architecture {
 
-ooniprobe the client side component of ooni that is responsible for performing
+        subgraph cluster_0 {
+            style=filled;
+            color=lightgrey;
+            node [style=filled,color=white];
+            "ooni-probe";
+            "measurement-kit";
+            label="clients";
+        }
+
+        "ooni-probe" -> "ooni-backend";
+        "measurement-kit" -> "ooni-backend";
+        "ooni-wui" -> "ooni-probe";
+        "lepidopter" -> "ooni-probe";
+        "ooni-backend" -> "ooni-pipeline";
+        "ooni-pipeline" -> "ooni-explorer";
+    }
+
+
+The main software components are the following:
+
+* ooni-probe_: what users interested in contributing measurements will run.
+  It also includes a web based user interface for running measurements and
+  inspecting the results.
+  code repository: `<https://github.com/TheTorProject/ooni-probe>`_
+
+* measurement-kit_: a portable C++ library that implements some ooniprobe
+  tests and is currently being used to port ooniprobe to mobile platforms
+  (Android and iOS).
+  In the future the measurement engine of ooniprobe will be replaced with
+  measurement-kit.
+  code repository: `<https://github.com/measurement-kit/measurement-kit>`_
+
+* ooni-backend_: the software component that measurement clients communicate
+  with to learn the address of where they should submit results, submit results
+  (collector) and run certain tests against (see: `Test Helpers`_).
+  code repository: `<https://github.com/TheTorProject/ooni-backend>`_
+
+* ooni-pipeline_: responsible for taking raw measurement data (from
+  collectors) normalising it, extracting insight from it and preparing it for
+  being presented inside of the `ooni-explorer`_ interface.
+  code repository: `<https://github.com/TheTorProject/ooni-pipeline>`_
+
+* ooni-explorer_: a web front-end to the measurements collected by the OONI
+  platform. It features a world map view showcasing the countries where we have
+  identified network anomalies.
+  code repository: `<https://github.com/TheTorProject/ooni-explorer>`_
+
+* ooni-wui_: web user interface assets and the implementation of the
+  ooni-probe web interface. Components in here are meant to be re-used across
+  the various software components (ooni-probe, ooni-explorer, net-probe, etc.),
+  though work on this from is not yet complete.
+  code repository: `<https://github.com/TheTorProject/ooni-wui>`_
+
+* lepidopter_: a raspberry pi image for running ooniprobe.
+  code repository: `<https://github.com/TheTorProject/lepidopter>`_
+
+* ooni-web_: the canonical ooni.torproject.org website.
+  code repository: `<https://github.com/TheTorProject/ooni-web>`_
+
+
+
+.. _ooni-probe:
+---------------
+
+ooni-probe the client side component of OONI that is responsible for performing
 measurements on the to be tested network.
 
-The main design goals for ooniprobe are:
+Originally thought of as a tool to be used by users to investigate network
+anomalies on their own and quickly implement new tests to check for new
+censorship conditions, the focus is now shifting more towards something
+meant to be used in an unattended manner.
+
+As such it's evolving into being a system daemon that is always running on
+a users machine and automatically performs the network measurements the user
+has instructed it to perform.
+
+Design goals
+.............
+
+The current design goals are:
+
+**Unattended measurement collection**
+
+It should be possible for a user of the system to install it and forget about
+it. This means that it shouldn't be necessary to constantly interact with the 
tool
+itself.
 
-Test specification decoupling
-.............................
+Previously some of the design considerations for ooni-probe used to be:
 
-By this I mean that the definition of the test should be as loosely coupled to
-the code that is used for running the test.
+**Test specification decoupling**
+
+This design goal is still largely valid, though as ooni-probe grows as mainly
+an enduser tool it's importance will be decreasing.
+
+Moreover the long-term plan for this is given the fact that tests are going to
+be run based on measurement-kit_ is to have the testing framework logic be 
+implemented in the measurement-kit_ scripting language.
+
+The outline of this design goal nonetheless is that the definition of the test
+should be as loosely coupled to the code that is used for running the test.
 
 This is achieved via what are called **Test Templates**. Test Templates a high
 level interface to the test developer specific to the protocol they are writing
@@ -46,8 +133,7 @@ received, but a developer may with to include inside of 
their report the
 checksum of the of the content as is show in the example in `Writing Tests
 <writing_tests.html>`_.
 
-Support for high concurrency
-............................
+**Support for high concurrency**
 
 By this I mean that we want to be able to scan through big lists as fast as
 possible.
@@ -68,68 +154,72 @@ For this purpose we have chosen to use the `Twisted 
networking framework
 If you have an argument for which you believe Twisted is not a good idea, I
 would love to know :).
 
-Notes:
-.. XXX
+Running lot's of tests concurrently can reduce their accuracy. The ideal
+strategy for dealing with this would involve adjusting the concurrency 
+based on failure rate.
+Currently this is not implemented inside of ooniprobe and instead we use
+a configurable concurrency value that is set to default as 3.
 
-Running lot's of tests concurrently can reduce their accuracy.  The strategy
-for dealing with this involves doing proper error handling and adjusting the
-concurrency window over time if the amount of error rates increases.
+Implementation details
+......................
 
-Currently the level of concurrency for tests is implemented inside of
-:class:`ooni.inputunit`_, but we do not expose to the user a way of setting
-this. Such feature will be something that will be controllable via the
-ooniprobe API.
+Below is a high level diagram of how the various modules of ooniprobe
+are interrelated to each other.
 
-Why Tor Hidden Services?
-........................
+.. graphviz::
 
-We chose to use Tor Hidden Services as the means of exposing a backend
-reporting system for the following reasons:
+    digraph ooniprobe_impl {
 
-Easy addressing
-_______________
+        "agent" -> "director";
+        "scheduler" -> "director";
 
-Using Tor Hidden Service allows us to have a globally unique identifier to be
-passed to the ooni-probe clients. This identifier does not need to change even
-if we decide to migrate the collector backend to a different machine (all we
-have to do is copy the private key to the new box).
+        "director" -> "deck";
 
-It also allows people to run a collector backend if they do not have a public
-IP address (if they are behing NAT for example).
+        "deck" -> "nettest";
+        "deck" -> "backend_client";
+        "deck" -> "nettests";
+    }
 
-Security
-________
+ooni-probe is written in python using the `Twisted networking framework
+<http://twistedmatrix.org>`_.
 
-Tor Hidden Services give us for free and with little thought end to end
-encryption and authentication. Once the address for the collector has been
-transmitted to the probe you do not need to do any extra authenticatication, 
because
-the address is self authenticating.
+The two main concepts in ooniprobe are a decks and nettests. A nettest is a
+particular network test that is designed to identify one class of anomalies.
 
-Possible drawbacks
-__________________
+A deck is a collection of one or more nettests and some associated inputs (such
+as a list of URLs).
 
-Supporting Tor Hidden Services as the only system for reporting means a
-ooni-probe user is required to have Tor working to be able to submit reports to
-a collector. In some cases this is not possible, because the user is in a
-country where Tor is censored and they do not have any Tor bridges available.
+The director is responsible for starting the measurement and reporting task
+managers, starting tor, looking up the IP address of the probe and in general
+controlling the lifecycle of the application.
 
-Latency is also a big issue in Tor Hidden Services and this can make the
-reporting process very long especially if the users network is not very good.
+The schedulers are periodic tasks that need to be executed (think cron). Their
+state is kept track of on disk (in particular the last time a successful
+execution was performed).
 
-For these reasons we plan to support in the future also non Tor HS based
-reporting to oonib. 
-Currently this can easily be achieved by simply using tor2web.org.
+The agent is responsible for starting director, the schedulers and exposing the
+web user interface.
 
-Standardization
-...............
+.. _measurement-kit:
+--------------------
 
-.. TODO
+Measurement-kit is a C++ library that implements network measurement primitives
+and some of the ooniprobe tests.
+
+It has been developed with the goal of being able to target mobile platforms
+(Android and iOS), but is growing with the intent of eventually replacing the
+measurement engine of ooniprobe entirely with native code.
+
+There is work in progress to support calling it from python (see:
+`<https://github.com/measurement-kit/measurement-kit/pull/697>`_) and there
+are plans to implement a scripting interface around it to aid the development 
of
+tests (see: 
`<https://github.com/measurement-kit/measurement-kit/issues/702>`_).
 
-oonib
------
+.. _ooni-backend:
+-----------------
 
 This is the backend component of OONI. It is responsible for exposing `test
-helpers`_ and the `report collector`_.
+helpers`_ , the `measurement collector`_ and the `bouncer service`_
 
 Test Helpers
 ............
@@ -139,120 +229,79 @@ ooniprobes when running tests.
 
 If you would like to see a test helper implemented inside of oonib, thats
 great!
-All you have to do is `open a ticket on trac
-<https://trac.torproject.org/projects/tor/newticket?component=Ooni&keywords=oonib_testhelpers%20ooni_wishlist&summary=Add%20support%20for%20PROTOCOL_NAME%20test%20helper>`_.
+All you have to do is `open a ticket on github
+<https://github.com/TheTorProject/ooni-backend/issues/new?title=[new%20test-helper%20request]%20YOUR_TESTHELPER_NAME>`_.
 
 To get an idea of the current implementation status of test helpers see the
 `oonib/testhelpers/
-<https://gitweb.torproject.org/ooni-probe.git/tree/HEAD:/oonib/testhelpers>`_
+<https://github.com/TheTorProject/ooni-backend/tree/master/oonib/testhelpers>`_
 directory of the ooniprobe git repository.
 
 .. TODO
    write up the list of currently implemented test helpers and how to use them.
 
-Report collector
+Measurement collector
 ................
 
-.. autoclass:: oonib.report.file_collector.NewReportHandlerFile
-    :noindex:
+This is the service that is used for submitting measurement results to.
 
+The specification for the API of the measurement collector can be found here:
+`<https://github.com/TheTorProject/ooni-spec/blob/master/oonib.md#20-collector>`_
 
-An ooniprobe run
-----------------
-
-Here we describe how an ooniprobe run should look like:
-
-  1. If configured to do so ooniprobe will start a connection to the Tor
-       network for the purpose of having a known good test channel and for
-       having a way of reporting to the backend collector
-
-  2. It will obtain it's IP Address from Tor via the getinfo addr Tor Ctrl port
-       request.
-
-  3. If a collect is specified it will connect to the reporting system and get
-       a report id that allows them to submit reports to the collector.
-
-  4. If inputs are specified it will slice them up into chunks of request to be
-       performed in parallel.
-
-  5. Once every chunk of inputs (called an InputUnit) will have completed the
-       report file and/or the collector will be updated.
-
-
-OONIprobe Control Interface
----------------------------
-.. XXX update this section once interface is implemented.
-The ooniprobe client provides a rich and simple JSON-based interface for 
-control over HTTP. While the implementation of this interface is currently
-a work in progress, the specification may be found `here 
<control_interface.rst>`_.
-
-
-Implementation status
----------------------
-
-ooniprobe
-.........
-
-**Reporting**
-
-  * To flat YAML file: *alpha*
-
-  * To remote httpo backend: *alpha*
-
-**Test templates**
-
-  * HTTP test template: *alpha*
-
-  * Scapy test template: *alpha*
-
-  * DNS test template: *alpha*
-
-  * TCP test template: *prototype*
-
-**Tests**
-
-To see the list of implemented tests see:
-https://ooni.torproject.org/docs/#core-ooniprobe-tests
-
-**ooniprobe API**
-
-  * Specification: *draft*
-
-  * HTTP API: *not implemented*
-
-**ooniprobe HTML5/JS user interface**
+Bouncer service
+................
 
-  Not implemented.
+This is the service that is responsible for informing clients of where they
+should be submitting their results to and what are the addresses of the
+test-helpers they require to perform their measurements.
 
-**ooniprobe build system**
+The specification for the API of the bouncer can be found here:
+`<https://github.com/TheTorProject/ooni-spec/blob/master/oonib.md#40-bouncer>`_
 
-  Not implemented.
+.. _ooni-pipeline:
+------------------
 
-**ooniprobe command line interface**
+When measurements are submitted to a measurement collector they are then
+processed by the data pipeline.
 
-  Implemented in alpha quality, though needs to be ported to use the HTTP based
-  API.
+The measurements are first normalised (to take into account the different data
+formats that ooniprobe has supported over time), then sanitised (to redact 
from them
+sensitive information such a private bridge IP address) and then put inside of 
a
+database to be served via the ooni-explorer_.
 
-oonib
-.....
+It is currently written in python using the `luigi workflow manager
+<https://luigi.readthedocs.org>`_, but that may change in the near future.
+For future plans see: 
`<https://github.com/TheTorProject/ooni-pipeline/issues/32>`_
 
-**Collector**
+.. _ooni-explorer:
+------------------
 
-  * collection of YAML reports to flat file: *alpha*
+This is the web interface that is used by end users to inspect measurements
+collected by ooniprobe.
 
-  * collection of pcap reports: *not implemented*
+It is written as a node.js web app (based on the strongloop framework), with
+angular.js and d3.js.
 
-  * association of reports with test helpers: *not implemented*
+.. _ooni-wui:
+-------------
 
-**Test helpers**
+Web user interface assets and the implementation of the ooni-probe web
+interface. Components in here are meant to be re-used across the various
+software components (ooni-probe, ooni-explorer, net-probe, etc.), though work
+on this from is not yet complete.
 
-  * HTTP Return JSON Helper: *alpha*
+.. _lepidopter:
+---------------
 
-  * DNS Test helper: *prototype*
+A raspberry pi image for running ooniprobe.
 
-  * Test Helper - collector mapping: *Not implemented*
+Amongst other things it takes care of automatically updating ooniprobe to the
+latest version and packaging all the dependencies required to run ooniprobe.
 
-  * TCP Test helper: *prototype*
+.. _ooni-web:
+-------------
 
-  * Daphn3 Test helper: *prototype*
+The canonical ooni.torproject.org website.
 
+It is implemented using `hugo <https://gohugo.io>`_ a golang based static
+website generator.
diff --git a/docs/source/conf.py b/docs/source/conf.py
index e3c6544..2eba8e4 100644
--- a/docs/source/conf.py
+++ b/docs/source/conf.py
@@ -30,7 +30,7 @@ from ooni import __version__ as ooniprobe_version
 # Add any Sphinx extension module names here, as strings. They can be 
extensions
 # coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
 extensions = ['sphinx.ext.todo', 'sphinx.ext.coverage', 'sphinx.ext.pngmath',
-'sphinx.ext.viewcode', 'sphinx.ext.autodoc']
+'sphinx.ext.viewcode', 'sphinx.ext.autodoc', 'sphinx.ext.graphviz']
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']



_______________________________________________
tor-commits mailing list
tor-commits@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-commits

Reply via email to