This is an automated email from the ASF dual-hosted git repository.
rkk pushed a commit to branch SDAP-520
in repository https://gitbox.apache.org/repos/asf/sdap-nexus.git
The following commit(s) were added to refs/heads/SDAP-520 by this push:
new 7dcb19c Attempt to fix the many Sphinx warnings on build
7dcb19c is described below
commit 7dcb19c7b9955cc91d0448be79d4457bcfafc314
Author: rileykk <[email protected]>
AuthorDate: Fri Jul 5 10:49:40 2024 -0700
Attempt to fix the many Sphinx warnings on build
---
docs/build.rst | 24 +++++++--------
docs/index.rst | 2 +-
docs/intro.rst | 4 +--
docs/ningester.rst | 89 -----------------------------------------------------
docs/quickstart.rst | 49 +++++++++++++++--------------
5 files changed, 39 insertions(+), 129 deletions(-)
diff --git a/docs/build.rst b/docs/build.rst
index 5992c9a..de7d546 100644
--- a/docs/build.rst
+++ b/docs/build.rst
@@ -1,13 +1,13 @@
.. _build:
-*****************
+***********************************
How to Build the SDAP Docker Images
-*****************
+***********************************
In this guide, we will go over how to build the docker images for the various
components of SDAP.
Prepare
-===========
+=======
First, we must ensure we have Docker installed and running. For this guide, we
used Docker Desktop version 4.12.0. Download Docker Desktop `here.
<https://www.docker.com/products/docker-desktop/>`_
@@ -52,7 +52,7 @@ Create a temporary directory to extract to.
mv
${TMP_DIR}/Apache-SDAP/apache-sdap-nexus-${NEXUS_VERSION}-incubating-src/*
${NEXUS_DIR}
Set Default Docker Platform
----
+---------------------------
To ensure consistency when building/running on different hardware
architectures, we should set this variable to ensure docker uses
``linux/amd64``.
@@ -72,7 +72,7 @@ For both of these, we must first move to the ingester root
directory.
cd ${INGESTER_DIR}
Build the Collection Manager
--------
+----------------------------
From the ingester root directory, run:
@@ -81,7 +81,7 @@ From the ingester root directory, run:
docker build . -f collection_manager/docker/Dockerfile -t
sdap-local/sdap-collection-manager:${INGESTER_VERSION}
Build the Granule Ingester
--------
+--------------------------
From the ingester root directory, run:
@@ -100,7 +100,7 @@ From the ingester root directory, run:
Note: Building does not currently work on Apple Silicon (M1/M2). (`SDAP-488
<https://issues.apache.org/jira/browse/SDAP-488>`_)
Build the Solr & Webapp Components
-======
+==================================
For the remaining three components, we must now move to the nexus root
directory.
@@ -109,7 +109,7 @@ For the remaining three components, we must now move to the
nexus root directory
cd ${NEXUS_DIR}
Build the Solr Image
--------
+--------------------
First we must move to the Solr Docker directory.
@@ -124,7 +124,7 @@ Now to build the image:
docker build . -t sdap-local/sdap-solr-cloud:${NEXUS_VERSION}
Build the Solr Initialization Image
--------
+-----------------------------------
Now to build the image:
@@ -133,7 +133,7 @@ Now to build the image:
docker build . -t sdap-local/sdap-solr-cloud-init:${NEXUS_VERSION} -f
cloud-init/Dockerfile
Build the Webapp Image
----------
+----------------------
For the final image, we must return to the NEXUS root directory.
@@ -158,13 +158,13 @@ Now we can build the webapp with:
Note: Building does not currently work on Apple Silicon (M1/M2). (`SDAP-488
<https://issues.apache.org/jira/browse/SDAP-488>`_)
Verify Successful Build
-====
+=======================
To verify build success, follow the :ref:`quickstart guide<quickstart>`.
Finished!
-=====
+=========
Congratulations! You have successfully built the images required for running
NEXUS.
diff --git a/docs/index.rst b/docs/index.rst
index 32fba5b..160c30f 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -1,5 +1,5 @@
Welcome to the Apache SDAP project documentation!
-================================================
+=================================================
.. toctree::
:maxdepth: 2
diff --git a/docs/intro.rst b/docs/intro.rst
index 508e75c..86e4dbc 100644
--- a/docs/intro.rst
+++ b/docs/intro.rst
@@ -1,8 +1,8 @@
.. _intro:
-*******************
+***********
About NEXUS
-*******************
+***********
NEXUS is a data-intensive analysis solution using a new approach for handling
science data to enable large-scale data analysis.
diff --git a/docs/ningester.rst b/docs/ningester.rst
deleted file mode 100644
index 82634ef..0000000
--- a/docs/ningester.rst
+++ /dev/null
@@ -1,89 +0,0 @@
-.. _ningester:
-
-*****************
-Ningester
-*****************
-
-How to run this image
-======================
-
-The basic command is:
-
-.. code-block:: bash
-
- docker run -it --rm -v <absolute path to config directory on
host>:/config/ -v <absolute path to granule on host>:/data/<granule name>
sdap/ningester <profiles to activate>
-
-Replacing the following:
-
- * ``<absolute path to config directory on host>`` should be the absolute
path on the host to the configuration for the job
- * ``<absolute path to granule on host>`` should be the absolute path on the
host to the granule intended for ingestion
- * ``<granule name>`` should be the filename of the granule
- * ``<profiles to activate>`` is a comma-separated list of profiles to
activate
-
-The `ExampleJob.yml
<https://github.com/apache/incubator-sdap-ningester/blob/bc596c2749a7a2b44a01558b60428f6d008f4f45/docker/example_config/ExampleJob.yml>`_
file shows an example Job configuration that would ingest an AVHRR granule.
-
-Configuration
-==============
-
-Upon running the image, the ningester job will scan the ``/config`` directory
for any files that end with the ``.yml`` extension. Specifically it uses find:
-
-.. code-block:: bash
-
- find /config -name "*.yml" | awk -vORS=, '{ print $1 }'
-
-Therefore, to configure the job, mount your configuration files into
``/config`` using a Docker volume. Alternatively, configuration is loaded via
Spring Boot's `relaxed binding rules
<https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-external-config.html#boot-features-external-config-relaxed-binding>`_.
-So, you can also configure the job through environment variables where each
'level' of the yaml file gets replaced by an '_'.
-
-For example, given a configuration option in yaml that looks like:
-
-.. code-block:: yaml
-
- ningester:
- tile_slicer: sliceFileByTilesDesired
- sliceFileByTilesDesired:
- tilesDesired: 1296
- timeDimension: time
- dimensions:
- - lat
- - lon
-
-These could be replaced with the following corresponding Environment variables:
-
-.. code-block:: bash
-
- NINGESTER_TILE_SLICER=sliceFileByTilesDesired
- NINGESTER_SLICE_FILE_BY_TILES_DESIRED_TILES_DESIRED=1296
- NINGESTER_SLICE_FILE_BY_TILES_DESIRED_TIME_DIMENSION=time
- NINGESTER_SLICE_FILE_BY_TILES_DESIRED_DIMENSIONS[0]=lat
- NINGESTER_SLICE_FILE_BY_TILES_DESIRED_DIMENSIONS[1]=lon
-
-However, because ningester has a lot of configuration options, it is
recommended to use the yaml option.
-
-Data
-=====
-
-Ningester is designed to ingest 1 granule per run. It looks for the granule to
ingest in the ``/data`` directory of the container image.
-Use a Docker volume to mount your data into ``/data``.
-
-The image relies on this command to find the first file in ``/data`` and it
will use that file for ingestion:
-
-.. code-block:: bash
-
- find /data -type f -print -quit
-
-Examples
-==========
-
-A few example commands are shared here.
-
-Docker for Mac
-====================
-
-The `ConnectionSettings-DockerForMac.yml
<https://github.com/apache/incubator-sdap-ningester/blob/bc596c2749a7a2b44a01558b60428f6d008f4f45/docker/example_config/ConnectionSettings-DockerForMac.yml>`_
file shows an example of how to configure the connection settings
-when running this job under Docker for Mac with Solr and Cassandra running on
your host Mac.
-
-Replace ``<path to ningester>`` with the path on your local workstation to the
ningester github project.
-
-.. code-block:: bash
-
- docker run -it --rm -v <path to ningester>/docker/example_config/:/config/
-v <path to
ningester>/src/test/resources/granules/20050101120000-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc:/data/20050101120000-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc.nc
sdap/ningester dockermachost,solr,cassandra
diff --git a/docs/quickstart.rst b/docs/quickstart.rst
index f393493..34553b1 100644
--- a/docs/quickstart.rst
+++ b/docs/quickstart.rst
@@ -1,13 +1,13 @@
.. _quickstart:
-*****************
+*************************
Quickstart Guide - Docker
-*****************
+*************************
This quickstart will take approximately 45 minutes to complete.
Introduction
-=============
+============
NEXUS is a collection of software that enables the analysis of scientific
data. In order to achieve fast analysis, NEXUS takes the approach of breaking
apart, or "tiling", the original data into smaller tiles for storage. Metadata
about each tile is stored in a fast searchable index with a pointer to the
original data array. When an analysis is requested, the necessary tiles are
looked up in the index and then the data for only those tiles is loaded for
processing.
@@ -16,7 +16,7 @@ This quickstart guide will walk you through how to install
and run NEXUS on your
.. _quickstart-prerequisites:
Prerequisites
-==============
+=============
* Docker (tested on v20.10.17)
* Internet Connection
@@ -25,14 +25,14 @@ Prerequisites
* 8.5 GB of disk space
Prepare
-========
+=======
Start downloading the Docker images and set up the Docker bridge network.
.. _quickstart-step1:
Set Default Docker Platform
----
+---------------------------
To ensure consistency when building/running on different hardware
architectures, we should set this variable to ensure docker uses
``linux/amd64``.
@@ -41,7 +41,7 @@ To ensure consistency when building/running on different
hardware architectures,
export DOCKER_DEFAULT_PLATFORM=linux/amd64
Set Tag Variables
--------------------
+-----------------
Pull the necessary Docker images from the `Apache SDAP repository
<https://hub.docker.com/search?q=apache%2Fsdap>`_ on Docker Hub. Please check
the repository for the latest version tag.
@@ -74,7 +74,7 @@ For Local Builds
Continue to the section: "Create a new Docker Bridge Network"
For Release Builds: Pull Docker Images
--------------------
+--------------------------------------
.. code-block:: bash
@@ -94,7 +94,7 @@ For Release Builds: Pull Docker Images
.. _quickstart-step2:
Create a new Docker Bridge Network
-------------------------------------
+----------------------------------
This quickstart consists of launching several Docker containers that need to
communicate with one another. To facilitate this communication, we want to be
able to reference containers via hostname instead of IP address. The default
bridge network used by Docker only supports this by using the ``--link`` option
which is now considered to be `deprecated
<https://docs.docker.com/network/links/>`_.
@@ -109,7 +109,7 @@ The network we will be using for this quickstart will be
called ``sdap-net``. Cr
.. _quickstart-step3:
Start Core Components
-======================
+=====================
NEXUS relies on Apache Solr and Apache Cassandra to store tile metadata and
science data, so let's start those first.
@@ -131,7 +131,7 @@ We then need to ensure the ``/solr`` znode is present.
.. _quickstart-step4:
Start Solr
------------
+----------
SDAP is tested with Solr version 8.11.1.
@@ -156,7 +156,7 @@ When the init script finishes, kill the container by typing
``Ctrl + C``
.. _quickstart-step5:
Starting Cassandra
--------------------
+------------------
SDAP is tested with Cassandra version 3.11.6.
@@ -196,7 +196,7 @@ With Solr and Cassandra started and initialized, we can now
start the collection
.. _quickstart-step6:
Start the Ingester
-===================
+==================
In this section, we will start the components for the ingester. These
components are:
@@ -207,7 +207,7 @@ In this section, we will start the components for the
ingester. These components
We will also be downloading a number of NetCDF files containing science data
for use in this demo.
Create Data Directory
-------------------------
+---------------------
Let's start by creating the directory to hold the science data to ingest.
@@ -221,7 +221,7 @@ Choose a location that is mountable by Docker (typically
needs to be under the u
.. _quickstart-step7:
Start RabbitMQ
-----------------
+--------------
The collection manager and granule ingester(s) use RabbitMQ to communicate, so
we need to start that up first.
@@ -257,7 +257,7 @@ The granule ingester(s) read new granules from the message
queue and process the
.. _quickstart-optional-step:
[OPTIONAL] Run Message Queue Monitor
--------------------------------------
+------------------------------------
The granule ingestion process can take some time. To monitor its progress, we
wrote a simple python script to monitor the message queue. It will wait until
some granules show up and then will exit once they have all been ingested.
@@ -278,7 +278,7 @@ And then run it in a separate shell
.. _quickstart-step9:
Download Sample Data
----------------------
+--------------------
The data we will be downloading is part of the `AVHRR OI dataset
<https://podaac.jpl.nasa.gov/dataset/AVHRR_OI-NCEI-L4-GLOB-v2.0>`_ which
measures sea surface temperature. We will download 1 month of data and ingest
it into a local Solr and Cassandra instance.
@@ -303,7 +303,7 @@ You should now have 30 files downloaded to your data
directory, one for each day
.. _quickstart-step10:
Create Collection Configuration
---------------------------------
+-------------------------------
The collection configuration is a ``.yml`` file that tells the collection
manager what datasets it is managing, where the granules are stored, and how
they are to be tiled.
@@ -338,7 +338,7 @@ The collection configuration is a ``.yml`` file that tells
the collection manage
.. _quickstart-step11:
Start the Collection Manager
------------------------------
+----------------------------
Now we can start the collection manager.
@@ -370,7 +370,7 @@ When it starts, it will publish messages for the downloaded
granules to RabbitMQ
.. _quickstart-step13:
Start the Webapp
-=================
+================
Now that the data is being (has been) ingested, we need to start the webapp
that provides the HTTP interface to the analysis capabilities. This is
currently a python webapp running Tornado and is contained in the nexus-webapp
Docker image. To start the webapp and expose port 8083 use the following
command:
@@ -422,7 +422,7 @@ Click on the ``Time Series Example`` notebook to start it.
This will open the no
.. _quickstart-finished:
Finished!
-================
+=========
Congratulations you have completed the quickstart! In this example you:
@@ -432,7 +432,7 @@ Congratulations you have completed the quickstart! In this
example you:
#. Ran a time series analysis on 1 month of AVHRR OI data and plotted the
result
Cleanup
-========
+=======
To shut down the Solr container cleanly, run the following command:
@@ -449,7 +449,7 @@ The remaining containers can safely be stopped using Docker
Desktop or by runnin
.. _issues:
Known Issues
-=============
+============
This section contains a list of issues that may be encountered while running
this guide, their causes and solutions.
@@ -470,11 +470,10 @@ There are two solutions to this issue:
* Try running only one ingester container.
Collection Manager Messages Not Publishing
--------------------------------------------
+------------------------------------------
RabbitMQ may not receive the messages published by the Collection Manager.
When this happens, new granules added to monitored collections will not be
processed by the ingester(s).
The cause of this issue seems to be due to the RMQ container having limited
resources, which causes message publication to block indefinitely.
To solve this, first figure out which resource is causing issues by navigating
to http://localhost:15672/#/ and sign in with username ``user`` and password
``bitnami``. View the 'Nodes' section. Insufficient resources will be shown in
red. Allocate more of those resources in Docker and restart the Docker daemon.
-
