This is an automated email from the ASF dual-hosted git repository.
rkk pushed a commit to branch develop
in repository https://gitbox.apache.org/repos/asf/sdap-nexus.git
The following commit(s) were added to refs/heads/develop by this push:
new da370b6 SDAP-520 - Guide for evaluating SDAP release candidates (#318)
da370b6 is described below
commit da370b6964bea66b78611d799aeda2d77bbbdfc2
Author: Riley Kuttruff <[email protected]>
AuthorDate: Mon Sep 23 13:06:26 2024 -0700
SDAP-520 - Guide for evaluating SDAP release candidates (#318)
* SDAP-520 Added RC eval guide to RTD
* remove incubator
* add to toctree
* Attempt to fix the many Sphinx warnings on build
* Changelog
---------
Co-authored-by: rileykk <[email protected]>
---
CHANGELOG.md | 1 +
docs/Makefile | 2 +-
docs/build.rst | 24 +++++-----
docs/conf.py | 14 +++---
docs/index.rst | 3 +-
docs/intro.rst | 4 +-
docs/make.bat | 2 +-
docs/ningester.rst | 89 -------------------------------------
docs/quickstart.rst | 49 ++++++++++-----------
docs/release.rst | 123 ++++++++++++++++++++++++++++++++++++++++++++++++++++
10 files changed, 173 insertions(+), 138 deletions(-)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 616ddad..3618908 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,6 +9,7 @@ and this project adheres to [Semantic
Versioning](https://semver.org/spec/v2.0.0
- SDAP-469: Support for three dimensional data. 2D data defaults to a layer at
0m elevation.
- SDAP-492: Added some demo algorithms for working with and visualizing
tomography data. Currently designed for data from airborne SAR campaigns, but
can be readily generalized.
- SDAP-497: Added tool to ease building of releases. Can build from ASF
distributions, git repos, and local
+- SDAP-520: (Documentation) Added guide to docs for evaluating official
release candidates.
### Changed
### Deprecated
### Removed
diff --git a/docs/Makefile b/docs/Makefile
index 59dd51b..a41277d 100644
--- a/docs/Makefile
+++ b/docs/Makefile
@@ -19,7 +19,7 @@
# You can set these variables from the command line.
SPHINXOPTS =
SPHINXBUILD = sphinx-build
-SPHINXPROJ = incubator-sdap-nexus
+SPHINXPROJ = sdap-nexus
SOURCEDIR = .
BUILDDIR = _build
diff --git a/docs/build.rst b/docs/build.rst
index 5992c9a..de7d546 100644
--- a/docs/build.rst
+++ b/docs/build.rst
@@ -1,13 +1,13 @@
.. _build:
-*****************
+***********************************
How to Build the SDAP Docker Images
-*****************
+***********************************
In this guide, we will go over how to build the docker images for the various
components of SDAP.
Prepare
-===========
+=======
First, we must ensure we have Docker installed and running. For this guide, we
used Docker Desktop version 4.12.0. Download Docker Desktop `here.
<https://www.docker.com/products/docker-desktop/>`_
@@ -52,7 +52,7 @@ Create a temporary directory to extract to.
mv
${TMP_DIR}/Apache-SDAP/apache-sdap-nexus-${NEXUS_VERSION}-incubating-src/*
${NEXUS_DIR}
Set Default Docker Platform
----
+---------------------------
To ensure consistency when building/running on different hardware
architectures, we should set this variable to ensure docker uses
``linux/amd64``.
@@ -72,7 +72,7 @@ For both of these, we must first move to the ingester root
directory.
cd ${INGESTER_DIR}
Build the Collection Manager
--------
+----------------------------
From the ingester root directory, run:
@@ -81,7 +81,7 @@ From the ingester root directory, run:
docker build . -f collection_manager/docker/Dockerfile -t
sdap-local/sdap-collection-manager:${INGESTER_VERSION}
Build the Granule Ingester
--------
+--------------------------
From the ingester root directory, run:
@@ -100,7 +100,7 @@ From the ingester root directory, run:
Note: Building does not currently work on Apple Silicon (M1/M2). (`SDAP-488
<https://issues.apache.org/jira/browse/SDAP-488>`_)
Build the Solr & Webapp Components
-======
+==================================
For the remaining three components, we must now move to the nexus root
directory.
@@ -109,7 +109,7 @@ For the remaining three components, we must now move to the
nexus root directory
cd ${NEXUS_DIR}
Build the Solr Image
--------
+--------------------
First we must move to the Solr Docker directory.
@@ -124,7 +124,7 @@ Now to build the image:
docker build . -t sdap-local/sdap-solr-cloud:${NEXUS_VERSION}
Build the Solr Initialization Image
--------
+-----------------------------------
Now to build the image:
@@ -133,7 +133,7 @@ Now to build the image:
docker build . -t sdap-local/sdap-solr-cloud-init:${NEXUS_VERSION} -f
cloud-init/Dockerfile
Build the Webapp Image
----------
+----------------------
For the final image, we must return to the NEXUS root directory.
@@ -158,13 +158,13 @@ Now we can build the webapp with:
Note: Building does not currently work on Apple Silicon (M1/M2). (`SDAP-488
<https://issues.apache.org/jira/browse/SDAP-488>`_)
Verify Successful Build
-====
+=======================
To verify build success, follow the :ref:`quickstart guide<quickstart>`.
Finished!
-=====
+=========
Congratulations! You have successfully built the images required for running
NEXUS.
diff --git a/docs/conf.py b/docs/conf.py
index b1a35dd..ab9fa64 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -33,8 +33,8 @@
# -- Project information -----------------------------------------------------
-project = 'incubator-sdap-nexus'
-copyright = '2017-2023, Apache SDAP'
+project = 'sdap-nexus'
+copyright = '2017-2024, Apache SDAP'
author = 'Apache SDAP'
# The short X.Y version
@@ -120,7 +120,7 @@ html_static_path = ['_static']
# -- Options for HTMLHelp output ---------------------------------------------
# Output file base name for HTML help builder.
-htmlhelp_basename = 'incubator-sdap-nexusdoc'
+htmlhelp_basename = 'sdap-nexusdoc'
# -- Options for LaTeX output ------------------------------------------------
@@ -147,7 +147,7 @@ latex_elements = {
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
- (master_doc, 'incubator-sdap-nexus.tex', 'incubator-sdap-nexus
Documentation',
+ (master_doc, 'sdap-nexus.tex', 'sdap-nexus Documentation',
'Apache SDAP', 'manual'),
]
@@ -157,7 +157,7 @@ latex_documents = [
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
- (master_doc, 'incubator-sdap-nexus', 'incubator-sdap-nexus Documentation',
+ (master_doc, 'sdap-nexus', 'sdap-nexus Documentation',
[author], 1)
]
@@ -168,8 +168,8 @@ man_pages = [
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
- (master_doc, 'incubator-sdap-nexus', 'incubator-sdap-nexus Documentation',
- author, 'incubator-sdap-nexus', 'One line description of project.',
+ (master_doc, 'sdap-nexus', 'sdap-nexus Documentation',
+ author, 'sdap-nexus', 'One line description of project.',
'Miscellaneous'),
]
diff --git a/docs/index.rst b/docs/index.rst
index d942a6f..160c30f 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -1,5 +1,5 @@
Welcome to the Apache SDAP project documentation!
-================================================
+=================================================
.. toctree::
:maxdepth: 2
@@ -10,6 +10,7 @@ Welcome to the Apache SDAP project documentation!
quickstart
build
dockerimages
+ release
Check out the :ref:`quickstart guide<quickstart>`.
diff --git a/docs/intro.rst b/docs/intro.rst
index 508e75c..86e4dbc 100644
--- a/docs/intro.rst
+++ b/docs/intro.rst
@@ -1,8 +1,8 @@
.. _intro:
-*******************
+***********
About NEXUS
-*******************
+***********
NEXUS is a data-intensive analysis solution using a new approach for handling
science data to enable large-scale data analysis.
diff --git a/docs/make.bat b/docs/make.bat
index a9e146c..97bc5ca 100644
--- a/docs/make.bat
+++ b/docs/make.bat
@@ -9,7 +9,7 @@ if "%SPHINXBUILD%" == "" (
)
set SOURCEDIR=.
set BUILDDIR=_build
-set SPHINXPROJ=incubator-sdap-nexus
+set SPHINXPROJ=sdap-nexus
if "%1" == "" goto help
diff --git a/docs/ningester.rst b/docs/ningester.rst
deleted file mode 100644
index 82634ef..0000000
--- a/docs/ningester.rst
+++ /dev/null
@@ -1,89 +0,0 @@
-.. _ningester:
-
-*****************
-Ningester
-*****************
-
-How to run this image
-======================
-
-The basic command is:
-
-.. code-block:: bash
-
- docker run -it --rm -v <absolute path to config directory on
host>:/config/ -v <absolute path to granule on host>:/data/<granule name>
sdap/ningester <profiles to activate>
-
-Replacing the following:
-
- * ``<absolute path to config directory on host>`` should be the absolute
path on the host to the configuration for the job
- * ``<absolute path to granule on host>`` should be the absolute path on the
host to the granule intended for ingestion
- * ``<granule name>`` should be the filename of the granule
- * ``<profiles to activate>`` is a comma-separated list of profiles to
activate
-
-The `ExampleJob.yml
<https://github.com/apache/incubator-sdap-ningester/blob/bc596c2749a7a2b44a01558b60428f6d008f4f45/docker/example_config/ExampleJob.yml>`_
file shows an example Job configuration that would ingest an AVHRR granule.
-
-Configuration
-==============
-
-Upon running the image, the ningester job will scan the ``/config`` directory
for any files that end with the ``.yml`` extension. Specifically it uses find:
-
-.. code-block:: bash
-
- find /config -name "*.yml" | awk -vORS=, '{ print $1 }'
-
-Therefore, to configure the job, mount your configuration files into
``/config`` using a Docker volume. Alternatively, configuration is loaded via
Spring Boot's `relaxed binding rules
<https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-external-config.html#boot-features-external-config-relaxed-binding>`_.
-So, you can also configure the job through environment variables where each
'level' of the yaml file gets replaced by an '_'.
-
-For example, given a configuration option in yaml that looks like:
-
-.. code-block:: yaml
-
- ningester:
- tile_slicer: sliceFileByTilesDesired
- sliceFileByTilesDesired:
- tilesDesired: 1296
- timeDimension: time
- dimensions:
- - lat
- - lon
-
-These could be replaced with the following corresponding Environment variables:
-
-.. code-block:: bash
-
- NINGESTER_TILE_SLICER=sliceFileByTilesDesired
- NINGESTER_SLICE_FILE_BY_TILES_DESIRED_TILES_DESIRED=1296
- NINGESTER_SLICE_FILE_BY_TILES_DESIRED_TIME_DIMENSION=time
- NINGESTER_SLICE_FILE_BY_TILES_DESIRED_DIMENSIONS[0]=lat
- NINGESTER_SLICE_FILE_BY_TILES_DESIRED_DIMENSIONS[1]=lon
-
-However, because ningester has a lot of configuration options, it is
recommended to use the yaml option.
-
-Data
-=====
-
-Ningester is designed to ingest 1 granule per run. It looks for the granule to
ingest in the ``/data`` directory of the container image.
-Use a Docker volume to mount your data into ``/data``.
-
-The image relies on this command to find the first file in ``/data`` and it
will use that file for ingestion:
-
-.. code-block:: bash
-
- find /data -type f -print -quit
-
-Examples
-==========
-
-A few example commands are shared here.
-
-Docker for Mac
-====================
-
-The `ConnectionSettings-DockerForMac.yml
<https://github.com/apache/incubator-sdap-ningester/blob/bc596c2749a7a2b44a01558b60428f6d008f4f45/docker/example_config/ConnectionSettings-DockerForMac.yml>`_
file shows an example of how to configure the connection settings
-when running this job under Docker for Mac with Solr and Cassandra running on
your host Mac.
-
-Replace ``<path to ningester>`` with the path on your local workstation to the
ningester github project.
-
-.. code-block:: bash
-
- docker run -it --rm -v <path to ningester>/docker/example_config/:/config/
-v <path to
ningester>/src/test/resources/granules/20050101120000-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc:/data/20050101120000-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc.nc
sdap/ningester dockermachost,solr,cassandra
diff --git a/docs/quickstart.rst b/docs/quickstart.rst
index f393493..34553b1 100644
--- a/docs/quickstart.rst
+++ b/docs/quickstart.rst
@@ -1,13 +1,13 @@
.. _quickstart:
-*****************
+*************************
Quickstart Guide - Docker
-*****************
+*************************
This quickstart will take approximately 45 minutes to complete.
Introduction
-=============
+============
NEXUS is a collection of software that enables the analysis of scientific
data. In order to achieve fast analysis, NEXUS takes the approach of breaking
apart, or "tiling", the original data into smaller tiles for storage. Metadata
about each tile is stored in a fast searchable index with a pointer to the
original data array. When an analysis is requested, the necessary tiles are
looked up in the index and then the data for only those tiles is loaded for
processing.
@@ -16,7 +16,7 @@ This quickstart guide will walk you through how to install
and run NEXUS on your
.. _quickstart-prerequisites:
Prerequisites
-==============
+=============
* Docker (tested on v20.10.17)
* Internet Connection
@@ -25,14 +25,14 @@ Prerequisites
* 8.5 GB of disk space
Prepare
-========
+=======
Start downloading the Docker images and set up the Docker bridge network.
.. _quickstart-step1:
Set Default Docker Platform
----
+---------------------------
To ensure consistency when building/running on different hardware
architectures, we should set this variable to ensure docker uses
``linux/amd64``.
@@ -41,7 +41,7 @@ To ensure consistency when building/running on different
hardware architectures,
export DOCKER_DEFAULT_PLATFORM=linux/amd64
Set Tag Variables
--------------------
+-----------------
Pull the necessary Docker images from the `Apache SDAP repository
<https://hub.docker.com/search?q=apache%2Fsdap>`_ on Docker Hub. Please check
the repository for the latest version tag.
@@ -74,7 +74,7 @@ For Local Builds
Continue to the section: "Create a new Docker Bridge Network"
For Release Builds: Pull Docker Images
--------------------
+--------------------------------------
.. code-block:: bash
@@ -94,7 +94,7 @@ For Release Builds: Pull Docker Images
.. _quickstart-step2:
Create a new Docker Bridge Network
-------------------------------------
+----------------------------------
This quickstart consists of launching several Docker containers that need to
communicate with one another. To facilitate this communication, we want to be
able to reference containers via hostname instead of IP address. The default
bridge network used by Docker only supports this by using the ``--link`` option
which is now considered to be `deprecated
<https://docs.docker.com/network/links/>`_.
@@ -109,7 +109,7 @@ The network we will be using for this quickstart will be
called ``sdap-net``. Cr
.. _quickstart-step3:
Start Core Components
-======================
+=====================
NEXUS relies on Apache Solr and Apache Cassandra to store tile metadata and
science data, so let's start those first.
@@ -131,7 +131,7 @@ We then need to ensure the ``/solr`` znode is present.
.. _quickstart-step4:
Start Solr
------------
+----------
SDAP is tested with Solr version 8.11.1.
@@ -156,7 +156,7 @@ When the init script finishes, kill the container by typing
``Ctrl + C``
.. _quickstart-step5:
Starting Cassandra
--------------------
+------------------
SDAP is tested with Cassandra version 3.11.6.
@@ -196,7 +196,7 @@ With Solr and Cassandra started and initialized, we can now
start the collection
.. _quickstart-step6:
Start the Ingester
-===================
+==================
In this section, we will start the components for the ingester. These
components are:
@@ -207,7 +207,7 @@ In this section, we will start the components for the
ingester. These components
We will also be downloading a number of NetCDF files containing science data
for use in this demo.
Create Data Directory
-------------------------
+---------------------
Let's start by creating the directory to hold the science data to ingest.
@@ -221,7 +221,7 @@ Choose a location that is mountable by Docker (typically
needs to be under the u
.. _quickstart-step7:
Start RabbitMQ
-----------------
+--------------
The collection manager and granule ingester(s) use RabbitMQ to communicate, so
we need to start that up first.
@@ -257,7 +257,7 @@ The granule ingester(s) read new granules from the message
queue and process the
.. _quickstart-optional-step:
[OPTIONAL] Run Message Queue Monitor
--------------------------------------
+------------------------------------
The granule ingestion process can take some time. To monitor its progress, we
wrote a simple python script to monitor the message queue. It will wait until
some granules show up and then will exit once they have all been ingested.
@@ -278,7 +278,7 @@ And then run it in a separate shell
.. _quickstart-step9:
Download Sample Data
----------------------
+--------------------
The data we will be downloading is part of the `AVHRR OI dataset
<https://podaac.jpl.nasa.gov/dataset/AVHRR_OI-NCEI-L4-GLOB-v2.0>`_ which
measures sea surface temperature. We will download 1 month of data and ingest
it into a local Solr and Cassandra instance.
@@ -303,7 +303,7 @@ You should now have 30 files downloaded to your data
directory, one for each day
.. _quickstart-step10:
Create Collection Configuration
---------------------------------
+-------------------------------
The collection configuration is a ``.yml`` file that tells the collection
manager what datasets it is managing, where the granules are stored, and how
they are to be tiled.
@@ -338,7 +338,7 @@ The collection configuration is a ``.yml`` file that tells
the collection manage
.. _quickstart-step11:
Start the Collection Manager
------------------------------
+----------------------------
Now we can start the collection manager.
@@ -370,7 +370,7 @@ When it starts, it will publish messages for the downloaded
granules to RabbitMQ
.. _quickstart-step13:
Start the Webapp
-=================
+================
Now that the data is being (has been) ingested, we need to start the webapp
that provides the HTTP interface to the analysis capabilities. This is
currently a python webapp running Tornado and is contained in the nexus-webapp
Docker image. To start the webapp and expose port 8083 use the following
command:
@@ -422,7 +422,7 @@ Click on the ``Time Series Example`` notebook to start it.
This will open the no
.. _quickstart-finished:
Finished!
-================
+=========
Congratulations you have completed the quickstart! In this example you:
@@ -432,7 +432,7 @@ Congratulations you have completed the quickstart! In this
example you:
#. Ran a time series analysis on 1 month of AVHRR OI data and plotted the
result
Cleanup
-========
+=======
To shut down the Solr container cleanly, run the following command:
@@ -449,7 +449,7 @@ The remaining containers can safely be stopped using Docker
Desktop or by runnin
.. _issues:
Known Issues
-=============
+============
This section contains a list of issues that may be encountered while running
this guide, their causes and solutions.
@@ -470,11 +470,10 @@ There are two solutions to this issue:
* Try running only one ingester container.
Collection Manager Messages Not Publishing
--------------------------------------------
+------------------------------------------
RabbitMQ may not receive the messages published by the Collection Manager.
When this happens, new granules added to monitored collections will not be
processed by the ingester(s).
The cause of this issue seems to be due to the RMQ container having limited
resources, which causes message publication to block indefinitely.
To solve this, first figure out which resource is causing issues by navigating
to http://localhost:15672/#/ and sign in with username ``user`` and password
``bitnami``. View the 'Nodes' section. Insufficient resources will be shown in
red. Allocate more of those resources in Docker and restart the Docker daemon.
-
diff --git a/docs/release.rst b/docs/release.rst
new file mode 100644
index 0000000..c476ecc
--- /dev/null
+++ b/docs/release.rst
@@ -0,0 +1,123 @@
+.. _release:
+
+**********************************
+Release Candidate Evaluation Guide
+**********************************
+
+The following guide is intended for SDAP PMC members as instruction for
evaluating release candidates. Non-PMC members should
+also feel free to evaluate candidate releases, though their inputs on release
VOTEs are considered advisory and are non-binding.
+SDAP encourages its whole community to participate in discussion regardless.
+
+Download & Verify Release Candidate
+===================================
+
+Follow the download link in the VOTE email and download all files in the
release candidate directory
+(``https://dist.apache.org/repos/dist/dev/sdap/apache-sdap-<version>-rc<candidate_number>/*``).
+
+Verify Checksums
+----------------
+
+To verify checksums, for each ``.tar.gz`` file in the RC:
+
+.. code-block:: bash
+
+ shasum -a 512 <release-file>.tar.gz | cat - <release-file>.tar.gz.sha512
+
+This will produce a SHA-512 checksum of the release file printed atop the
expected checksum from the release for an easy
+visual comparison.
+
+For each of these, you should also check the checksums against the checksums
provided in the VOTE email.
+
+Verify Signatures
+-----------------
+
+To verify signatures, for each ``.tar.gz`` file in the RC:
+
+.. code-block:: bash
+
+ gpg --verify <release-file>.tar.gz.asc <release-file>.tar.gz.sha512
+
+The expected output should be something similar to
+
+.. code-block::
+
+ gpg: Signature made Mon Jun 10 14:32:40 2024 PDT
+ gpg: using RSA key 4E98C4A32026656E14E0B570FC20035A010E3B7B
+ gpg: Good signature from "Riley Kuttruff (CODE SIGNING KEY)
<[email protected]>" [ultimate]
+
+The name and email of the signing key should correspond to the name and email
that initiated the VOTE thread, and the key
+MUST be in the `KEYS file <https://downloads.apache.org/sdap/KEYS>`_, which
should be linked in the email and also available
+through the `official SDAP Downloads page
<https://sdap.apache.org/downloads>`_.
+
+Build and Check Images
+======================
+
+Image Builds
+------------
+
+Follow the :ref:`Build Guide<build>` to build the SDAP Docker Images.
+
+Check the Images
+----------------
+
+It's a requirement that ASF releases be free of code that is under `certain
3rd-party licenses <https://www.apache.org/legal/resolved.html>`_,
+so the images should be inspected to ensure they are free of any such
dependencies.
+
+We specifically check for Python packages in the sdap-solr-init,
sdap-collection-manager, sdap-granule-ingester and sdap-nexus-webapp
+images:
+
+.. code-block:: bash
+
+ $ docker run --rm --entrypoint /bin/bash <image> -c 'pip install -q
"pip-licenses<4.0" && pip-licenses'
+
+.. note::
+
+ For the sdap-solr-init image, replace ``pip-licenses<4.0`` in the above
command with ``pip-licenses``.
+
+Verify the packages do not include any GPL/LGPL licenses.
+
+Acceptable licenses for a binary:
+
+* Apache
+* MIT
+* BSD-2 / BSD-3
+* MPL
+* Python Software Foundation License
+* HPND (for Pillow)
+* OSI approved (for netCDF4)
+
+Some licenses may be reported as UNKNOWN, this is ok if the package name is
+
+* sdap-collection-manager
+* sdap-ingester-common
+* nexusproto
+
+Otherwise, this should be looked into further.
+
+Any other licences not enumerated above should be checked at the link at the
top of this section, any further questions
+should be relayed to the `SDAP PMC <mailto:[email protected]>`_.
+
+Testing the Images
+==================
+
+Minimum Test
+------------
+
+Verify the images are working by using them in the :ref:`Quickstart
Guide<quickstart>`.
+
+Extended Testing
+----------------
+
+Section coming soon...
+
+Vote
+====
+
+Draft a response to the VOTE thread (`guide on ASF voting
<https://www.apache.org/foundation/voting.html>`_).
+
+It is important you include what you checked/verified and, if applicable, what
issues you found. **Do not just vote +1 or
+-1 without any reasoning!**
+
+Send your completed response.
+
+This completes the release candidate evaluation process.
