This is an automated email from the ASF dual-hosted git repository.
nchung pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-sdap-nexus.git
The following commit(s) were added to refs/heads/master by this push:
new 036daed Build doc (#214)
036daed is described below
commit 036daed824dbddf12f4d8c6d27ab1f5b8c27a40d
Author: Riley Kuttruff <[email protected]>
AuthorDate: Wed Nov 30 15:16:00 2022 -0800
Build doc (#214)
* Guide for building docker images
* TOC & refs
* Verify step
* Docker requirement
* Switched image repos to generic sdap instead of our nexuxjpl
* changelog
* Explicitly defined equality for DomsPoint.
This prevents the duplicate primary points from appearing in the final
results by merging them in the combineByKey step.
* Revert "Explicitly defined equality for DomsPoint."
This reverts commit 3c01a9958d6ca825c9b794e59e69ba31108e3a05.
* Pointed quickstart guide to use ASF images for CM & GI
* Tarball extraction
* Pointed webapp image in qs to ASF repo
* Changelog
* Split version vars to reflect release artifacts
* Language: This guide is for SDAP not NEXUS
* Renamed build directory
* Pointed remaining sdap images to ASF repos
* Renamed solr image to solr cloud to better match public image name
* Changed repository link to ASF instead of JPL
* Update changelog
* Improved qs interoperability w/ locally built images
* Improved qs interoperability w/ locally built images
Co-authored-by: rileykk <[email protected]>
---
CHANGELOG.md | 2 +
docs/build.rst | 133 ++++++++++++++++++++++++++++++++++++++++++++++++++++
docs/index.rst | 4 +-
docs/quickstart.rst | 56 +++++++++++++++-------
4 files changed, 175 insertions(+), 20 deletions(-)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 5f2a2d8..2256ba7 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -23,6 +23,8 @@ and this project adheres to [Semantic
Versioning](https://semver.org/spec/v2.0.0
- RabbitMQ monitor script used in Docker quickstart guide
- Added new option for NCAR so either NCAR or JPL Insitu API can be specified
- SDAP-407: Added depth to `/domsresults` endpoint
+- Added documentation for building SDAP docker images
+ - Prepared documentation for v1.0.0 release.
### Changed
- SDAP-390: Changed `/doms` to `/cdms` and `doms_reader.py` to `cdms_reader.py`
- domslist endpoint points to AWS insitu instead of doms insitu
diff --git a/docs/build.rst b/docs/build.rst
new file mode 100644
index 0000000..da940c0
--- /dev/null
+++ b/docs/build.rst
@@ -0,0 +1,133 @@
+.. _build:
+
+*****************
+How to Build the SDAP Docker Images
+*****************
+
+In this guide, we will go over how to build the docker images for the various
components of SDAP.
+
+Prepare
+===========
+
+First, we must ensure we have Docker installed and running. For this guide, we
used Docker Desktop version 4.12.0. Download Docker Desktop `here.
<https://www.docker.com/products/docker-desktop/>`_
+
+Now we must download and extract the source code for NEXUS and the ingester.
+
+.. code-block:: bash
+
+ export NEXUS_DIR=~/sdap-build/nexus
+ export INGESTER_DIR=~/sdap-build/ingester
+
+ mkdir -p ${NEXUS_DIR}
+ mkdir -p ${INGESTER_DIR}
+
+We should also set variables for a consistent tag across all images. These
should be consistent with the release versions we plan to build.
+
+.. code-block:: bash
+
+ export NEXUS_VERSION=1.0.0
+ export INGESTER_VERSION=1.0.0
+
+To build the necessary images, we will need to first download the latest
releases for SDAP NEXUS and SDAP Ingester ((URL)). We will not need to download
the SDAP Nexusproto release directly.
+
+Now we must extract the releases to their respective directories.
+
+.. note::
+
+ In the following code block, make sure the versions in the filenames
correspond with the versions that were downloaded.
+
+.. code-block:: bash
+
+ tar xvf apache-sdap-ingester-${INGESTER_VERSION}-src.tar.gz -C
${INGESTER_DIR}
+ tar xvf apache-sdap-nexus-${NEXUS_VERSION}-src.tar.gz -C ${NEXUS_DIR}
+
+Build Ingester Components
+=========================
+
+There are two components to build: the Collection Manager & Granule Ingester.
+
+For both of these, we must first move to the ingester root directory.
+
+.. code-block:: bash
+
+ cd ${INGESTER_DIR}
+
+Build the Collection Manager
+-------
+
+From the ingester root directory, run:
+
+.. code-block:: bash
+
+ docker build . -f collection_manager/docker/Dockerfile -t
sdap-local/sdap-collection-manager:${INGESTER_VERSION}
+
+Build the Granule Ingester
+-------
+
+From the ingester root directory, run:
+
+.. code-block:: bash
+
+ docker build . -f granule_ingester/docker/Dockerfile -t
sdap-local/sdap-granule-ingester:${INGESTER_VERSION}
+
+Build the Solr & Webapp Components
+======
+
+For the remaining three components, we must now move to the nexus root
directory.
+
+.. code-block:: bash
+
+ cd ${NEXUS_DIR}
+
+Build the Solr Image
+-------
+
+First we must move to the Solr Docker directory.
+
+.. code-block:: bash
+
+ cd docker/solr
+
+Now to build the image:
+
+.. code-block:: bash
+
+ docker build . -t sdap-local/sdap-solr-cloud:${NEXUS_VERSION}
+
+Build the Solr Initialization Image
+-------
+
+Now to build the image:
+
+.. code-block:: bash
+
+ docker build . -t sdap-local/sdap-solr-cloud-init:${NEXUS_VERSION} -f
cloud-init/Dockerfile
+
+Build the Webapp Image
+---------
+
+For the final image, we must return to the NEXUS root directory.
+
+.. code-block:: bash
+
+ cd ${NEXUS_DIR}
+
+Now we can build the webapp with:
+
+.. code-block:: bash
+
+ docker build . -f docker/nexus-webapp/Dockerfile -t
sdap-local/sdap-nexus-webapp:${NEXUS_VERSION}
+
+Verify Successful Build
+====
+
+To verify build success, follow the :ref:`quickstart guide<quickstart>`.
+
+
+Finished!
+=====
+
+Congratulations! You have successfully built the images required for running
NEXUS.
+
+If you used your own repository for the image tags, you can push them using
``docker push``.
+
diff --git a/docs/index.rst b/docs/index.rst
index 0db88d4..649d0e7 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -11,8 +11,8 @@ Welcome to incubator-sdap-nexus's documentation!
intro
quickstart
- ningester
+ build
dockerimages
-Check out the :ref:`quickstart`.
+Check out the :ref:`quickstart guide<quickstart>`.
diff --git a/docs/quickstart.rst b/docs/quickstart.rst
index f9dc3cf..cedb2ec 100644
--- a/docs/quickstart.rst
+++ b/docs/quickstart.rst
@@ -31,33 +31,53 @@ Start downloading the Docker images and set up the Docker
bridge network.
.. _quickstart-step1:
-Pull Docker Images
+Set Tag Variables
-------------------
-Pull the necessary Docker images from the `NEXUS JPL repository
<https://hub.docker.com/u/nexusjpl>`_ on Docker Hub. Please check the
repository for the latest version tag.
+Pull the necessary Docker images from the `Apache SDAP repository
<https://hub.docker.com/search?q=apache%2Fsdap>`_ on Docker Hub. Please check
the repository for the latest version tag.
.. code-block:: bash
export CASSANDRA_VERSION=3.11.6-debian-10-r138
export RMQ_VERSION=3.8.9-debian-10-r37
- export COLLECTION_MANAGER_VERSION=0.1.6a14
- export GRANULE_INGESTER_VERSION=0.1.6a30
- export WEBAPP_VERSION=distributed.0.4.5a54
- export SOLR_VERSION=8.11.1
- export SOLR_CLOUD_INIT_VERSION=1.0.2
+ export COLLECTION_MANAGER_VERSION=1.0.0
+ export GRANULE_INGESTER_VERSION=1.0.0
+ export WEBAPP_VERSION=1.0.0
+ export SOLR_VERSION=1.0.0
+ export SOLR_CLOUD_INIT_VERSION=1.0.0
export ZK_VERSION=3.5.5
export JUPYTER_VERSION=1.0.0-rc2
+For Local Builds
+----------------
+
.. code-block:: bash
+ export REPO=sdap-local
+
+ docker pull bitnami/cassandra:${CASSANDRA_VERSION}
+ docker pull bitnami/rabbitmq:${RMQ_VERSION}
+ docker pull zookeeper:${ZK_VERSION}
+
+ docker pull nexusjpl/jupyter:${JUPYTER_VERSION}
+
+Continue to the section: "Create a new Docker Bridge Network"
+
+For Release Builds: Pull Docker Images
+-------------------
+
+.. code-block:: bash
+
+ export REPO=apache
+
docker pull bitnami/cassandra:${CASSANDRA_VERSION}
docker pull bitnami/rabbitmq:${RMQ_VERSION}
- docker pull nexusjpl/collection-manager:${COLLECTION_MANAGER_VERSION}
- docker pull nexusjpl/granule-ingester:${GRANULE_INGESTER_VERSION}
- docker pull nexusjpl/nexus-webapp:${WEBAPP_VERSION}
- docker pull nexusjpl/solr:${SOLR_VERSION}
- docker pull nexusjpl/solr-cloud-init:${SOLR_CLOUD_INIT_VERSION}
+ docker pull apache/sdap-collection-manager:${COLLECTION_MANAGER_VERSION}
+ docker pull apache/sdap-granule-ingester:${GRANULE_INGESTER_VERSION}
+ docker pull apache/sdap-nexus-webapp:${WEBAPP_VERSION}
+ docker pull apache/sdap-solr-cloud:${SOLR_VERSION}
+ docker pull apache/sdap-solr-cloud-init:${SOLR_CLOUD_INIT_VERSION}
docker pull zookeeper:${ZK_VERSION}
docker pull nexusjpl/jupyter:${JUPYTER_VERSION}
@@ -114,13 +134,13 @@ To start Solr using a volume mount and expose the admin
webapp on port 8983:
export SOLR_DATA=~/nexus-quickstart/solr
mkdir -p ${SOLR_DATA}
- docker run --name solr --network sdap-net -v
${SOLR_DATA}/:/opt/solr/server/solr/nexustiles/data -p 8983:8983 -e
ZK_HOST="host.docker.internal:2181/solr" -d nexusjpl/solr:${SOLR_VERSION}
+ docker run --name solr --network sdap-net -v
${SOLR_DATA}/:/opt/solr/server/solr/nexustiles/data -p 8983:8983 -e
ZK_HOST="host.docker.internal:2181/solr" -d
${REPO}/sdap-solr-cloud:${SOLR_VERSION}
This will start an instance of Solr. To initialize it, we need to run the
``solr-cloud-init`` image.
.. code-block:: bash
- docker run -it --rm --name solr-init --network sdap-net -e
SDAP_ZK_SOLR="host.docker.internal:2181/solr" -e
SDAP_SOLR_URL="http://host.docker.internal:8983/solr/"; -e
CREATE_COLLECTION_PARAMS="name=nexustiles&numShards=1&waitForFinalState=true"
nexusjpl/solr-cloud-init:${SOLR_CLOUD_INIT_VERSION}
+ docker run -it --rm --name solr-init --network sdap-net -e
SDAP_ZK_SOLR="host.docker.internal:2181/solr" -e
SDAP_SOLR_URL="http://host.docker.internal:8983/solr/"; -e
CREATE_COLLECTION_PARAMS="name=nexustiles&numShards=1&waitForFinalState=true"
${REPO}/sdap-solr-cloud-init:${SOLR_CLOUD_INIT_VERSION}
When the init script finishes, kill the container by typing ``Ctrl + C``
@@ -220,10 +240,10 @@ The granule ingester(s) read new granules from the
message queue and process the
EOF
docker run --name granule-ingester-1 --network sdap-net -d --env-file
granule-ingester.env \
- -v ${DATA_DIRECTORY}:/data/granules/
nexusjpl/granule-ingester:${GRANULE_INGESTER_VERSION}
+ -v ${DATA_DIRECTORY}:/data/granules/
${REPO}/sdap-granule-ingester:${GRANULE_INGESTER_VERSION}
docker run --name granule-ingester-2 --network sdap-net -d --env-file
granule-ingester.env \
- -v ${DATA_DIRECTORY}:/data/granules/
nexusjpl/granule-ingester:${GRANULE_INGESTER_VERSION}
+ -v ${DATA_DIRECTORY}:/data/granules/
${REPO}/sdap-granule-ingester:${GRANULE_INGESTER_VERSION}
.. _quickstart-optional-step:
@@ -317,7 +337,7 @@ Now we can start the collection manager.
.. code-block:: bash
- docker run --name collection-manager --network sdap-net -v
${DATA_DIRECTORY}:/data/granules/ -v ${CONFIG_DIR}:/home/ingester/config/ -e
COLLECTIONS_PATH="/home/ingester/config/collectionConfig.yml" -e
HISTORY_URL="http://host.docker.internal:8983/"; -e
RABBITMQ_HOST="host.docker.internal:5672" -e RABBITMQ_USERNAME="user" -e
RABBITMQ_PASSWORD="bitnami" -d
nexusjpl/collection-manager:${COLLECTION_MANAGER_VERSION}
+ docker run --name collection-manager --network sdap-net -v
${DATA_DIRECTORY}:/data/granules/ -v ${CONFIG_DIR}:/home/ingester/config/ -e
COLLECTIONS_PATH="/home/ingester/config/collectionConfig.yml" -e
HISTORY_URL="http://host.docker.internal:8983/"; -e
RABBITMQ_HOST="host.docker.internal:5672" -e RABBITMQ_USERNAME="user" -e
RABBITMQ_PASSWORD="bitnami" -d
${REPO}/sdap-collection-manager:${COLLECTION_MANAGER_VERSION}
.. _quickstart-step12:
@@ -349,7 +369,7 @@ Now that the data is being (has been) ingested, we need to
start the webapp that
.. code-block:: bash
- docker run -d --name nexus-webapp --network sdap-net -p 8083:8083
nexusjpl/nexus-webapp:${WEBAPP_VERSION} python3
/incubator-sdap-nexus/analysis/webservice/webapp.py
--solr_host="http://host.docker.internal:8983";
--cassandra_host=host.docker.internal --cassandra_username=cassandra
--cassandra_password=cassandra
+ docker run -d --name nexus-webapp --network sdap-net -p 8083:8083
${REPO}/sdap-nexus-webapp:${WEBAPP_VERSION} python3
/incubator-sdap-nexus/analysis/webservice/webapp.py
--solr_host="http://host.docker.internal:8983";
--cassandra_host=host.docker.internal --cassandra_username=cassandra
--cassandra_password=cassandra
.. note:: If you see a message like ``docker: invalid reference format`` it
likely means you need to re-export the ``WEBAPP_VERSION`` environment variable
again. This can happen when you open a new terminal window or tab.
