This is an automated email from the ASF dual-hosted git repository.
nchung pushed a commit to branch release/1.1.0
in repository https://gitbox.apache.org/repos/asf/incubator-sdap-nexus.git
The following commit(s) were added to refs/heads/release/1.1.0 by this push:
new 68d0870 SDAP-451 - Updates to quickstart guide for the next release
(#242)
68d0870 is described below
commit 68d0870fecc4209470bd1b305ef6c5b858b033c0
Author: Riley Kuttruff <[email protected]>
AuthorDate: Wed Apr 26 11:46:50 2023 -0700
SDAP-451 - Updates to quickstart guide for the next release (#242)
* Copy button
* Platform env variable
* Removed potential syntax error
* C&S persist mount paths
* Removed broken pip install option
* SDAP-451 added granule download script from EarthData Search
* SDAP-451 Updated script to remove my username as default
* SDAP-451 Updated quickstart granule download section
* Changelog
* Updated script download URL
* Bumped image tag version
1.0.0 -> 1.1.0
* Added granule download script from #242 to master in advance for release
logistics (#244)
Co-authored-by: rileykk <[email protected]>
* Removed script from PR since it was added to master
* Updated Changelog
---------
Co-authored-by: rileykk <[email protected]>
---
CHANGELOG.md | 8 ++++++++
docker/nexus-webapp/Dockerfile | 2 +-
docs/build.rst | 13 +++++++++++--
docs/conf.py | 1 +
docs/quickstart.rst | 41 ++++++++++++++++++++++++-----------------
tools/rmqmonitor/monitor.py | 2 +-
6 files changed, 46 insertions(+), 21 deletions(-)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 933ccb6..b0b8fb3 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -21,6 +21,13 @@ and this project adheres to [Semantic
Versioning](https://semver.org/spec/v2.0.0
- `platforms` parameter in `/match_spark` is now a multi-select list.
- Added note to `/stats` endpoint to note it is limited to satellite
datasets
- SDAP-450: Updated helm chart to reflect k8s 1.22 changes. Bumped RMQ
dependency version & updated Bitnami dependency chart URLs. Ingress template is
already up to date.
+- SDAP-451: Updated quickstart documentation
+ - Specified default Docker platform as `linux/amd64`
+ - Added copy button extension for RDT pages
+ - Corrected volume mount paths for Solr and Cassandra containers
+ - Fixed (potential) bug in RMQ monitor script
+ - Updated granule download section: Replaced with a script that also
accounts for dataset migration
+ - Bumped image tag versions (1.0.0 -> 1.1.0)
### Deprecated
### Removed
### Fixed
@@ -34,6 +41,7 @@ and this project adheres to [Semantic
Versioning](https://semver.org/spec/v2.0.0
- SDAP-412: Explicit definition of `__eq__` and `__hash__` in matchup
`DomsPoint` class. This ensures all primary-secondary pairs with the same
primary point are merged in the `combineByKey` step.
- SDAP-438: Replace variable value NaN with None to fix error in result storage
- SDAP-444: Fixed `resultSizeLimit` param in `/match_spark` truncating the
results that are stored for the results endpoint.
+- Fixed minor webapp image build issue
### Security
## [1.0.0] - 2022-12-05
diff --git a/docker/nexus-webapp/Dockerfile b/docker/nexus-webapp/Dockerfile
index 1caf462..79381f5 100644
--- a/docker/nexus-webapp/Dockerfile
+++ b/docker/nexus-webapp/Dockerfile
@@ -95,7 +95,7 @@ RUN python3 setup.py install clean && mamba clean -afy
WORKDIR /incubator-sdap-nexus/tools/deletebyquery
-RUN pip3 install cassandra-driver==3.20.1 --install-option="--no-cython"
+RUN pip3 install cassandra-driver==3.20.1
RUN pip3 install pyspark py4j
RUN pip3 install -r requirements.txt
RUN pip3 install cython
diff --git a/docs/build.rst b/docs/build.rst
index 43a8f22..bc1be2e 100644
--- a/docs/build.rst
+++ b/docs/build.rst
@@ -25,8 +25,8 @@ We should also set variables for a consistent tag across all
images. These shoul
.. code-block:: bash
- export NEXUS_VERSION=1.0.0
- export INGESTER_VERSION=1.0.0
+ export NEXUS_VERSION=1.1.0
+ export INGESTER_VERSION=1.1.0
To build the necessary images, we will need to first download the latest
releases for SDAP NEXUS and SDAP Ingester ((URL)). We will not need to download
the SDAP Nexusproto release directly.
@@ -51,6 +51,15 @@ Create a temporary directory to extract to.
mv
${TMP_DIR}/Apache-SDAP/apache-sdap-ingester-${INGESTER_VERSION}-incubating-src/*
${INGESTER_DIR}
mv
${TMP_DIR}/Apache-SDAP/apache-sdap-nexus-${NEXUS_VERSION}-incubating-src/*
${NEXUS_DIR}
+Set Default Docker Platform
+---
+
+To ensure consistency when building/running on different hardware
architectures, we should set this variable to ensure docker uses
``linux/amd64``.
+
+.. code-block:: bash
+
+ export DOCKER_DEFAULT_PLATFORM=linux/amd64
+
Build Ingester Components
=========================
diff --git a/docs/conf.py b/docs/conf.py
index 9c40b8a..b1a35dd 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -57,6 +57,7 @@ extensions = [
'sphinx.ext.doctest',
'sphinx.ext.intersphinx',
'sphinx.ext.viewcode',
+ 'sphinx_copybutton'
]
# Add any paths that contain templates here, relative to this directory.
diff --git a/docs/quickstart.rst b/docs/quickstart.rst
index cedb2ec..29ed8e3 100644
--- a/docs/quickstart.rst
+++ b/docs/quickstart.rst
@@ -31,6 +31,15 @@ Start downloading the Docker images and set up the Docker
bridge network.
.. _quickstart-step1:
+Set Default Docker Platform
+---
+
+To ensure consistency when building/running on different hardware
architectures, we should set this variable to ensure docker uses
``linux/amd64``.
+
+.. code-block:: bash
+
+ export DOCKER_DEFAULT_PLATFORM=linux/amd64
+
Set Tag Variables
-------------------
@@ -40,11 +49,11 @@ Pull the necessary Docker images from the `Apache SDAP
repository <https://hub.d
export CASSANDRA_VERSION=3.11.6-debian-10-r138
export RMQ_VERSION=3.8.9-debian-10-r37
- export COLLECTION_MANAGER_VERSION=1.0.0
- export GRANULE_INGESTER_VERSION=1.0.0
- export WEBAPP_VERSION=1.0.0
- export SOLR_VERSION=1.0.0
- export SOLR_CLOUD_INIT_VERSION=1.0.0
+ export COLLECTION_MANAGER_VERSION=1.1.0
+ export GRANULE_INGESTER_VERSION=1.1.0
+ export WEBAPP_VERSION=1.1.0
+ export SOLR_VERSION=1.1.0
+ export SOLR_CLOUD_INIT_VERSION=1.1.0
export ZK_VERSION=3.5.5
export JUPYTER_VERSION=1.0.0-rc2
@@ -134,7 +143,7 @@ To start Solr using a volume mount and expose the admin
webapp on port 8983:
export SOLR_DATA=~/nexus-quickstart/solr
mkdir -p ${SOLR_DATA}
- docker run --name solr --network sdap-net -v
${SOLR_DATA}/:/opt/solr/server/solr/nexustiles/data -p 8983:8983 -e
ZK_HOST="host.docker.internal:2181/solr" -d
${REPO}/sdap-solr-cloud:${SOLR_VERSION}
+ docker run --name solr --network sdap-net -v ${SOLR_DATA}/:/bitnami -p
8983:8983 -e ZK_HOST="host.docker.internal:2181/solr" -d
${REPO}/sdap-solr-cloud:${SOLR_VERSION}
This will start an instance of Solr. To initialize it, we need to run the
``solr-cloud-init`` image.
@@ -174,7 +183,7 @@ Now we can start the image and run the initialization
script.
export CASSANDRA_DATA=~/nexus-quickstart/cassandra
mkdir -p ${CASSANDRA_DATA}
- docker run --name cassandra --network sdap-net -p 9042:9042 -v
${CASSANDRA_DATA}/cassandra/:/var/lib/cassandra -v
"${CASSANDRA_INIT}/initdb.cql:/scripts/initdb.cql" -d
bitnami/cassandra:${CASSANDRA_VERSION}
+ docker run --name cassandra --network sdap-net -p 9042:9042 -v
${CASSANDRA_DATA}/cassandra/:/bitnami -v
"${CASSANDRA_INIT}/initdb.cql:/scripts/initdb.cql" -d
bitnami/cassandra:${CASSANDRA_VERSION}
Wait a few moments for the database to start.
@@ -273,23 +282,21 @@ Download Sample Data
The data we will be downloading is part of the `AVHRR OI dataset
<https://podaac.jpl.nasa.gov/dataset/AVHRR_OI-NCEI-L4-GLOB-v2.0>`_ which
measures sea surface temperature. We will download 1 month of data and ingest
it into a local Solr and Cassandra instance.
+.. note::
+
+ Before you are able to download the test data, you will need to `Create an
Earthdata Login <https://urs.earthdata.nasa.gov/>`_.
+
Then go ahead and download 1 month worth of AVHRR netCDF files.
.. code-block:: bash
cd $DATA_DIRECTORY
- export
URL_LIST="https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/305/20151101120000-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/306/20151102120000-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/307/20151103120000
[...]
-
- for url in ${URL_LIST}; do
- curl -O "${url}"
- done
-
-.. note::
-
- The dataset is pending migration from PO.DAAC to the Earthdata Cloud in AWS.
Following this migration, the above links may not work (though an updated
download method will soon follow)
+ curl -O
https://raw.githubusercontent.com/apache/incubator-sdap-nexus/master/docs/granule-download.sh
+ chmod 700 granule-download.sh
+ ./granule-download.sh
- For reference: There are 30 granules for every day in November of 2015.
+ rm granule-download.sh
You should now have 30 files downloaded to your data directory, one for each
day in November 2015.
diff --git a/tools/rmqmonitor/monitor.py b/tools/rmqmonitor/monitor.py
index 271388e..cabe0b2 100644
--- a/tools/rmqmonitor/monitor.py
+++ b/tools/rmqmonitor/monitor.py
@@ -63,6 +63,6 @@ while True:
print(f"RMQ request {req_number} failed: {response.status_code}",
end='\n', flush=True)
except Exception as err:
delete_last_line()
- print(f"An exception occurred: Request {req_number}, error:
{type(err)=}", end='\n', flush=True)
+ print(f"An exception occurred: Request {req_number}, error:
{type(err)}", end='\n', flush=True)
time.sleep(5)
