[GitHub] [incubator-sdap-website] RKuttruff merged pull request #6: [SDAP-431] Downloads page w/ links to v1.0.0 - Deploy

[GitBox]

RKuttruff merged PR #6:
URL: https://github.com/apache/incubator-sdap-website/pull/6


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-website] RKuttruff opened a new pull request, #6: [SDAP-431] Downloads page w/ links to v1.0.0 - Deploy

[GitBox]

RKuttruff opened a new pull request, #6:
URL: https://github.com/apache/incubator-sdap-website/pull/6

   Actually do the deploy this time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-website] RKuttruff merged pull request #5: [SDAP-431] Downloads page w/ links to v1.0.0

[GitBox]

RKuttruff merged PR #5:
URL: https://github.com/apache/incubator-sdap-website/pull/5


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff merged pull request #225: Update build guide to reflect updated release file structure

[GitBox]

RKuttruff merged PR #225:
URL: https://github.com/apache/incubator-sdap-nexus/pull/225


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-mudrod] dependabot[bot] commented on pull request #56: Bump jettison from 1.3.8 to 1.5.1 in /core

[GitBox]

dependabot[bot] commented on PR #56:
URL: 
https://github.com/apache/incubator-sdap-mudrod/pull/56#issuecomment-1369244161

   Superseded by #57.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-mudrod] dependabot[bot] closed pull request #56: Bump jettison from 1.3.8 to 1.5.1 in /core

[GitBox]

dependabot[bot] closed pull request #56: Bump jettison from 1.3.8 to 1.5.1 in 
/core
URL: https://github.com/apache/incubator-sdap-mudrod/pull/56


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-mudrod] dependabot[bot] opened a new pull request, #57: Bump jettison from 1.3.8 to 1.5.2 in /core

[GitBox]

dependabot[bot] opened a new pull request, #57:
URL: https://github.com/apache/incubator-sdap-mudrod/pull/57

   Bumps [jettison](https://github.com/jettison-json/jettison) from 1.3.8 to 
1.5.2.
   
   Release notes
   Sourced from https://github.com/jettison-json/jettison/releases;>jettison's 
releases.
   
   Jettison 1.5.2
   What's Changed
   
   Bump woodstox-core from 6.2.8 to 6.4.0 by https://github.com/dependabot;>@​dependabot in https://github-redirect.dependabot.com/jettison-json/jettison/pull/51;>jettison-json/jettison#51
   Fixing StackOverflow error by https://github.com/coheigea;>@​coheigea in https://github-redirect.dependabot.com/jettison-json/jettison/pull/53;>jettison-json/jettison#53
   
   Full Changelog: https://github.com/jettison-json/jettison/compare/jettison-1.5.1...jettison-1.5.2;>https://github.com/jettison-json/jettison/compare/jettison-1.5.1...jettison-1.5.2
   Jettison 1.5.1
   What's Changed
   
   Stack Overflow fix on malformed JSON by https://github.com/coheigea;>@​coheigea in https://github-redirect.dependabot.com/jettison-json/jettison/pull/48;>jettison-json/jettison#48
   Prevent infinite loop when a /* comment is not terminated by https://github.com/coheigea;>@​coheigea in https://github-redirect.dependabot.com/jettison-json/jettison/pull/49;>jettison-json/jettison#49
   
   Full Changelog: https://github.com/jettison-json/jettison/compare/jettison-1.5.0...jettison-1.5.1;>https://github.com/jettison-json/jettison/compare/jettison-1.5.0...jettison-1.5.1
   
   
   
   Commits
   
   https://github.com/jettison-json/jettison/commit/6dc73a07ac47c7e3b4c7d041761fbc11aee2eb50;>6dc73a0
 [maven-release-plugin] prepare release jettison-1.5.2
   https://github.com/jettison-json/jettison/commit/19ae19ff57d00dbfa6f6c3af4fc4cb14fb5ca2df;>19ae19f
 Fixing StackOverflow error
   https://github.com/jettison-json/jettison/commit/325b51b36982b5ddfb10d76d2809ec422e9de0b1;>325b51b
 Bump woodstox-core from 6.2.8 to 6.4.0
   https://github.com/jettison-json/jettison/commit/81d3786242c3c7f68eead911b432a81c7f24fa0b;>81d3786
 [maven-release-plugin] prepare for next development iteration
   https://github.com/jettison-json/jettison/commit/bdb3982aaea184c2143e41981e33877c2fd66292;>bdb3982
 [maven-release-plugin] prepare release jettison-1.5.1
   https://github.com/jettison-json/jettison/commit/1268b7558bad9b989687009a094466b64d4da533;>1268b75
 Prevent infinite loop when a /* comment is not terminated
   https://github.com/jettison-json/jettison/commit/cff9f2814ec8ae573e38f38abed039311bb7bfda;>cff9f28
 Create codeql-analysis.yml
   https://github.com/jettison-json/jettison/commit/395f8625bcf688743872c8e7f59360d372e77811;>395f862
 Stack Overflow fix on malformed JSON
   https://github.com/jettison-json/jettison/commit/a5d2223bafd4c5ec37723533fa5c4066ead4eaaa;>a5d2223
 [maven-release-plugin] prepare for next development iteration
   https://github.com/jettison-json/jettison/commit/e1bf5293d10af3ebfcd8763eb51a20a659b9047c;>e1bf529
 [maven-release-plugin] prepare release jettison-1.5.0
   Additional commits viewable in https://github.com/jettison-json/jettison/compare/jettison-1.3.8...jettison-1.5.2;>compare
 view
   
   
   
   
   
   [![Dependabot compatibility 
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=org.codehaus.jettison:jettison=maven=1.3.8=1.5.2)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
   
   Dependabot will resolve any conflicts with this PR as long as you don't 
alter it yourself. You can also trigger a rebase manually by commenting 
`@dependabot rebase`.
   
   [//]: # (dependabot-automerge-start)
   [//]: # (dependabot-automerge-end)
   
   ---
   
   
   Dependabot commands and options
   
   
   You can trigger Dependabot actions by commenting on this PR:
   - `@dependabot rebase` will rebase this PR
   - `@dependabot recreate` will recreate this PR, overwriting any edits that 
have been made to it
   - `@dependabot merge` will merge this PR after your CI passes on it
   - `@dependabot squash and merge` will squash and merge this PR after your CI 
passes on it
   - `@dependabot cancel merge` will cancel a previously requested merge and 
block automerging
   - `@dependabot reopen` will reopen this PR if it is closed
   - `@dependabot close` will close this PR and stop Dependabot recreating it. 
You can achieve the same result by closing it manually
   - `@dependabot ignore this major version` will close this PR and stop 
Dependabot creating any more for this major version (unless you reopen the PR 
or upgrade to it yourself)
   - `@dependabot ignore this minor version` will close this PR and stop 
Dependabot creating any more for this minor version (unless you reopen the PR 
or upgrade to it yourself)
   - `@dependabot ignore this dependency` will close this PR and stop 
Dependabot creating any more for this dependency (unless you reopen the PR or 

[GitHub] [incubator-sdap-ingester] RKuttruff opened a new pull request, #69: SDAP-417: Prevent SolrStore from generating documents with incorrect WKT for small tiles

[GitBox]

RKuttruff opened a new pull request, #69:
URL: https://github.com/apache/incubator-sdap-ingester/pull/69

   For narrow tiles (max/min lat/lon difference ~<0.001) the generated WKT for 
the geo field for the Solr document would essentially collapse to a lower 
dimensional shape while being described as a higher dimensional shape (ie a 
POLYGON that was a line or a LINESTRING that was a point). Solr would throw an 
error when trying to write such documents, which has been observed occurring 
when ingesting granules for the VIIRS_NPP-JPL-L2P-v2016.2 dataset.
   
   To fix this, we now check to see if the collapse will occur (if the bbox 
values differ but the string representations don't) and then force the string 
representations of the max/min values to have at least a 0.001 difference.
   
   This has been tested successfully with a granule that previously caused this 
issue.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff opened a new pull request, #224: Script to verify tiles described in Solr are present in Cassandra

[GitBox]

RKuttruff opened a new pull request, #224:
URL: https://github.com/apache/incubator-sdap-nexus/pull/224

   Arguments are handled similarly to the deletebyquery tool. This script is 
currently quite slow but works.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-ingester] ngachung merged pull request #68: README file draft

[GitBox]

ngachung merged PR #68:
URL: https://github.com/apache/incubator-sdap-ingester/pull/68


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] ngachung merged pull request #223: Rc2 readme

[GitBox]

ngachung merged PR #223:
URL: https://github.com/apache/incubator-sdap-nexus/pull/223


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexusproto] ngachung merged pull request #13: Rc2 readme

[GitBox]

ngachung merged PR #13:
URL: https://github.com/apache/incubator-sdap-nexusproto/pull/13


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-ingester] ngachung merged pull request #67: Create DISCLAIMER

[GitBox]

ngachung merged PR #67:
URL: https://github.com/apache/incubator-sdap-ingester/pull/67


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] ngachung merged pull request #222: Create DISCLAIMER

[GitBox]

ngachung merged PR #222:
URL: https://github.com/apache/incubator-sdap-nexus/pull/222


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] DendriteQ commented on pull request #174: CDMS-122 - Limited Zarr Proxy

[GitBox]

DendriteQ commented on PR #174:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/174#issuecomment-1343990463

   Eeh can someone reinstall cloning zebra fish.
   I need the parent prosess instructions. And whats left after falling is the
   start child prosess.and in private reality seems their after my body. Theft.
   
   On Thu, Dec 8, 2022, 00:11 Riley Kuttruff ***@***.***> wrote:
   
   > Merged #174 
   > into zarr-testing.
   >
   > —
   > Reply to this email directly, view it on GitHub
   > ,
   > or unsubscribe
   > 

   > .
   > You are receiving this because you are subscribed to this thread.Message
   > ID: ***@***.***
   > com>
   >
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff merged pull request #174: CDMS-122 - Limited Zarr Proxy

[GitBox]

RKuttruff merged PR #174:
URL: https://github.com/apache/incubator-sdap-nexus/pull/174


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] skorper commented on pull request #220: cdmssubset bugfix

[GitBox]

skorper commented on PR #220:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/220#issuecomment-1338813448

   Using the latest commit, generated the zip below with the following request: 
`/cdmssubset?dataset=ASCATB-L2-Coastal=ICOADS Release 
3.0=2017-07-01T00:00:00Z=2017-07-01T23:59:59Z=-90,27,-85,29=0=5=0,16,17,30,41,42=ZIP`
   
   
[response.zip](https://github.com/apache/incubator-sdap-nexus/files/10163119/response.zip)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexusproto] ngachung merged pull request #12: Create NOTICE

[GitBox]

ngachung merged PR #12:
URL: https://github.com/apache/incubator-sdap-nexusproto/pull/12


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-ingester] ngachung merged pull request #66: Create NOTICE

[GitBox]

ngachung merged PR #66:
URL: https://github.com/apache/incubator-sdap-ingester/pull/66


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-ingester] ngachung merged pull request #65: Added ASF header to all files missing it in this repo

[GitBox]

ngachung merged PR #65:
URL: https://github.com/apache/incubator-sdap-ingester/pull/65


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] ngachung merged pull request #221: Create NOTICE

[GitBox]

ngachung merged PR #221:
URL: https://github.com/apache/incubator-sdap-nexus/pull/221


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] ngachung merged pull request #219: Added missing ASF headers to all .py files

[GitBox]

ngachung merged PR #219:
URL: https://github.com/apache/incubator-sdap-nexus/pull/219


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #220: cdmssubset bugfix

[GitBox]

RKuttruff commented on code in PR #220:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/220#discussion_r1040163330


##
analysis/webservice/algorithms/doms/subsetter.py:
##
@@ -309,16 +300,21 @@ def toCsv(self):
 ]
 data_variables = list(set([keys for result in results for keys in 
result['data'].keys()]))
 data_variables.sort()
+
+if 'id' in list(set([keys for result in results for keys in 
result.keys()])):
+headers.append('id')
+
 headers.extend(data_variables)
 for i, result in enumerate(results):
 cols = []
 
 cols.append(result['longitude'])
 cols.append(result['latitude'])
 
cols.append(datetime.utcfromtimestamp(result['time']).strftime('%Y-%m-%dT%H:%M:%SZ'))
+cols.append(result.get('id'))

Review Comment:
   Gotcha  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] skorper commented on a diff in pull request #220: cdmssubset bugfix

[GitBox]

skorper commented on code in PR #220:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/220#discussion_r1040162217


##
analysis/webservice/algorithms/doms/subsetter.py:
##
@@ -309,16 +300,21 @@ def toCsv(self):
 ]
 data_variables = list(set([keys for result in results for keys in 
result['data'].keys()]))
 data_variables.sort()
+
+if 'id' in list(set([keys for result in results for keys in 
result.keys()])):
+headers.append('id')
+
 headers.extend(data_variables)
 for i, result in enumerate(results):
 cols = []
 
 cols.append(result['longitude'])
 cols.append(result['latitude'])
 
cols.append(datetime.utcfromtimestamp(result['time']).strftime('%Y-%m-%dT%H:%M:%SZ'))
+cols.append(result.get('id'))

Review Comment:
   Thanks Riley, I actually made that change locally but never pushed it! I 
split the id from the default header because there are some cases where "id" is 
not present and some where there aren't



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #220: cdmssubset bugfix

[GitBox]

RKuttruff commented on code in PR #220:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/220#discussion_r1040150796


##
analysis/webservice/algorithms/doms/subsetter.py:
##
@@ -309,16 +300,21 @@ def toCsv(self):
 ]
 data_variables = list(set([keys for result in results for keys in 
result['data'].keys()]))
 data_variables.sort()
+
+if 'id' in list(set([keys for result in results for keys in 
result.keys()])):
+headers.append('id')
+
 headers.extend(data_variables)
 for i, result in enumerate(results):
 cols = []
 
 cols.append(result['longitude'])
 cols.append(result['latitude'])
 
cols.append(datetime.utcfromtimestamp(result['time']).strftime('%Y-%m-%dT%H:%M:%SZ'))
+cols.append(result.get('id'))

Review Comment:
   I think this should only be called if the id header was added. Otherwise I'm 
seeing csv output with 4 header fields and 5 data fields which is being 
incorrectly parsed by pandas
   
   Eg:
   ```
   longitude,latitude,time,sea_surface_foundation_temperature
   172.625,-29.875,2018-09-29T09:00:00Z,None,19.509003
   ...
   ```
   Parsed as:
   ```
   >>> csv_data
longitude  latitude  time  
sea_surface_foundation_temperature
   172.625-29.875  2018-09-29T09:00:00Z  None   
19.509003
   172.875-29.875  2018-09-29T09:00:00Z  None   
18.703003
   173.125-29.875  2018-09-29T09:00:00Z  None   
18.308990
   173.375-29.875  2018-09-29T09:00:00Z  None   
18.298004
   173.625-29.875  2018-09-29T09:00:00Z  None   
18.329010
   ......   ...   ...   
  ...
   171.375-25.125  2018-09-24T09:00:00Z  None   
21.669006
   171.625-25.125  2018-09-24T09:00:00Z  None   
21.360992
   171.875-25.125  2018-09-24T09:00:00Z  None   
21.075012
   172.125-25.125  2018-09-24T09:00:00Z  None   
20.730011
   172.375-25.125  2018-09-24T09:00:00Z  None   
20.737000
   
   [9600 rows x 4 columns]
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-ingester] RKuttruff commented on pull request #65: Added ASF header to all files missing it in this repo

[GitBox]

RKuttruff commented on PR #65:
URL: 
https://github.com/apache/incubator-sdap-ingester/pull/65#issuecomment-1338152197

   Think I got all files that were missing headers.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] skorper commented on pull request #220: cdmssubset bugfix

[GitBox]

skorper commented on PR #220:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/220#issuecomment-1335870590

   Update: Added insitu uid to cdmssubset request as requested by Amanda. I 
just used the entire `meta` field for now, which includes the uid in the icoads 
case. 
   
   Example: [ICOADS Release 
3.0.csv](https://github.com/apache/incubator-sdap-nexus/files/10144539/ICOADS.Release.3.0.csv)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] skorper opened a new pull request, #220: cdmssubset bugfix

[GitBox]

skorper opened a new pull request, #220:
URL: https://github.com/apache/incubator-sdap-nexus/pull/220

   - Removed int requirement from list of platforms
   - Made parameter optional
   - Updated openapi spec
   - Fixed issue in subsetter when multiple platforms are matched. Since this 
matches with different science variables, it's possible not every science 
variable will have a value for each row. 
   
   Tested with:
   
   ```bash
   /cdmssubset?insitu=ICOADS Release 
3.0=2018-10-21T00:00:00Z=2018-10-22T01:00:00Z=-30,15,-45,30=0=5=0,16,17,30,41,42=ZIP
   ```
   
   Result looks as expected. 
[download.zip](https://github.com/apache/incubator-sdap-nexus/files/10136100/download.zip)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff opened a new pull request, #219: Added missing ASF headers to all .py files

[GitBox]

RKuttruff opened a new pull request, #219:
URL: https://github.com/apache/incubator-sdap-nexus/pull/219

   There may still be missing headers in other source files that I haven't 
checked yet


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] ngachung merged pull request #214: Build doc

[GitBox]

ngachung merged PR #214:
URL: https://github.com/apache/incubator-sdap-nexus/pull/214


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] ngachung merged pull request #215: Label changelog for v1.0.0 SDAP release

[GitBox]

ngachung merged PR #215:
URL: https://github.com/apache/incubator-sdap-nexus/pull/215


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] ngachung merged pull request #218: SDAP-416: Improve speed of install of nexus-analysis package

[GitBox]

ngachung merged PR #218:
URL: https://github.com/apache/incubator-sdap-nexus/pull/218


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff opened a new pull request, #218: SDAP-416: Improve speed of install of nexus-analysis package

[GitBox]

RKuttruff opened a new pull request, #218:
URL: https://github.com/apache/incubator-sdap-nexus/pull/218

   Used mamba to install nexus-analysis packages.
   
   This change improved the build time for webapp Docker image to ~400 seconds.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #214: Build doc

[GitBox]

RKuttruff commented on code in PR #214:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/214#discussion_r1036422362


##
docs/build.rst:
##
@@ -0,0 +1,133 @@
+.. _build:
+
+*
+How to Build the SDAP Docker Images
+*
+
+In this guide, we will go over how to build the docker images for the various 
components of SDAP.
+
+Prepare
+===
+
+First, we must ensure we have Docker installed and running. For this guide, we 
used Docker Desktop version 4.12.0. Download Docker Desktop `here. 
`_
+
+Now we must download and extract the source code for NEXUS and the ingester.
+
+.. code-block:: bash
+
+  export NEXUS_DIR=~/sdap-build/nexus
+  export INGESTER_DIR=~/sdap-build/ingester
+
+  mkdir -p ${NEXUS_DIR}
+  mkdir -p ${INGESTER_DIR}
+
+We should also set variables for a consistent tag across all images. These 
should be consistent with the release versions we plan to build.
+
+.. code-block:: bash
+
+  export NEXUS_VERSION=1.0.0
+  export INGESTER_VERSION=1.0.0
+
+To build the necessary images, we will need to first download the latest 
releases for SDAP NEXUS and SDAP Ingester ((URL)). We will not need to download 
the SDAP Nexusproto release directly.
+
+Now we must extract the releases to their respective directories.
+
+.. note::
+
+  In the following code block, make sure the versions in the filenames 
correspond with the versions that were downloaded.
+
+.. code-block:: bash
+
+  tar xvf apache-sdap-ingester-${INGESTER_VERSION}-src.tar.gz -C 
${INGESTER_DIR}
+  tar xvf apache-sdap-nexus-${NEXUS_VERSION}-src.tar.gz -C ${NEXUS_DIR}
+
+Build Ingester Components
+=
+
+There are two components to build: the Collection Manager & Granule Ingester.
+
+For both of these, we must first move to the ingester root directory.
+
+.. code-block:: bash
+
+  cd ${INGESTER_DIR}
+
+Build the Collection Manager
+---
+
+From the ingester root directory, run:
+
+.. code-block:: bash
+
+  docker build . -f collection_manager/docker/Dockerfile -t 
sdap/collection-manager:${INGESTER_VERSION}
+
+Build the Granule Ingester
+---
+
+From the ingester root directory, run:
+
+.. code-block:: bash
+
+  docker build . -f granule_ingester/docker/Dockerfile -t 
sdap/granule-ingester:${INGESTER_VERSION}
+
+Build the Solr & Webapp Components
+==
+
+For the remaining three components, we must now move to the nexus root 
directory.
+
+.. code-block:: bash
+
+  cd ${NEXUS_DIR}
+
+Build the Solr Image
+---
+
+First we must move to the Solr Docker directory.
+
+.. code-block:: bash
+
+  cd docker/solr
+
+Now to build the image:
+
+.. code-block:: bash
+
+  docker build . -t sdap/solr:${NEXUS_VERSION}

Review Comment:
   Or I can make it a variable in the quickstart
   ie: 
   `export REPO=apache/sdap-` for official
   vs
   `export REPO=sdap/` for local
   
   And then refer to the images with something like 
`${REPO}solr-cloud:${SOLR_VERSION }`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #214: Build doc

[GitBox]

RKuttruff commented on code in PR #214:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/214#discussion_r1036422362


##
docs/build.rst:
##
@@ -0,0 +1,133 @@
+.. _build:
+
+*
+How to Build the SDAP Docker Images
+*
+
+In this guide, we will go over how to build the docker images for the various 
components of SDAP.
+
+Prepare
+===
+
+First, we must ensure we have Docker installed and running. For this guide, we 
used Docker Desktop version 4.12.0. Download Docker Desktop `here. 
`_
+
+Now we must download and extract the source code for NEXUS and the ingester.
+
+.. code-block:: bash
+
+  export NEXUS_DIR=~/sdap-build/nexus
+  export INGESTER_DIR=~/sdap-build/ingester
+
+  mkdir -p ${NEXUS_DIR}
+  mkdir -p ${INGESTER_DIR}
+
+We should also set variables for a consistent tag across all images. These 
should be consistent with the release versions we plan to build.
+
+.. code-block:: bash
+
+  export NEXUS_VERSION=1.0.0
+  export INGESTER_VERSION=1.0.0
+
+To build the necessary images, we will need to first download the latest 
releases for SDAP NEXUS and SDAP Ingester ((URL)). We will not need to download 
the SDAP Nexusproto release directly.
+
+Now we must extract the releases to their respective directories.
+
+.. note::
+
+  In the following code block, make sure the versions in the filenames 
correspond with the versions that were downloaded.
+
+.. code-block:: bash
+
+  tar xvf apache-sdap-ingester-${INGESTER_VERSION}-src.tar.gz -C 
${INGESTER_DIR}
+  tar xvf apache-sdap-nexus-${NEXUS_VERSION}-src.tar.gz -C ${NEXUS_DIR}
+
+Build Ingester Components
+=
+
+There are two components to build: the Collection Manager & Granule Ingester.
+
+For both of these, we must first move to the ingester root directory.
+
+.. code-block:: bash
+
+  cd ${INGESTER_DIR}
+
+Build the Collection Manager
+---
+
+From the ingester root directory, run:
+
+.. code-block:: bash
+
+  docker build . -f collection_manager/docker/Dockerfile -t 
sdap/collection-manager:${INGESTER_VERSION}
+
+Build the Granule Ingester
+---
+
+From the ingester root directory, run:
+
+.. code-block:: bash
+
+  docker build . -f granule_ingester/docker/Dockerfile -t 
sdap/granule-ingester:${INGESTER_VERSION}
+
+Build the Solr & Webapp Components
+==
+
+For the remaining three components, we must now move to the nexus root 
directory.
+
+.. code-block:: bash
+
+  cd ${NEXUS_DIR}
+
+Build the Solr Image
+---
+
+First we must move to the Solr Docker directory.
+
+.. code-block:: bash
+
+  cd docker/solr
+
+Now to build the image:
+
+.. code-block:: bash
+
+  docker build . -t sdap/solr:${NEXUS_VERSION}

Review Comment:
   Or I can make it a variable in the quickstart
   ie: 
   `export REPO=apache/sdap-` for official
   vs
   `export REPO=sdap/` for local
   
   And then refer to the images with something like `${REPO}solr-cloud`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] ngachung commented on a diff in pull request #214: Build doc

[GitBox]

ngachung commented on code in PR #214:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/214#discussion_r1036419935


##
docs/build.rst:
##
@@ -0,0 +1,133 @@
+.. _build:
+
+*
+How to Build the SDAP Docker Images
+*
+
+In this guide, we will go over how to build the docker images for the various 
components of SDAP.
+
+Prepare
+===
+
+First, we must ensure we have Docker installed and running. For this guide, we 
used Docker Desktop version 4.12.0. Download Docker Desktop `here. 
`_
+
+Now we must download and extract the source code for NEXUS and the ingester.
+
+.. code-block:: bash
+
+  export NEXUS_DIR=~/sdap-build/nexus
+  export INGESTER_DIR=~/sdap-build/ingester
+
+  mkdir -p ${NEXUS_DIR}
+  mkdir -p ${INGESTER_DIR}
+
+We should also set variables for a consistent tag across all images. These 
should be consistent with the release versions we plan to build.
+
+.. code-block:: bash
+
+  export NEXUS_VERSION=1.0.0
+  export INGESTER_VERSION=1.0.0
+
+To build the necessary images, we will need to first download the latest 
releases for SDAP NEXUS and SDAP Ingester ((URL)). We will not need to download 
the SDAP Nexusproto release directly.
+
+Now we must extract the releases to their respective directories.
+
+.. note::
+
+  In the following code block, make sure the versions in the filenames 
correspond with the versions that were downloaded.
+
+.. code-block:: bash
+
+  tar xvf apache-sdap-ingester-${INGESTER_VERSION}-src.tar.gz -C 
${INGESTER_DIR}
+  tar xvf apache-sdap-nexus-${NEXUS_VERSION}-src.tar.gz -C ${NEXUS_DIR}
+
+Build Ingester Components
+=
+
+There are two components to build: the Collection Manager & Granule Ingester.
+
+For both of these, we must first move to the ingester root directory.
+
+.. code-block:: bash
+
+  cd ${INGESTER_DIR}
+
+Build the Collection Manager
+---
+
+From the ingester root directory, run:
+
+.. code-block:: bash
+
+  docker build . -f collection_manager/docker/Dockerfile -t 
sdap/collection-manager:${INGESTER_VERSION}
+
+Build the Granule Ingester
+---
+
+From the ingester root directory, run:
+
+.. code-block:: bash
+
+  docker build . -f granule_ingester/docker/Dockerfile -t 
sdap/granule-ingester:${INGESTER_VERSION}
+
+Build the Solr & Webapp Components
+==
+
+For the remaining three components, we must now move to the nexus root 
directory.
+
+.. code-block:: bash
+
+  cd ${NEXUS_DIR}
+
+Build the Solr Image
+---
+
+First we must move to the Solr Docker directory.
+
+.. code-block:: bash
+
+  cd docker/solr
+
+Now to build the image:
+
+.. code-block:: bash
+
+  docker build . -t sdap/solr:${NEXUS_VERSION}

Review Comment:
   True, how about local/sdap-nexus-webapp to distinguish between pulling 
official image from docker hub?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #214: Build doc

[GitBox]

RKuttruff commented on code in PR #214:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/214#discussion_r1036417885


##
docs/build.rst:
##
@@ -0,0 +1,133 @@
+.. _build:
+
+*
+How to Build the SDAP Docker Images
+*
+
+In this guide, we will go over how to build the docker images for the various 
components of SDAP.
+
+Prepare
+===
+
+First, we must ensure we have Docker installed and running. For this guide, we 
used Docker Desktop version 4.12.0. Download Docker Desktop `here. 
`_
+
+Now we must download and extract the source code for NEXUS and the ingester.
+
+.. code-block:: bash
+
+  export NEXUS_DIR=~/sdap-build/nexus
+  export INGESTER_DIR=~/sdap-build/ingester
+
+  mkdir -p ${NEXUS_DIR}
+  mkdir -p ${INGESTER_DIR}
+
+We should also set variables for a consistent tag across all images. These 
should be consistent with the release versions we plan to build.
+
+.. code-block:: bash
+
+  export NEXUS_VERSION=1.0.0
+  export INGESTER_VERSION=1.0.0
+
+To build the necessary images, we will need to first download the latest 
releases for SDAP NEXUS and SDAP Ingester ((URL)). We will not need to download 
the SDAP Nexusproto release directly.
+
+Now we must extract the releases to their respective directories.
+
+.. note::
+
+  In the following code block, make sure the versions in the filenames 
correspond with the versions that were downloaded.
+
+.. code-block:: bash
+
+  tar xvf apache-sdap-ingester-${INGESTER_VERSION}-src.tar.gz -C 
${INGESTER_DIR}
+  tar xvf apache-sdap-nexus-${NEXUS_VERSION}-src.tar.gz -C ${NEXUS_DIR}
+
+Build Ingester Components
+=
+
+There are two components to build: the Collection Manager & Granule Ingester.
+
+For both of these, we must first move to the ingester root directory.
+
+.. code-block:: bash
+
+  cd ${INGESTER_DIR}
+
+Build the Collection Manager
+---
+
+From the ingester root directory, run:
+
+.. code-block:: bash
+
+  docker build . -f collection_manager/docker/Dockerfile -t 
sdap/collection-manager:${INGESTER_VERSION}
+
+Build the Granule Ingester
+---
+
+From the ingester root directory, run:
+
+.. code-block:: bash
+
+  docker build . -f granule_ingester/docker/Dockerfile -t 
sdap/granule-ingester:${INGESTER_VERSION}
+
+Build the Solr & Webapp Components
+==
+
+For the remaining three components, we must now move to the nexus root 
directory.
+
+.. code-block:: bash
+
+  cd ${NEXUS_DIR}
+
+Build the Solr Image
+---
+
+First we must move to the Solr Docker directory.
+
+.. code-block:: bash
+
+  cd docker/solr
+
+Now to build the image:
+
+.. code-block:: bash
+
+  docker build . -t sdap/solr:${NEXUS_VERSION}

Review Comment:
   Though it would be a bit easier to use with the qs since you'd only have to 
change the repo instead of repo + image



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] ngachung commented on a diff in pull request #214: Build doc

[GitBox]

ngachung commented on code in PR #214:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/214#discussion_r1036413641


##
docs/build.rst:
##
@@ -0,0 +1,133 @@
+.. _build:
+
+*
+How to Build the SDAP Docker Images
+*
+
+In this guide, we will go over how to build the docker images for the various 
components of SDAP.
+
+Prepare
+===
+
+First, we must ensure we have Docker installed and running. For this guide, we 
used Docker Desktop version 4.12.0. Download Docker Desktop `here. 
`_
+
+Now we must download and extract the source code for NEXUS and the ingester.
+
+.. code-block:: bash
+
+  export NEXUS_DIR=~/sdap-build/nexus
+  export INGESTER_DIR=~/sdap-build/ingester
+
+  mkdir -p ${NEXUS_DIR}
+  mkdir -p ${INGESTER_DIR}
+
+We should also set variables for a consistent tag across all images. These 
should be consistent with the release versions we plan to build.
+
+.. code-block:: bash
+
+  export NEXUS_VERSION=1.0.0
+  export INGESTER_VERSION=1.0.0
+
+To build the necessary images, we will need to first download the latest 
releases for SDAP NEXUS and SDAP Ingester ((URL)). We will not need to download 
the SDAP Nexusproto release directly.
+
+Now we must extract the releases to their respective directories.
+
+.. note::
+
+  In the following code block, make sure the versions in the filenames 
correspond with the versions that were downloaded.
+
+.. code-block:: bash
+
+  tar xvf apache-sdap-ingester-${INGESTER_VERSION}-src.tar.gz -C 
${INGESTER_DIR}
+  tar xvf apache-sdap-nexus-${NEXUS_VERSION}-src.tar.gz -C ${NEXUS_DIR}
+
+Build Ingester Components
+=
+
+There are two components to build: the Collection Manager & Granule Ingester.
+
+For both of these, we must first move to the ingester root directory.
+
+.. code-block:: bash
+
+  cd ${INGESTER_DIR}
+
+Build the Collection Manager
+---
+
+From the ingester root directory, run:
+
+.. code-block:: bash
+
+  docker build . -f collection_manager/docker/Dockerfile -t 
sdap/collection-manager:${INGESTER_VERSION}
+
+Build the Granule Ingester
+---
+
+From the ingester root directory, run:
+
+.. code-block:: bash
+
+  docker build . -f granule_ingester/docker/Dockerfile -t 
sdap/granule-ingester:${INGESTER_VERSION}
+
+Build the Solr & Webapp Components
+==
+
+For the remaining three components, we must now move to the nexus root 
directory.
+
+.. code-block:: bash
+
+  cd ${NEXUS_DIR}
+
+Build the Solr Image
+---
+
+First we must move to the Solr Docker directory.
+
+.. code-block:: bash
+
+  cd docker/solr
+
+Now to build the image:
+
+.. code-block:: bash
+
+  docker build . -t sdap/solr:${NEXUS_VERSION}

Review Comment:
   Maybe not since we're using sdap/ whereas in the apache official images we 
have apache/sdap-



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #214: Build doc

[GitBox]

RKuttruff commented on code in PR #214:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/214#discussion_r1036400851


##
docs/build.rst:
##
@@ -0,0 +1,133 @@
+.. _build:
+
+*
+How to Build the SDAP Docker Images
+*
+
+In this guide, we will go over how to build the docker images for the various 
components of SDAP.
+
+Prepare
+===
+
+First, we must ensure we have Docker installed and running. For this guide, we 
used Docker Desktop version 4.12.0. Download Docker Desktop `here. 
`_
+
+Now we must download and extract the source code for NEXUS and the ingester.
+
+.. code-block:: bash
+
+  export NEXUS_DIR=~/sdap-build/nexus
+  export INGESTER_DIR=~/sdap-build/ingester
+
+  mkdir -p ${NEXUS_DIR}
+  mkdir -p ${INGESTER_DIR}
+
+We should also set variables for a consistent tag across all images. These 
should be consistent with the release versions we plan to build.
+
+.. code-block:: bash
+
+  export NEXUS_VERSION=1.0.0
+  export INGESTER_VERSION=1.0.0
+
+To build the necessary images, we will need to first download the latest 
releases for SDAP NEXUS and SDAP Ingester ((URL)). We will not need to download 
the SDAP Nexusproto release directly.
+
+Now we must extract the releases to their respective directories.
+
+.. note::
+
+  In the following code block, make sure the versions in the filenames 
correspond with the versions that were downloaded.
+
+.. code-block:: bash
+
+  tar xvf apache-sdap-ingester-${INGESTER_VERSION}-src.tar.gz -C 
${INGESTER_DIR}
+  tar xvf apache-sdap-nexus-${NEXUS_VERSION}-src.tar.gz -C ${NEXUS_DIR}
+
+Build Ingester Components
+=
+
+There are two components to build: the Collection Manager & Granule Ingester.
+
+For both of these, we must first move to the ingester root directory.
+
+.. code-block:: bash
+
+  cd ${INGESTER_DIR}
+
+Build the Collection Manager
+---
+
+From the ingester root directory, run:
+
+.. code-block:: bash
+
+  docker build . -f collection_manager/docker/Dockerfile -t 
sdap/collection-manager:${INGESTER_VERSION}
+
+Build the Granule Ingester
+---
+
+From the ingester root directory, run:
+
+.. code-block:: bash
+
+  docker build . -f granule_ingester/docker/Dockerfile -t 
sdap/granule-ingester:${INGESTER_VERSION}
+
+Build the Solr & Webapp Components
+==
+
+For the remaining three components, we must now move to the nexus root 
directory.
+
+.. code-block:: bash
+
+  cd ${NEXUS_DIR}
+
+Build the Solr Image
+---
+
+First we must move to the Solr Docker directory.
+
+.. code-block:: bash
+
+  cd docker/solr
+
+Now to build the image:
+
+.. code-block:: bash
+
+  docker build . -t sdap/solr:${NEXUS_VERSION}

Review Comment:
   Should I prefix all the images with `sdap-` as well?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] ngachung commented on a diff in pull request #214: Build doc

[GitBox]

ngachung commented on code in PR #214:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/214#discussion_r1036278011


##
docs/build.rst:
##
@@ -0,0 +1,133 @@
+.. _build:
+
+*
+How to Build the SDAP Docker Images
+*
+
+In this guide, we will go over how to build the docker images for the various 
components of SDAP.
+
+Prepare
+===
+
+First, we must ensure we have Docker installed and running. For this guide, we 
used Docker Desktop version 4.12.0. Download Docker Desktop `here. 
`_
+
+Now we must download and extract the source code for NEXUS and the ingester.
+
+.. code-block:: bash
+
+  export NEXUS_DIR=~/sdap-build/nexus
+  export INGESTER_DIR=~/sdap-build/ingester
+
+  mkdir -p ${NEXUS_DIR}
+  mkdir -p ${INGESTER_DIR}
+
+We should also set variables for a consistent tag across all images. These 
should be consistent with the release versions we plan to build.
+
+.. code-block:: bash
+
+  export NEXUS_VERSION=1.0.0
+  export INGESTER_VERSION=1.0.0
+
+To build the necessary images, we will need to first download the latest 
releases for SDAP NEXUS and SDAP Ingester ((URL)). We will not need to download 
the SDAP Nexusproto release directly.
+
+Now we must extract the releases to their respective directories.
+
+.. note::
+
+  In the following code block, make sure the versions in the filenames 
correspond with the versions that were downloaded.
+
+.. code-block:: bash
+
+  tar xvf apache-sdap-ingester-${INGESTER_VERSION}-src.tar.gz -C 
${INGESTER_DIR}
+  tar xvf apache-sdap-nexus-${NEXUS_VERSION}-src.tar.gz -C ${NEXUS_DIR}
+
+Build Ingester Components
+=
+
+There are two components to build: the Collection Manager & Granule Ingester.
+
+For both of these, we must first move to the ingester root directory.
+
+.. code-block:: bash
+
+  cd ${INGESTER_DIR}
+
+Build the Collection Manager
+---
+
+From the ingester root directory, run:
+
+.. code-block:: bash
+
+  docker build . -f collection_manager/docker/Dockerfile -t 
sdap/collection-manager:${INGESTER_VERSION}
+
+Build the Granule Ingester
+---
+
+From the ingester root directory, run:
+
+.. code-block:: bash
+
+  docker build . -f granule_ingester/docker/Dockerfile -t 
sdap/granule-ingester:${INGESTER_VERSION}
+
+Build the Solr & Webapp Components
+==
+
+For the remaining three components, we must now move to the nexus root 
directory.
+
+.. code-block:: bash
+
+  cd ${NEXUS_DIR}
+
+Build the Solr Image
+---
+
+First we must move to the Solr Docker directory.
+
+.. code-block:: bash
+
+  cd docker/solr
+
+Now to build the image:
+
+.. code-block:: bash
+
+  docker build . -t sdap/solr:${NEXUS_VERSION}

Review Comment:
   For consistency with public docker images, this should be named 
sdap/solr-cloud



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-ingester] ngachung merged pull request #63: Label changelog for v1.0.0 SDAP release

[GitBox]

ngachung merged PR #63:
URL: https://github.com/apache/incubator-sdap-ingester/pull/63


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] DendriteQ commented on pull request #217: deletebyquery: Parameter to set the number of rows to fetch from Solr

[GitBox]

DendriteQ commented on PR #217:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/217#issuecomment-1330741575

   This deal seams quick fixed and from one trigger to a solution. Just for
   letting out some steam. I actually had enough of you guys andcan pick and
   choose. So i am not consenting to that price on that information. Heard
   rumors that thats it a rhino.
   
   On Tue, Nov 29, 2022, 00:45 Riley Kuttruff ***@***.***> wrote:
   
   > --
   > You can view, comment on, or merge this pull request online at:
   >
   >   https://github.com/apache/incubator-sdap-nexus/pull/217
   > Commit Summary
   >
   >- 87075f3
   >

   >Parameter to set the number of rows to fetch from Solr
   >
   > File Changes
   >
   > (2 files )
   >
   >- *M* CHANGELOG.md
   >

   >(1)
   >- *M* tools/deletebyquery/deletebyquery.py
   >

   >(9)
   >
   > Patch Links:
   >
   >- https://github.com/apache/incubator-sdap-nexus/pull/217.patch
   >- https://github.com/apache/incubator-sdap-nexus/pull/217.diff
   >
   > —
   > Reply to this email directly, view it on GitHub
   > , or unsubscribe
   > 

   > .
   > You are receiving this because you are subscribed to this thread.Message
   > ID: ***@***.***>
   >
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-ingester] ngachung merged pull request #64: Fixed bug in versioning for granule ingester

[GitBox]

ngachung merged PR #64:
URL: https://github.com/apache/incubator-sdap-ingester/pull/64


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-ingester] RKuttruff opened a new pull request, #64: Fixed bug in versioning for granule ingester

[GitBox]

RKuttruff opened a new pull request, #64:
URL: https://github.com/apache/incubator-sdap-ingester/pull/64

   Version string from `VERSION.txt` was being overwritten


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff opened a new pull request, #216: SDAP 412 - Potential Solution to Duplicate Primary Issue in /match_spark Endpoint

[GitBox]

RKuttruff opened a new pull request, #216:
URL: https://github.com/apache/incubator-sdap-nexus/pull/216

   Prevents duplicate primary points from appearing in result of matchup by 
implementing object equality for `DomsPoint` class such that different 
`DomsPoint` objects that happen to correspond to the same primary point are 
merged in the `combineByKey` step.
   
   This PR does not find/solve the root cause of this issue but does prevent it 
from effecting the final results.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-ingester] skorper merged pull request #61: SDAP-408 - Improvements to ingestion

[GitBox]

skorper merged PR #61:
URL: https://github.com/apache/incubator-sdap-ingester/pull/61


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexusproto] ngachung merged pull request #11: Updated development status to production/stable in setup.py

[GitBox]

ngachung merged PR #11:
URL: https://github.com/apache/incubator-sdap-nexusproto/pull/11


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-ingester] RKuttruff commented on a diff in pull request #61: SDAP-408 - Improvements to ingestion

[GitBox]

RKuttruff commented on code in PR #61:
URL: 
https://github.com/apache/incubator-sdap-ingester/pull/61#discussion_r1029667525


##
granule_ingester/granule_ingester/pipeline/Pipeline.py:
##
@@ -48,36 +53,56 @@
 
 
 def _init_worker(processor_list, dataset, data_store_factory, 
metadata_store_factory, shared_memory):
-global _worker_data_store
-global _worker_metadata_store
 global _worker_processor_list
 global _worker_dataset
 global _shared_memory
 
 # _worker_data_store and _worker_metadata_store open multiple TCP sockets 
from each worker process;
 # however, these sockets will be automatically closed by the OS once the 
worker processes die so no need to worry.
-_worker_data_store = data_store_factory()
-_worker_metadata_store = metadata_store_factory()
 _worker_processor_list = processor_list
 _worker_dataset = dataset
 _shared_memory = shared_memory
 
+logger.debug("worker init")
 
 async def _process_tile_in_worker(serialized_input_tile: str):
 try:
+logger.info('Starting tile creation subprocess')
 logger.debug(f'serialized_input_tile: {serialized_input_tile}')
 input_tile = nexusproto.NexusTile.FromString(serialized_input_tile)
-logger.debug(f'_recurse params: _worker_processor_list = 
{_worker_processor_list}, _worker_dataset = {_worker_dataset}, input_tile = 
{input_tile}')
-processed_tile = _recurse(_worker_processor_list, _worker_dataset, 
input_tile)
+# logger.debug(f'_recurse params: _worker_processor_list = 
{_worker_processor_list}, _worker_dataset = {_worker_dataset}, input_tile = 
{input_tile}')

Review Comment:
   Sure.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-ingester] skorper commented on a diff in pull request #61: SDAP-408 - Improvements to ingestion

[GitBox]

skorper commented on code in PR #61:
URL: 
https://github.com/apache/incubator-sdap-ingester/pull/61#discussion_r1029663766


##
granule_ingester/granule_ingester/pipeline/Pipeline.py:
##
@@ -48,36 +53,56 @@
 
 
 def _init_worker(processor_list, dataset, data_store_factory, 
metadata_store_factory, shared_memory):
-global _worker_data_store
-global _worker_metadata_store
 global _worker_processor_list
 global _worker_dataset
 global _shared_memory
 
 # _worker_data_store and _worker_metadata_store open multiple TCP sockets 
from each worker process;
 # however, these sockets will be automatically closed by the OS once the 
worker processes die so no need to worry.
-_worker_data_store = data_store_factory()
-_worker_metadata_store = metadata_store_factory()
 _worker_processor_list = processor_list
 _worker_dataset = dataset
 _shared_memory = shared_memory
 
+logger.debug("worker init")
 
 async def _process_tile_in_worker(serialized_input_tile: str):
 try:
+logger.info('Starting tile creation subprocess')
 logger.debug(f'serialized_input_tile: {serialized_input_tile}')
 input_tile = nexusproto.NexusTile.FromString(serialized_input_tile)
-logger.debug(f'_recurse params: _worker_processor_list = 
{_worker_processor_list}, _worker_dataset = {_worker_dataset}, input_tile = 
{input_tile}')
-processed_tile = _recurse(_worker_processor_list, _worker_dataset, 
input_tile)
+# logger.debug(f'_recurse params: _worker_processor_list = 
{_worker_processor_list}, _worker_dataset = {_worker_dataset}, input_tile = 
{input_tile}')

Review Comment:
   Can you remove this line? There's a similarly commented out line above we 
should also remove



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexusproto] ngachung opened a new pull request, #11: Updated development status to production/stable in setup.py

[GitBox]

ngachung opened a new pull request, #11:
URL: https://github.com/apache/incubator-sdap-nexusproto/pull/11

   Cannot push nexusproto to pypi if we leave `Development Status :: 1 - 
Pre-Alpha`. Get this error message
   
   `Invalid value for classifiers. Error: Classifier 'Development Status :: 1 - 
Pre-Alpha' is not a valid classifier.`
   
   Since we're working on releasing version 1.0.0, update to `Development 
Status :: 5 - Production/Stable`
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff opened a new pull request, #214: Build doc

[GitBox]

RKuttruff opened a new pull request, #214:
URL: https://github.com/apache/incubator-sdap-nexus/pull/214

   Documentation for building the Docker images used by SDAP.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-ingester] ngachung merged pull request #62: SDAP-411: Used global versioning for SDAP ingester artifacts

[GitBox]

ngachung merged PR #62:
URL: https://github.com/apache/incubator-sdap-ingester/pull/62


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] ngachung merged pull request #213: SDAP-411: Used global versioning for SDAP NEXUS artifacts

[GitBox]

ngachung merged PR #213:
URL: https://github.com/apache/incubator-sdap-nexus/pull/213


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-in-situ-data-services] dependabot[bot] closed pull request #8: chore(deps): bump pyspark from 3.1.2 to 3.1.3

[GitBox]

dependabot[bot] closed pull request #8: chore(deps): bump pyspark from 3.1.2 to 
3.1.3
URL: https://github.com/apache/incubator-sdap-in-situ-data-services/pull/8


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-in-situ-data-services] dependabot[bot] opened a new pull request, #10: chore(deps): bump pyspark from 3.1.2 to 3.2.2

[GitBox]

dependabot[bot] opened a new pull request, #10:
URL: https://github.com/apache/incubator-sdap-in-situ-data-services/pull/10

   Bumps [pyspark](https://github.com/apache/spark) from 3.1.2 to 3.2.2.
   
   Commits
   
   https://github.com/apache/spark/commit/78a5825fe266c0884d2dd18cbca9625fa258d7f7;>78a5825
 Preparing Spark release v3.2.2-rc1
   https://github.com/apache/spark/commit/ba978b3c533f202c6203ac514261655e9cee0cff;>ba978b3
 [SPARK-39099][BUILD] Add dependencies to Dockerfile for building Spark 
releases
   https://github.com/apache/spark/commit/001d8b0cddcec46a44e7c6e31612dc2baada05d5;>001d8b0
 [SPARK-37554][BUILD] Add PyArrow, pandas and plotly to release Docker image 
d...
   https://github.com/apache/spark/commit/9dd4c07475c82f922c29d67a4db4bb42676c5c07;>9dd4c07
 [SPARK-37730][PYTHON][FOLLOWUP] Split comments to comply pycodestyle check
   https://github.com/apache/spark/commit/bc54a3f0c2e08893702c3929bfe7a9d543a08cdb;>bc54a3f
 [SPARK-37730][PYTHON] Replace use of MPLPlot._add_legend_handle with 
MPLPlot
   https://github.com/apache/spark/commit/c5983c1691f20590abf80b17bdc029b584b89521;>c5983c1
 [SPARK-38018][SQL][3.2] Fix ColumnVectorUtils.populate to handle 
CalendarInte...
   https://github.com/apache/spark/commit/32aff86477ac001b0ee047db08591d89e90c6eb8;>32aff86
 [SPARK-39447][SQL][3.2] Avoid AssertionError in 
AdaptiveSparkPlanExec.doExecu...
   https://github.com/apache/spark/commit/be891ad99083564a7bf7f421e00b2cc4759a679f;>be891ad
 [SPARK-39551][SQL][3.2] Add AQE invalid plan check
   https://github.com/apache/spark/commit/1c0bd4c15a28d7c6a2dca846a5b8d0eb1d152aae;>1c0bd4c
 [SPARK-39656][SQL][3.2] Fix wrong namespace in DescribeNamespaceExec
   https://github.com/apache/spark/commit/3d084fe3217bea9af4c544f10ead8a2e5b97dad4;>3d084fe
 [SPARK-39677][SQL][DOCS][3.2] Fix args formatting of the regexp and like 
func...
   Additional commits viewable in https://github.com/apache/spark/compare/v3.1.2...v3.2.2;>compare 
view
   
   
   
   
   
   [![Dependabot compatibility 
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=pyspark=pip=3.1.2=3.2.2)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
   
   Dependabot will resolve any conflicts with this PR as long as you don't 
alter it yourself. You can also trigger a rebase manually by commenting 
`@dependabot rebase`.
   
   [//]: # (dependabot-automerge-start)
   [//]: # (dependabot-automerge-end)
   
   ---
   
   
   Dependabot commands and options
   
   
   You can trigger Dependabot actions by commenting on this PR:
   - `@dependabot rebase` will rebase this PR
   - `@dependabot recreate` will recreate this PR, overwriting any edits that 
have been made to it
   - `@dependabot merge` will merge this PR after your CI passes on it
   - `@dependabot squash and merge` will squash and merge this PR after your CI 
passes on it
   - `@dependabot cancel merge` will cancel a previously requested merge and 
block automerging
   - `@dependabot reopen` will reopen this PR if it is closed
   - `@dependabot close` will close this PR and stop Dependabot recreating it. 
You can achieve the same result by closing it manually
   - `@dependabot ignore this major version` will close this PR and stop 
Dependabot creating any more for this major version (unless you reopen the PR 
or upgrade to it yourself)
   - `@dependabot ignore this minor version` will close this PR and stop 
Dependabot creating any more for this minor version (unless you reopen the PR 
or upgrade to it yourself)
   - `@dependabot ignore this dependency` will close this PR and stop 
Dependabot creating any more for this dependency (unless you reopen the PR or 
upgrade to it yourself)
   - `@dependabot use these labels` will set the current labels as the default 
for future PRs for this repo and language
   - `@dependabot use these reviewers` will set the current reviewers as the 
default for future PRs for this repo and language
   - `@dependabot use these assignees` will set the current assignees as the 
default for future PRs for this repo and language
   - `@dependabot use this milestone` will set the current milestone as the 
default for future PRs for this repo and language
   
   You can disable automated security fix PRs for this repo from the [Security 
Alerts 
page](https://github.com/apache/incubator-sdap-in-situ-data-services/network/alerts).
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-in-situ-data-services] dependabot[bot] commented on pull request #8: chore(deps): bump pyspark from 3.1.2 to 3.1.3

[GitBox]

dependabot[bot] commented on PR #8:
URL: 
https://github.com/apache/incubator-sdap-in-situ-data-services/pull/8#issuecomment-1311380642

   Superseded by #10.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-ingester] RKuttruff opened a new pull request, #61: Improvements to ingestion

[GitBox]

RKuttruff opened a new pull request, #61:
URL: https://github.com/apache/incubator-sdap-ingester/pull/61

   This PR has several improvements to the granule ingester
   
   1. Made the writes to data and metadata stores more fault tolerant. In the 
existing implementation, we write to the underlying stores very frequently. 
This seems to overburden them, and eventually a write may fail. This was 
treated as an unrecoverable loss of connection and the GI fails with the 
granule needing to be restarted from the beginning. This PR has the writes 
retry a few times on failure with some backoff time.
   2. Consolidated writes. For small tile sizes (and thus a large number of 
tiles/granule), there seems to be a lot of time spent writing the data to the 
underlying stores. This PR consolidates them into a large push after all the 
tiles have been generated, significantly reducing the number of network I/O 
operations.
   3. Optimized data subset conversion to `np.ndarray`. Initially done through 
a call to `np.ma.filled` with a `list` of `xr.DataArrays`. This is unnecessary 
because a) `np.ma.filled` is supposed to be used with masked arrays, and with 
`list` objects, just calls `np.array`; and b) there seems to be a lot on 
inefficiency with calling `np.array` with `xr.DataArrays`, especially when the 
`xr.DataArray` objects are backed by `np.ndarray` objects that can be easily 
fetched from them. This PR does just that.
   
   With these changes listed above, a benchmark ingest of 1 VIIRS granule into 
100x100 tiles went from 4580 seconds to just 163 seconds.
   
   Further testing + benchmarking coming soon.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] wphyojpl opened a new pull request, #212: fix: tiles search

[GitBox]

wphyojpl opened a new pull request, #212:
URL: https://github.com/apache/incubator-sdap-nexus/pull/212

   - re-enable tiles search endpoint
   - also fixed json encoding error for tile object


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] skorper merged pull request #211: Updated cdmssubset to reduce memory consumption

[GitBox]

skorper merged PR #211:
URL: https://github.com/apache/incubator-sdap-nexus/pull/211


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] ngachung merged pull request #210: SDAP-407: Return depth in CSV and NETCDF matchup outputs

[GitBox]

ngachung merged PR #210:
URL: https://github.com/apache/incubator-sdap-nexus/pull/210


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] ngachung commented on pull request #210: SDAP-407: Return depth in CSV and NETCDF matchup outputs

[GitBox]

ngachung commented on PR #210:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/210#issuecomment-1293762825

   This table is only used for match-up results so should be okay.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff opened a new pull request, #211: Updated cdmssubset to reduce memory consumption

[GitBox]

RKuttruff opened a new pull request, #211:
URL: https://github.com/apache/incubator-sdap-nexus/pull/211

   Potential solution to issues with subsetting L2 data crashing the webapp due 
to memory exhaustion.
   
   Verified new subsetter code with validation utility (comparing to OPeNDAP 
(L4) and Harmony (L2)) and results are consistent with existing code.
   
   Additionally, I added some logging statements.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #207: Insitu query logging

[GitBox]

RKuttruff commented on code in PR #207:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/207#discussion_r1004727473


##
analysis/webservice/algorithms/doms/insitu.py:
##
@@ -60,12 +66,22 @@ def query_insitu(dataset, variable, start_time, end_time, 
bbox, platform, depth_
 # Page through all insitu results
 next_page_url = insitu_endpoints.getEndpoint(provider, dataset)
 while next_page_url is not None and next_page_url != 'NA':
-if session is not None:
-response = session.get(next_page_url, params=params)
+thetime = datetime.now()
+if params == {}:
+logging.info(f"Starting insitu request: {next_page_url}")
 else:
-response = requests.get(next_page_url, params=params)
+logging.info(f"Starting insitu request: 
{next_page_url}?{urlencode(params)}")
+
+
+try:
+if session is not None:
+response = session.get(next_page_url, params=params, 
timeout=(15.05, 331))

Review Comment:
   First one is connect timeout, second is read timeout. 
   I suppose it would be beneficial to fail quickly if we can't even connect 
rather than wait a whole 5 minutes 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] skorper commented on a diff in pull request #207: Insitu query logging

[GitBox]

skorper commented on code in PR #207:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/207#discussion_r1004725621


##
analysis/webservice/algorithms/doms/insitu.py:
##
@@ -60,12 +66,22 @@ def query_insitu(dataset, variable, start_time, end_time, 
bbox, platform, depth_
 # Page through all insitu results
 next_page_url = insitu_endpoints.getEndpoint(provider, dataset)
 while next_page_url is not None and next_page_url != 'NA':
-if session is not None:
-response = session.get(next_page_url, params=params)
+thetime = datetime.now()
+if params == {}:
+logging.info(f"Starting insitu request: {next_page_url}")
 else:
-response = requests.get(next_page_url, params=params)
+logging.info(f"Starting insitu request: 
{next_page_url}?{urlencode(params)}")
+
+
+try:
+if session is not None:
+response = session.get(next_page_url, params=params, 
timeout=(15.05, 331))

Review Comment:
   I meant why set `timeout=(15.05, 331)` rather than `timeout=331`? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #207: Insitu query logging

[GitBox]

RKuttruff commented on code in PR #207:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/207#discussion_r1004721938


##
analysis/webservice/algorithms/doms/insitu.py:
##
@@ -60,12 +66,22 @@ def query_insitu(dataset, variable, start_time, end_time, 
bbox, platform, depth_
 # Page through all insitu results
 next_page_url = insitu_endpoints.getEndpoint(provider, dataset)
 while next_page_url is not None and next_page_url != 'NA':
-if session is not None:
-response = session.get(next_page_url, params=params)
+thetime = datetime.now()
+if params == {}:
+logging.info(f"Starting insitu request: {next_page_url}")
 else:
-response = requests.get(next_page_url, params=params)
+logging.info(f"Starting insitu request: 
{next_page_url}?{urlencode(params)}")
+
+
+try:
+if session is not None:
+response = session.get(next_page_url, params=params, 
timeout=(15.05, 331))

Review Comment:
   Not sure what you mean by the first question.
   
   I chose the numbers (primarily 15.05) since the documentation for setting 
timeouts recommends setting it to a little bit higher than a multiple of 3, so 
I picked one admittedly arbitrarily. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] skorper commented on a diff in pull request #207: Insitu query logging

[GitBox]

skorper commented on code in PR #207:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/207#discussion_r1004719665


##
analysis/webservice/algorithms/doms/insitu.py:
##
@@ -60,12 +66,22 @@ def query_insitu(dataset, variable, start_time, end_time, 
bbox, platform, depth_
 # Page through all insitu results
 next_page_url = insitu_endpoints.getEndpoint(provider, dataset)
 while next_page_url is not None and next_page_url != 'NA':
-if session is not None:
-response = session.get(next_page_url, params=params)
+thetime = datetime.now()
+if params == {}:
+logging.info(f"Starting insitu request: {next_page_url}")
 else:
-response = requests.get(next_page_url, params=params)
+logging.info(f"Starting insitu request: 
{next_page_url}?{urlencode(params)}")
+
+
+try:
+if session is not None:
+response = session.get(next_page_url, params=params, 
timeout=(15.05, 331))

Review Comment:
   What's the justification of setting these values separately? Also, why 331 
and not 300 (the gateway timeout we have configured)? Also curious why you 
chose 15.05
   
   Also, we should probably avoid [magic 
numbers](https://en.wikipedia.org/wiki/Magic_number_(programming)) and assign 
these values to variables



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] skorper commented on pull request #210: SDAP-407: Return depth in CSV and NETCDF matchup outputs

[GitBox]

skorper commented on PR #210:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/210#issuecomment-1289794091

   I think one question is how do we want to handle DB schema changes? I'm not 
sure how many other projects are using this table, but we should probably still 
make sure everyone knows about the DB schema update before they upgrade to this 
version.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] skorper opened a new pull request, #210: SDAP-407: Return depth in CSV and NETCDF matchup outputs

[GitBox]

skorper opened a new pull request, #210:
URL: https://github.com/apache/incubator-sdap-nexus/pull/210

   [SDAP-407](https://issues.apache.org/jira/browse/SDAP-407)
   
   Currently, depth is not being returned when the `/domsresults` endpoint is 
called. This is because depth is not stored in the database so that information 
is lost.
   
   Added depth to the database, after matchup this field is now populated. 
Updated CSV and NetCDF conversion to make sure this field is present. 
   
   Examples below. Please note the primary dataset is a satellite dataset which 
is why depth is null/not present. 
   
   Also note that I generated these by running a local instance of 
Solr/Cassandra and included massively large tolerances to make sure a match 
would occur. The matches are not valid and were only used for testing purposes. 
   
   ```sh
   
/match_spark?tt=43200=AVHRR_OI_L4_GHRSST_NCEI=ICOADS_JPL=2015-01-01T00%3A00%3A00Z=2018-04-05T00%3A00%3A00Z=25000=-140%2C10%2C-110%2C40=30=-5=5=true=9
   ```
   
   
[response.nc.txt](https://github.com/apache/incubator-sdap-nexus/files/9855618/response.nc.txt)
   
[response.csv](https://github.com/apache/incubator-sdap-nexus/files/9855620/response.csv)
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] ngachung merged pull request #185: SDAP-399: Quickstart update - Docker

[GitBox]

ngachung merged PR #185:
URL: https://github.com/apache/incubator-sdap-nexus/pull/185


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-mudrod] dependabot[bot] opened a new pull request, #56: Bump jettison from 1.3.8 to 1.5.1 in /core

[GitBox]

dependabot[bot] opened a new pull request, #56:
URL: https://github.com/apache/incubator-sdap-mudrod/pull/56

   Bumps [jettison](https://github.com/jettison-json/jettison) from 1.3.8 to 
1.5.1.
   
   Release notes
   Sourced from https://github.com/jettison-json/jettison/releases;>jettison's 
releases.
   
   Jettison 1.5.1
   What's Changed
   
   Stack Overflow fix on malformed JSON by https://github.com/coheigea;>@​coheigea in https://github-redirect.dependabot.com/jettison-json/jettison/pull/48;>jettison-json/jettison#48
   Prevent infinite loop when a /* comment is not terminated by https://github.com/coheigea;>@​coheigea in https://github-redirect.dependabot.com/jettison-json/jettison/pull/49;>jettison-json/jettison#49
   
   Full Changelog: https://github.com/jettison-json/jettison/compare/jettison-1.5.0...jettison-1.5.1;>https://github.com/jettison-json/jettison/compare/jettison-1.5.0...jettison-1.5.1
   
   
   
   Commits
   
   https://github.com/jettison-json/jettison/commit/bdb3982aaea184c2143e41981e33877c2fd66292;>bdb3982
 [maven-release-plugin] prepare release jettison-1.5.1
   https://github.com/jettison-json/jettison/commit/1268b7558bad9b989687009a094466b64d4da533;>1268b75
 Prevent infinite loop when a /* comment is not terminated
   https://github.com/jettison-json/jettison/commit/cff9f2814ec8ae573e38f38abed039311bb7bfda;>cff9f28
 Create codeql-analysis.yml
   https://github.com/jettison-json/jettison/commit/395f8625bcf688743872c8e7f59360d372e77811;>395f862
 Stack Overflow fix on malformed JSON
   https://github.com/jettison-json/jettison/commit/a5d2223bafd4c5ec37723533fa5c4066ead4eaaa;>a5d2223
 [maven-release-plugin] prepare for next development iteration
   https://github.com/jettison-json/jettison/commit/e1bf5293d10af3ebfcd8763eb51a20a659b9047c;>e1bf529
 [maven-release-plugin] prepare release jettison-1.5.0
   https://github.com/jettison-json/jettison/commit/9ccf9d00fbae12fd10863c5b78d22bac6e5b87e4;>9ccf9d0
 Update some deps/plugin versions
   https://github.com/jettison-json/jettison/commit/cfc7382c4dab4e207d2db2f0a70f60ce565e477d;>cfc7382
 Performance improvement
   https://github.com/jettison-json/jettison/commit/d3714681f61581810680df8e45858a4d30a602da;>d371468
 Use BigDecimal instead of Double to avoid loss of precision
   https://github.com/jettison-json/jettison/commit/23a64eedf97e29b6562de8c3b024af414df9b900;>23a64ee
 Avoid NullPointerException when used with Axis2 1.5
   Additional commits viewable in https://github.com/jettison-json/jettison/compare/jettison-1.3.8...jettison-1.5.1;>compare
 view
   
   
   
   
   
   [![Dependabot compatibility 
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=org.codehaus.jettison:jettison=maven=1.3.8=1.5.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
   
   Dependabot will resolve any conflicts with this PR as long as you don't 
alter it yourself. You can also trigger a rebase manually by commenting 
`@dependabot rebase`.
   
   [//]: # (dependabot-automerge-start)
   [//]: # (dependabot-automerge-end)
   
   ---
   
   
   Dependabot commands and options
   
   
   You can trigger Dependabot actions by commenting on this PR:
   - `@dependabot rebase` will rebase this PR
   - `@dependabot recreate` will recreate this PR, overwriting any edits that 
have been made to it
   - `@dependabot merge` will merge this PR after your CI passes on it
   - `@dependabot squash and merge` will squash and merge this PR after your CI 
passes on it
   - `@dependabot cancel merge` will cancel a previously requested merge and 
block automerging
   - `@dependabot reopen` will reopen this PR if it is closed
   - `@dependabot close` will close this PR and stop Dependabot recreating it. 
You can achieve the same result by closing it manually
   - `@dependabot ignore this major version` will close this PR and stop 
Dependabot creating any more for this major version (unless you reopen the PR 
or upgrade to it yourself)
   - `@dependabot ignore this minor version` will close this PR and stop 
Dependabot creating any more for this minor version (unless you reopen the PR 
or upgrade to it yourself)
   - `@dependabot ignore this dependency` will close this PR and stop 
Dependabot creating any more for this dependency (unless you reopen the PR or 
upgrade to it yourself)
   - `@dependabot use these labels` will set the current labels as the default 
for future PRs for this repo and language
   - `@dependabot use these reviewers` will set the current reviewers as the 
default for future PRs for this repo and language
   - `@dependabot use these assignees` will set the current assignees as the 
default for future PRs for this repo and language
   - `@dependabot use this milestone` will set the current milestone as the 
default for future PRs for this repo and language
   
   You can disable automated security fix PRs for this repo from the 

[GitHub] [incubator-sdap-nexus] ngachung merged pull request #209: SDAP-406: Fix comparison stats bug in timeseriesspark algorithm

[GitBox]

ngachung merged PR #209:
URL: https://github.com/apache/incubator-sdap-nexus/pull/209


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] kevinmarlis opened a new pull request, #209: SDAP-406: Fix comparison stats bug in timeseriesspark algorithm

[GitBox]

kevinmarlis opened a new pull request, #209:
URL: https://github.com/apache/incubator-sdap-nexus/pull/209

   This PR fixes a bug in the `/timeSeriesSpark` algorithm that would return 
NaNs in the comparison stats dictionary when comparing two datasets with only 
one overlapping date. 
   
   ex: Under certain conditions there can be only a single date matchup between 
DS1 and DS2 which results in the linear regression algorithm used to compute 
the comparison stats returning NaNs. 
   
   The fix (for now) is to check for NaNs in the results of the linear 
regression algorithm and set the comparison stats to an empty dictionary, 
following the precedent set when there are no matching dates. 
   
   This has been tested in SLCP's deployment.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] ngachung merged pull request #208: Add second NCAR provider

[GitBox]

ngachung merged PR #208:
URL: https://github.com/apache/incubator-sdap-nexus/pull/208


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] skorper opened a new pull request, #208: Add second NCAR provider

[GitBox]

skorper opened a new pull request, #208:
URL: https://github.com/apache/incubator-sdap-nexus/pull/208

   Added a new provider for NCAR so either the NCAR insitu api or JPL insitu 
api can be specified. This wasn't really possible due to the way this was 
written so I had to add an extra field in the insitu config `short_name`. This 
is optional, and can be specified when executing matching and specifying the 
secondary dataset. 
   
   With these changes, the following secondary ICOADS values can be provided:
   
   - `ICOADS%20Release%203.0` Uses the NCAR In-Situ API (this is the old method 
which still works)
   - `ICOADS_NCAR` Uses the NCAR In-Situ API
   - `ICOADS_JPL` Uses the JPL In-Situ API
   
   Once validated, I'd recommend we revert these changes as they won't be 
useful in a practical scenario.
   
   ---
   
   Testing:
   
   ```bash
   
{{big_data_url}}/match_spark?primary=MUR25-JPL-L4-GLOB-v04.2=ICOADS%20Release%203.0=2017-07-01T00%3A00%3A00Z=2017-07-07T23%3A59%3A59Z=-100%2C20%2C-79%2C30=30=-10=10=43200=5000=false=0
   ```
   
   ```json
   "details": {
   "timeToComplete": 72,
   "numSecondaryMatched": 258,
   "numPrimaryMatched": 101
   }
   ```
   
   ```bash
   
{{big_data_url}}/match_spark?primary=MUR25-JPL-L4-GLOB-v04.2=ICOADS_JPL=2017-07-01T00%3A00%3A00Z=2017-07-07T23%3A59%3A59Z=-100%2C20%2C-79%2C30=30=-10=10=43200=5000=false=0
   ```
   
   ```json
   "details": {
   "timeToComplete": 83,
   "numSecondaryMatched": 218,
   "numPrimaryMatched": 82
   }
   ```
   
   ```bash
   
{{big_data_url}}/match_spark?primary=MUR25-JPL-L4-GLOB-v04.2=ICOADS_NCAR=2017-07-01T00%3A00%3A00Z=2017-07-07T23%3A59%3A59Z=-100%2C20%2C-79%2C30=30=-10=10=43200=5000=false=0
   ```
   
   ```json
   "details": {
   "timeToComplete": 77,
   "numSecondaryMatched": 258,
   "numPrimaryMatched": 101
   }
   ```
   
   Also confirmed through logs that the expected API is hit. Note: the NCAR 
matchup returns different results so we should definitely look into that.
   
   Also tested with the `match_spark_doms` to make sure nothing was broken:
   
   ```json
   
{{big_data_url}}/domssubset?dataset=ASCATB-L2-Coastal=samos=2017-05-01T00%3A00%3A00Z=2017-05-14T23%3A59%3A59Z=-100%2C10%2C-79%2C-30=wind=-10=10=1%2C2%2C3%2C4%2C5%2C6%2C7%2C8%2C9
   ```
   
   No errors occured. 
   
   Also tested with unrelated `ASCATA-L2` -> shark matchup request, also worked 
as expected. 
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff opened a new pull request, #207: Insitu query logging

[GitBox]

RKuttruff opened a new pull request, #207:
URL: https://github.com/apache/incubator-sdap-nexus/pull/207

   Improved the logging for insitu queries to show when the query starts + its 
URL and then when the query ends, its URL (for correlation), status code, and 
elapsed time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #185: SDAP-399: Quickstart update - Docker

[GitBox]

RKuttruff commented on code in PR #185:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/185#discussion_r993848863


##
docs/quickstart.rst:
##
@@ -64,181 +80,238 @@ The network we will be using for this quickstart will be 
called ``sdap-net``. Cr
 
 .. _quickstart-step3:
 
-Download Sample Data
--
+Start Ingester Components and Ingest Some Science Data
+
 
-The data we will be downloading is part of the `AVHRR OI dataset 
`_ which 
measures sea surface temperature. We will download 1 month of data and ingest 
it into a local Solr and Cassandra instance.
+Create Data Directory
+
+
+Let's start by creating the directory to hold the science data to ingest.
 
 Choose a location that is mountable by Docker (typically needs to be under the 
User's home directory) to download the data files to.
 
 .. code-block:: bash
 
-  export DATA_DIRECTORY=~/nexus-quickstart/data/avhrr-granules
-  mkdir -p ${DATA_DIRECTORY}
+export DATA_DIRECTORY=~/nexus-quickstart/data/avhrr-granules
+mkdir -p ${DATA_DIRECTORY}
 
-Then go ahead and download 1 month worth of AVHRR netCDF files.
+Now we can start up the data storage components. We will be using Solr and 
Cassandra to store the tile metadata and data respectively.
 
-.. code-block:: bash
+.. _quickstart-step4:
 
-  cd $DATA_DIRECTORY
+Start Zookeeper
+---
 
-  export 
URL_LIST="https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/305/2015110112-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/306/2015110212-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/307/2015110312-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/308/2015110412-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/309/2015110512-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/
 2015/310/2015110612-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/311/2015110712-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/312/2015110812-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/313/2015110912-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/314/2015111012-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/315/201512-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 https://podaac-opendap.jpl.nasa.gov:443
 
/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/316/2015111212-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/317/2015111312-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/318/2015111412-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/319/2015111512-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/320/2015111612-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/321/2015111712-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-G
 LOB-v02.0-fv02.0.nc 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/322/2015111812-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/323/2015111912-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 

[GitHub] [incubator-sdap-nexus] ngachung merged pull request #205: RMQ monitor script for use in quickstart guide

[GitBox]

ngachung merged PR #205:
URL: https://github.com/apache/incubator-sdap-nexus/pull/205


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] ngachung merged pull request #202: SDAP-402: Update matchOnce logic

[GitBox]

ngachung merged PR #202:
URL: https://github.com/apache/incubator-sdap-nexus/pull/202


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] skorper commented on pull request #202: SDAP-402: Update matchOnce logic

[GitBox]

skorper commented on PR #202:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/202#issuecomment-1275745444

   @ngachung The duplicate primary issue seems to occur prior to these changes, 
so it shouldn't block moving forward with this PR. All conflicts have been 
resolved


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #185: SDAP-399: Quickstart update - Docker

[GitBox]

RKuttruff commented on code in PR #185:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/185#discussion_r992846816


##
docs/quickstart.rst:
##
@@ -18,43 +18,59 @@ This quickstart guide will walk you through how to install 
and run NEXUS on your
 Prerequisites
 ==
 
-* Docker (tested on v18.03.1-ce)
+* Docker (tested on v20.10.17)
 * Internet Connection
-* bash
+* bash or zsh
 * cURL
-* 500 MB of disk space
+* 10.5 GB of disk space
 
 Prepare
 
 
-Start downloading the Docker images and data files.
+Start downloading the Docker images and set up the Docker bridge network.
 
 .. _quickstart-step1:
 
 Pull Docker Images
 ---
 
-Pull the necessary Docker images from the `SDAP repository 
`_ on Docker Hub. Please check the repository 
for the latest version tag.
+Pull the necessary Docker images from the `NEXUS JPL repository 
`_ on Docker Hub. Please check the 
repository for the latest version tag.
 
 .. code-block:: bash
 
-  export VERSION=1.0.0-rc1
+  export CASSANDRA_VERSION=3.11.6-debian-10-r138
+  export RMQ_VERSION=3.8.9-debian-10-r37
+  export COLLECTION_MANAGER_VERSION=0.1.6a14
+  export GRANULE_INGESTER_VERSION=0.1.6a30
+  export WEBAPP_VERSION=distributed.0.4.5a49
+  export SOLR_VERSION=8.11.1
+  export SOLR_CLOUD_INIT_VERSION=1.0.2
+  export ZK_VERSION=3.5.5
+
+  export JUPYTER_VERSION=1.2
 
 .. code-block:: bash
 
-  docker pull sdap/ningester:${VERSION}
-  docker pull sdap/solr-singlenode:${VERSION}
-  docker pull sdap/cassandra:${VERSION}
-  docker pull sdap/nexus-webapp:standalone.${VERSION}
+  docker pull bitnami/cassandra:${CASSANDRA_VERSION}
+  docker pull bitnami/rabbitmq:${RMQ_VERSION}
+  docker pull nexusjpl/collection-manager:${COLLECTION_MANAGER_VERSION}
+  docker pull nexusjpl/granule-ingester:${GRANULE_INGESTER_VERSION}
+  docker pull nexusjpl/nexus-webapp:${WEBAPP_VERSION}
+  docker pull nexusjpl/solr:${SOLR_VERSION}
+  docker pull nexusjpl/solr-cloud-init:${SOLR_CLOUD_INIT_VERSION}
+  docker pull zookeeper:${ZK_VERSION}
+
+  # docker pull nexusjpl/jupyter:${JUPYTER_VERSION}

Review Comment:
   Updated



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] skorper merged pull request #206: Insitu schema

[GitBox]

skorper merged PR #206:
URL: https://github.com/apache/incubator-sdap-nexus/pull/206


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] skorper commented on pull request #206: Insitu schema

[GitBox]

skorper commented on PR #206:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/206#issuecomment-1273934832

   Thank you @RKuttruff !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff opened a new pull request, #206: Insitu schema

[GitBox]

RKuttruff opened a new pull request, #206:
URL: https://github.com/apache/incubator-sdap-nexus/pull/206

   Matchup will fetch the insitu schema when it is needed instead of fetching 
on import.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #185: SDAP-399: Quickstart update - Docker

[GitBox]

RKuttruff commented on code in PR #185:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/185#discussion_r989493866


##
docs/quickstart.rst:
##
@@ -18,43 +18,59 @@ This quickstart guide will walk you through how to install 
and run NEXUS on your
 Prerequisites
 ==
 
-* Docker (tested on v18.03.1-ce)
+* Docker (tested on v20.10.17)
 * Internet Connection
-* bash
+* bash or zsh
 * cURL
-* 500 MB of disk space
+* 10.5 GB of disk space
 
 Prepare
 
 
-Start downloading the Docker images and data files.
+Start downloading the Docker images and set up the Docker bridge network.
 
 .. _quickstart-step1:
 
 Pull Docker Images
 ---
 
-Pull the necessary Docker images from the `SDAP repository 
`_ on Docker Hub. Please check the repository 
for the latest version tag.
+Pull the necessary Docker images from the `NEXUS JPL repository 
`_ on Docker Hub. Please check the 
repository for the latest version tag.
 
 .. code-block:: bash
 
-  export VERSION=1.0.0-rc1
+  export CASSANDRA_VERSION=3.11.6-debian-10-r138
+  export RMQ_VERSION=3.8.9-debian-10-r37
+  export COLLECTION_MANAGER_VERSION=0.1.6a14
+  export GRANULE_INGESTER_VERSION=0.1.6a30
+  export WEBAPP_VERSION=distributed.0.4.5a49
+  export SOLR_VERSION=8.11.1
+  export SOLR_CLOUD_INIT_VERSION=1.0.2
+  export ZK_VERSION=3.5.5
+
+  export JUPYTER_VERSION=1.2
 
 .. code-block:: bash
 
-  docker pull sdap/ningester:${VERSION}
-  docker pull sdap/solr-singlenode:${VERSION}
-  docker pull sdap/cassandra:${VERSION}
-  docker pull sdap/nexus-webapp:standalone.${VERSION}
+  docker pull bitnami/cassandra:${CASSANDRA_VERSION}
+  docker pull bitnami/rabbitmq:${RMQ_VERSION}
+  docker pull nexusjpl/collection-manager:${COLLECTION_MANAGER_VERSION}
+  docker pull nexusjpl/granule-ingester:${GRANULE_INGESTER_VERSION}
+  docker pull nexusjpl/nexus-webapp:${WEBAPP_VERSION}
+  docker pull nexusjpl/solr:${SOLR_VERSION}
+  docker pull nexusjpl/solr-cloud-init:${SOLR_CLOUD_INIT_VERSION}
+  docker pull zookeeper:${ZK_VERSION}
+
+  # docker pull nexusjpl/jupyter:${JUPYTER_VERSION}

Review Comment:
   I've built and tested a new version, `rileykkjpl/jupyter:1.5`. Once it's 
pushed to `nexusjpl` I can fix this line.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #185: SDAP-399: Quickstart update - Docker

[GitBox]

RKuttruff commented on code in PR #185:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/185#discussion_r989493112


##
docs/quickstart.rst:
##
@@ -64,181 +80,240 @@ The network we will be using for this quickstart will be 
called ``sdap-net``. Cr
 
 .. _quickstart-step3:
 
-Download Sample Data
--
+Start Ingester Components and Ingest Some Science Data

Review Comment:
   Headers updated



##
docs/quickstart.rst:
##
@@ -64,181 +80,240 @@ The network we will be using for this quickstart will be 
called ``sdap-net``. Cr
 
 .. _quickstart-step3:
 
-Download Sample Data
--
+Start Ingester Components and Ingest Some Science Data

Review Comment:
   Updated in this version: 
https://github.com/RKuttruff/incubator-sdap-nexus/blob/d3aac0187fe3bcf6008a12508bbf7b8078a0e4dd/docs/quickstart.rst



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #185: SDAP-399: Quickstart update - Docker

[GitBox]

RKuttruff commented on code in PR #185:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/185#discussion_r989492574


##
docs/quickstart.rst:
##
@@ -64,181 +80,240 @@ The network we will be using for this quickstart will be 
called ``sdap-net``. Cr
 
 .. _quickstart-step3:
 
-Download Sample Data
--
+Start Ingester Components and Ingest Some Science Data
+
 
-The data we will be downloading is part of the `AVHRR OI dataset 
`_ which 
measures sea surface temperature. We will download 1 month of data and ingest 
it into a local Solr and Cassandra instance.
+Create Data Directory
+
+
+Let's start by creating the directory to hold the science data to ingest.
 
 Choose a location that is mountable by Docker (typically needs to be under the 
User's home directory) to download the data files to.
 
 .. code-block:: bash
 
-  export DATA_DIRECTORY=~/nexus-quickstart/data/avhrr-granules
-  mkdir -p ${DATA_DIRECTORY}
+export DATA_DIRECTORY=~/nexus-quickstart/data/avhrr-granules
+mkdir -p ${DATA_DIRECTORY}
 
-Then go ahead and download 1 month worth of AVHRR netCDF files.
+Now we can start up the data storage components. We will be using Solr and 
Cassandra to store the tile metadata and data respectively.
 
-.. code-block:: bash
+.. _quickstart-step4:
 
-  cd $DATA_DIRECTORY
+Start Zookeeper
+---
 
-  export 
URL_LIST="https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/305/2015110112-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/306/2015110212-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/307/2015110312-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/308/2015110412-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/309/2015110512-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/
 2015/310/2015110612-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/311/2015110712-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/312/2015110812-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/313/2015110912-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/314/2015111012-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/315/201512-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 https://podaac-opendap.jpl.nasa.gov:443
 
/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/316/2015111212-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/317/2015111312-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/318/2015111412-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/319/2015111512-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/320/2015111612-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/321/2015111712-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-G
 LOB-v02.0-fv02.0.nc 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/322/2015111812-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/323/2015111912-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 

[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #185: SDAP-399: Quickstart update - Docker

[GitBox]

RKuttruff commented on code in PR #185:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/185#discussion_r988389578


##
docs/quickstart.rst:
##
@@ -64,181 +80,238 @@ The network we will be using for this quickstart will be 
called ``sdap-net``. Cr
 
 .. _quickstart-step3:
 
-Download Sample Data
--
+Start Ingester Components and Ingest Some Science Data
+
 
-The data we will be downloading is part of the `AVHRR OI dataset 
`_ which 
measures sea surface temperature. We will download 1 month of data and ingest 
it into a local Solr and Cassandra instance.
+Create Data Directory
+
+
+Let's start by creating the directory to hold the science data to ingest.
 
 Choose a location that is mountable by Docker (typically needs to be under the 
User's home directory) to download the data files to.
 
 .. code-block:: bash
 
-  export DATA_DIRECTORY=~/nexus-quickstart/data/avhrr-granules
-  mkdir -p ${DATA_DIRECTORY}
+export DATA_DIRECTORY=~/nexus-quickstart/data/avhrr-granules
+mkdir -p ${DATA_DIRECTORY}
 
-Then go ahead and download 1 month worth of AVHRR netCDF files.
+Now we can start up the data storage components. We will be using Solr and 
Cassandra to store the tile metadata and data respectively.
 
-.. code-block:: bash
+.. _quickstart-step4:
 
-  cd $DATA_DIRECTORY
+Start Zookeeper
+---
 
-  export 
URL_LIST="https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/305/2015110112-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/306/2015110212-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/307/2015110312-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/308/2015110412-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/309/2015110512-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/
 2015/310/2015110612-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/311/2015110712-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/312/2015110812-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/313/2015110912-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/314/2015111012-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/315/201512-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 https://podaac-opendap.jpl.nasa.gov:443
 
/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/316/2015111212-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/317/2015111312-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/318/2015111412-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/319/2015111512-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/320/2015111612-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/321/2015111712-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-G
 LOB-v02.0-fv02.0.nc 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/322/2015111812-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/323/2015111912-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 

[GitHub] [incubator-sdap-nexus] ngachung commented on a diff in pull request #185: SDAP-399: Quickstart update - Docker

[GitBox]

ngachung commented on code in PR #185:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/185#discussion_r988383401


##
docs/quickstart.rst:
##
@@ -64,181 +80,238 @@ The network we will be using for this quickstart will be 
called ``sdap-net``. Cr
 
 .. _quickstart-step3:
 
-Download Sample Data
--
+Start Ingester Components and Ingest Some Science Data
+
 
-The data we will be downloading is part of the `AVHRR OI dataset 
`_ which 
measures sea surface temperature. We will download 1 month of data and ingest 
it into a local Solr and Cassandra instance.
+Create Data Directory
+
+
+Let's start by creating the directory to hold the science data to ingest.
 
 Choose a location that is mountable by Docker (typically needs to be under the 
User's home directory) to download the data files to.
 
 .. code-block:: bash
 
-  export DATA_DIRECTORY=~/nexus-quickstart/data/avhrr-granules
-  mkdir -p ${DATA_DIRECTORY}
+export DATA_DIRECTORY=~/nexus-quickstart/data/avhrr-granules
+mkdir -p ${DATA_DIRECTORY}
 
-Then go ahead and download 1 month worth of AVHRR netCDF files.
+Now we can start up the data storage components. We will be using Solr and 
Cassandra to store the tile metadata and data respectively.
 
-.. code-block:: bash
+.. _quickstart-step4:
 
-  cd $DATA_DIRECTORY
+Start Zookeeper
+---
 
-  export 
URL_LIST="https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/305/2015110112-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/306/2015110212-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/307/2015110312-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/308/2015110412-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/309/2015110512-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/
 2015/310/2015110612-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/311/2015110712-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/312/2015110812-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/313/2015110912-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/314/2015111012-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/315/201512-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 https://podaac-opendap.jpl.nasa.gov:443
 
/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/316/2015111212-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/317/2015111312-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/318/2015111412-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/319/2015111512-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/320/2015111612-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/321/2015111712-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-G
 LOB-v02.0-fv02.0.nc 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/322/2015111812-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/323/2015111912-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 

[GitHub] [incubator-sdap-nexus] skorper merged pull request #204: SDAP-405 Added SPURS AWS in situ api

[GitBox]

skorper merged PR #204:
URL: https://github.com/apache/incubator-sdap-nexus/pull/204


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] ngachung commented on a diff in pull request #204: SDAP-405 Added SPURS AWS in situ api

[GitBox]

ngachung commented on code in PR #204:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/204#discussion_r988317931


##
CHANGELOG.md:
##
@@ -45,6 +46,7 @@ and this project adheres to [Semantic 
Versioning](https://semver.org/spec/v2.0.0
 - Fixed import error causing `/timeSeriesSpark` queries to fail.
 - Fixed bug where domsresults no longer worked after successful matchup
 - Fixed certificate error in Dockerfile
+- SDAP-403: Remote timeout fix and HofMoeller bug fix

Review Comment:
   I realized I merged https://github.com/apache/incubator-sdap-nexus/pull/203 
but CHANGELOG.md wasn't updated so I squeezed that into this PR for 
completeness.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #185: SDAP-399: Quickstart update - Docker

[GitBox]

RKuttruff commented on code in PR #185:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/185#discussion_r984985448


##
CHANGELOG.md:
##
@@ -41,9 +41,9 @@ and this project adheres to [Semantic 
Versioning](https://semver.org/spec/v2.0.0
 - Fixed issue where satellite to satellite matchups with the same dataset 
don't return the expected result
 - Fixed CSV and NetCDF matchup output bug
 - Fixed NetCDF output switching latitude and longitude
+- SDAP-399: Updated quickstart guide for standalone docker deployment of SDAP.
+- SDAP-399: Updated quickstart Jupyter notebook
 - Fixed import error causing `/timeSeriesSpark` queries to fail.
 - Fixed bug where domsresults no longer worked after successful matchup
 - Fixed certificate error in Dockerfile
-### Security

Review Comment:
   Added a newline after. It no longer shows as deleted



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #185: SDAP-399: Quickstart update - Docker

[GitBox]

RKuttruff commented on code in PR #185:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/185#discussion_r984984394


##
docker/jupyter/Dockerfile:
##
@@ -29,7 +29,8 @@ ENV CHOWN_HOME_OPTS='-R'
 ENV REBUILD_CODE=true
 
 ARG APACHE_NEXUS=https://github.com/apache/incubator-sdap-nexus.git
-ARG APACHE_NEXUS_BRANCH=master
+ARG APACHE_NEXUS_COMMIT=be19c1d567301b09269e851cc5b5af55fea02c5d

Review Comment:
   Updated to point to nexuscli in master branch. Rebuilt & tested image
   `rileykkjpl/jupyter:1.5`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #185: SDAP-399: Quickstart update - Docker

[GitBox]

RKuttruff commented on code in PR #185:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/185#discussion_r984941752


##
docs/quickstart.rst:
##
@@ -64,181 +80,240 @@ The network we will be using for this quickstart will be 
called ``sdap-net``. Cr
 
 .. _quickstart-step3:
 
-Download Sample Data
--
+Start Ingester Components and Ingest Some Science Data
+
 
-The data we will be downloading is part of the `AVHRR OI dataset 
`_ which 
measures sea surface temperature. We will download 1 month of data and ingest 
it into a local Solr and Cassandra instance.
+Create Data Directory
+
+
+Let's start by creating the directory to hold the science data to ingest.
 
 Choose a location that is mountable by Docker (typically needs to be under the 
User's home directory) to download the data files to.
 
 .. code-block:: bash
 
-  export DATA_DIRECTORY=~/nexus-quickstart/data/avhrr-granules
-  mkdir -p ${DATA_DIRECTORY}
+export DATA_DIRECTORY=~/nexus-quickstart/data/avhrr-granules
+mkdir -p ${DATA_DIRECTORY}
 
-Then go ahead and download 1 month worth of AVHRR netCDF files.
+Now we can start up the data storage components. We will be using Solr and 
Cassandra to store the tile metadata and data respectively.
 
-.. code-block:: bash
+.. _quickstart-step4:
 
-  cd $DATA_DIRECTORY
+Start Zookeeper
+---
 
-  export 
URL_LIST="https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/305/2015110112-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/306/2015110212-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/307/2015110312-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/308/2015110412-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/309/2015110512-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/
 2015/310/2015110612-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/311/2015110712-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/312/2015110812-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/313/2015110912-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/314/2015111012-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/315/201512-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 https://podaac-opendap.jpl.nasa.gov:443
 
/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/316/2015111212-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/317/2015111312-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/318/2015111412-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/319/2015111512-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/320/2015111612-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/321/2015111712-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-G
 LOB-v02.0-fv02.0.nc 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/322/2015111812-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/323/2015111912-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 

[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #185: SDAP-399: Quickstart update - Docker

[GitBox]

RKuttruff commented on code in PR #185:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/185#discussion_r984909253


##
CHANGELOG.md:
##
@@ -41,9 +41,9 @@ and this project adheres to [Semantic 
Versioning](https://semver.org/spec/v2.0.0
 - Fixed issue where satellite to satellite matchups with the same dataset 
don't return the expected result
 - Fixed CSV and NetCDF matchup output bug
 - Fixed NetCDF output switching latitude and longitude
+- SDAP-399: Updated quickstart guide for standalone docker deployment of SDAP.
+- SDAP-399: Updated quickstart Jupyter notebook
 - Fixed import error causing `/timeSeriesSpark` queries to fail.
 - Fixed bug where domsresults no longer worked after successful matchup
 - Fixed certificate error in Dockerfile
-### Security

Review Comment:
   
https://github.com/RKuttruff/incubator-sdap-nexus/blob/quickstart-update/CHANGELOG.md?plain=1#L49



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #185: SDAP-399: Quickstart update - Docker

[GitBox]

RKuttruff commented on code in PR #185:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/185#discussion_r984909047


##
CHANGELOG.md:
##
@@ -41,9 +41,9 @@ and this project adheres to [Semantic 
Versioning](https://semver.org/spec/v2.0.0
 - Fixed issue where satellite to satellite matchups with the same dataset 
don't return the expected result
 - Fixed CSV and NetCDF matchup output bug
 - Fixed NetCDF output switching latitude and longitude
+- SDAP-399: Updated quickstart guide for standalone docker deployment of SDAP.
+- SDAP-399: Updated quickstart Jupyter notebook
 - Fixed import error causing `/timeSeriesSpark` queries to fail.
 - Fixed bug where domsresults no longer worked after successful matchup
 - Fixed certificate error in Dockerfile
-### Security

Review Comment:
   Well this is strange...
   It's there in the branch



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #185: SDAP-399: Quickstart update - Docker

[GitBox]

RKuttruff commented on code in PR #185:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/185#discussion_r984137556


##
docker/jupyter/requirements.txt:
##
@@ -1,4 +1,3 @@
-shapely
-requests
-numpy
-cassandra-driver==3.9.0

Review Comment:
   Not sure. The notebook runs just fine without it and it was causing build 
errors.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #185: SDAP-399: Quickstart update - Docker

[GitBox]

RKuttruff commented on code in PR #185:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/185#discussion_r984137949


##
docker/jupyter/requirements.txt:
##
@@ -1,4 +1,3 @@
-shapely
-requests
-numpy
-cassandra-driver==3.9.0

Review Comment:
   Possibly the notebook intended to do more. Or the the file was copy pasted 
from one that did and never checked.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #185: SDAP-399: Quickstart update - Docker

[GitBox]

RKuttruff commented on code in PR #185:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/185#discussion_r984137320


##
docs/quickstart.rst:
##
@@ -18,43 +18,59 @@ This quickstart guide will walk you through how to install 
and run NEXUS on your
 Prerequisites
 ==
 
-* Docker (tested on v18.03.1-ce)
+* Docker (tested on v20.10.17)
 * Internet Connection
-* bash
+* bash or zsh
 * cURL
-* 500 MB of disk space
+* 10.5 GB of disk space
 
 Prepare
 
 
-Start downloading the Docker images and data files.
+Start downloading the Docker images and set up the Docker bridge network.
 
 .. _quickstart-step1:
 
 Pull Docker Images
 ---
 
-Pull the necessary Docker images from the `SDAP repository 
`_ on Docker Hub. Please check the repository 
for the latest version tag.
+Pull the necessary Docker images from the `NEXUS JPL repository 
`_ on Docker Hub. Please check the 
repository for the latest version tag.
 
 .. code-block:: bash
 
-  export VERSION=1.0.0-rc1
+  export CASSANDRA_VERSION=3.11.6-debian-10-r138
+  export RMQ_VERSION=3.8.9-debian-10-r37
+  export COLLECTION_MANAGER_VERSION=0.1.6a14
+  export GRANULE_INGESTER_VERSION=0.1.6a30
+  export WEBAPP_VERSION=distributed.0.4.5a49
+  export SOLR_VERSION=8.11.1
+  export SOLR_CLOUD_INIT_VERSION=1.0.2
+  export ZK_VERSION=3.5.5
+
+  export JUPYTER_VERSION=1.2
 
 .. code-block:: bash
 
-  docker pull sdap/ningester:${VERSION}
-  docker pull sdap/solr-singlenode:${VERSION}
-  docker pull sdap/cassandra:${VERSION}
-  docker pull sdap/nexus-webapp:standalone.${VERSION}
+  docker pull bitnami/cassandra:${CASSANDRA_VERSION}
+  docker pull bitnami/rabbitmq:${RMQ_VERSION}
+  docker pull nexusjpl/collection-manager:${COLLECTION_MANAGER_VERSION}
+  docker pull nexusjpl/granule-ingester:${GRANULE_INGESTER_VERSION}
+  docker pull nexusjpl/nexus-webapp:${WEBAPP_VERSION}
+  docker pull nexusjpl/solr:${SOLR_VERSION}
+  docker pull nexusjpl/solr-cloud-init:${SOLR_CLOUD_INIT_VERSION}
+  docker pull zookeeper:${ZK_VERSION}
+
+  # docker pull nexusjpl/jupyter:${JUPYTER_VERSION}

Review Comment:
   Ok. I'll let you know after I test the newer version of the image since I 
can't push to nexusjpl



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #185: SDAP-399: Quickstart update - Docker

[GitBox]

RKuttruff commented on code in PR #185:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/185#discussion_r984137144


##
docs/quickstart.rst:
##
@@ -64,181 +80,240 @@ The network we will be using for this quickstart will be 
called ``sdap-net``. Cr
 
 .. _quickstart-step3:
 
-Download Sample Data
--
+Start Ingester Components and Ingest Some Science Data

Review Comment:
   Will do.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@sdap.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [incubator-sdap-nexus] RKuttruff commented on a diff in pull request #185: SDAP-399: Quickstart update - Docker

[GitBox]

RKuttruff commented on code in PR #185:
URL: 
https://github.com/apache/incubator-sdap-nexus/pull/185#discussion_r984136931


##
docs/quickstart.rst:
##
@@ -64,181 +80,240 @@ The network we will be using for this quickstart will be 
called ``sdap-net``. Cr
 
 .. _quickstart-step3:
 
-Download Sample Data
--
+Start Ingester Components and Ingest Some Science Data
+
 
-The data we will be downloading is part of the `AVHRR OI dataset 
`_ which 
measures sea surface temperature. We will download 1 month of data and ingest 
it into a local Solr and Cassandra instance.
+Create Data Directory
+
+
+Let's start by creating the directory to hold the science data to ingest.
 
 Choose a location that is mountable by Docker (typically needs to be under the 
User's home directory) to download the data files to.
 
 .. code-block:: bash
 
-  export DATA_DIRECTORY=~/nexus-quickstart/data/avhrr-granules
-  mkdir -p ${DATA_DIRECTORY}
+export DATA_DIRECTORY=~/nexus-quickstart/data/avhrr-granules
+mkdir -p ${DATA_DIRECTORY}
 
-Then go ahead and download 1 month worth of AVHRR netCDF files.
+Now we can start up the data storage components. We will be using Solr and 
Cassandra to store the tile metadata and data respectively.
 
-.. code-block:: bash
+.. _quickstart-step4:
 
-  cd $DATA_DIRECTORY
+Start Zookeeper
+---
 
-  export 
URL_LIST="https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/305/2015110112-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/306/2015110212-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/307/2015110312-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/308/2015110412-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/309/2015110512-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/
 2015/310/2015110612-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/311/2015110712-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/312/2015110812-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/313/2015110912-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/314/2015111012-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/315/201512-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 https://podaac-opendap.jpl.nasa.gov:443
 
/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/316/2015111212-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/317/2015111312-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/318/2015111412-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/319/2015111512-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/320/2015111612-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/321/2015111712-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-G
 LOB-v02.0-fv02.0.nc 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/322/2015111812-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc
 
https://podaac-opendap.jpl.nasa.gov:443/opendap/allData/ghrsst/data/GDS2/L4/GLOB/NCEI/AVHRR_OI/v2/2015/323/2015111912-NCEI-L4_GHRSST-SSTblend-AVHRR_OI-GLOB-v02.0-fv02.0.nc