This is an automated email from the ASF dual-hosted git repository.
nchung pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-sdap-nexus.git
The following commit(s) were added to refs/heads/master by this push:
new 112144a Preserve insitu schema (#181)
112144a is described below
commit 112144afc88c754e078f46f1c459c3e7bffb2afb
Author: Riley Kuttruff <[email protected]>
AuthorDate: Thu Jul 28 10:16:15 2022 -0700
Preserve insitu schema (#181)
* Preserve insitu schema
Prevent massive number of unneeded api hits
* Update changelog
* Changed method of preserving schema
* Revert changes to insitu api
Shouldn't be changed in pr
* Revert changes to insitu api
Shouldn't be changed in pr
* Update changelog
Co-authored-by: rileykk <[email protected]>
---
CHANGELOG.md | 1 +
analysis/webservice/algorithms_spark/Matchup.py | 10 +++++++---
2 files changed, 8 insertions(+), 3 deletions(-)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index a3183da..44639b1 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -25,6 +25,7 @@ and this project adheres to [Semantic
Versioning](https://semver.org/spec/v2.0.0
- SDAP-397: Added null unit field for later use in matchup
- Fixed issue where satellite satellite matchups failed
- Fixed issue where null results were returned if more than "resultSizeLimit"
matches are found
+- Preserve return value of `query_insitu_schema()` in matchup to avoid
excessive API hits
- Fixed issue where satellite to satellite matchups with the same dataset
don't return the expected result
- Fixed CSV and NetCDF matchup output bug
- Fixed NetCDF output switching latitude and longitude
diff --git a/analysis/webservice/algorithms_spark/Matchup.py
b/analysis/webservice/algorithms_spark/Matchup.py
index 2ba55f5..5ae74ea 100644
--- a/analysis/webservice/algorithms_spark/Matchup.py
+++ b/analysis/webservice/algorithms_spark/Matchup.py
@@ -430,8 +430,13 @@ class DomsPoint(object):
point.device = DomsPoint._variables_to_device(tile.variables)
return point
+ insitu_schema = None
+
@staticmethod
def from_edge_point(edge_point):
+ if DomsPoint.insitu_schema is None:
+ DomsPoint.insitu_schema = query_insitu_schema()
+
point = DomsPoint()
x, y = edge_point['longitude'], edge_point['latitude']
@@ -506,7 +511,7 @@ class DomsPoint(object):
val = edge_point.get(name)
if not val:
continue
- unit = get_insitu_unit(name)
+ unit = get_insitu_unit(name, DomsPoint.insitu_schema)
data.append(DataPoint(
variable_name=name,
cf_variable_name=name,
@@ -642,12 +647,11 @@ def add_meters_to_lon_lat(lon, lat, meters):
return longitude, latitude
-def get_insitu_unit(variable_name):
+def get_insitu_unit(variable_name, insitu_schema):
"""
Query the insitu API and retrieve the units for the given variable.
If no units are available for this variable, return "None"
"""
- insitu_schema = query_insitu_schema()
properties = insitu_schema.get('definitions', {}).get('observation',
{}).get('properties', {})
for observation_name, observation_value in properties.items():
if observation_name == variable_name:
