This is an automated email from the ASF dual-hosted git repository.
rkk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/sdap-ingester.git
The following commit(s) were added to refs/heads/master by this push:
new e877560 Version 1.3.0 release (#100)
e877560 is described below
commit e877560e8c940e992e1ed0e6442ceae8aa03c938
Author: Riley Kuttruff <[email protected]>
AuthorDate: Mon Jun 24 10:17:46 2024 -0700
Version 1.3.0 release (#100)
* bump version to 1.2.0
* Remove chardet from images (#91)
* CM build w/o chardet
* GI build w/o chardet
* Reworked collection manager dependencies
* Update CHANGELOG.md
* GI: Fallback to conda if mamba is not present
---------
Co-authored-by: rileykk <[email protected]>
* Updated ASF yaml (#94)
- Branch protections
- Branch auto-delete
Co-authored-by: rileykk <[email protected]>
* SDAP-511 - Switch package manager to poetry (#95)
* Removed conda from docker image build
- Base image -> Java (to support possible nexusproto build)
- Just use pip in setup.py
* Ingester poetry
* Bumped GI Python version
Fixed issue with xarray being unable to open netcdf files
* Changelogs
* No chardet
* Delete old dockerfile
* Removed old conda install files
---------
Co-authored-by: rileykk <[email protected]>
* SDAP-512 - Have GI close ZK/Solr/Cassandra connections when done with
writing to Solr (#97)
* Explicitly close Solr & ZK connections
* Try just using destructor instead of explicit close() calls
* Changelog
* Removed commented out code
* Also close Cassandra connections after write
* Update changelog
---------
Co-authored-by: rileykk <[email protected]>
* CM improvements (#93)
* Suppress overly verbose loggers instead of all loggers to INFO
* Collection manager improvements
- Yield S3 Observer task as it's creating update events so that those
events don't block until the iteration is complete
- Improved logging
- Log level settable by environment var LOG_LEVEL. Could be level name (ie,
INFO, WARNING, &c) or numerical value
* Changelog
* Moved changelog entry to correct section
---------
Co-authored-by: rileykk <[email protected]>
* SDAP-472 - Support for defining gridded Zarr datasets from the collection
config (#86)
* Zarr onboarding
* Dynamic dataset management
* nexusdatasets setup
* Fix bad field name in Solr doc
* Changelog
* Changelog
---------
Co-authored-by: rileykk <[email protected]>
* SDAP-502: Gridded tile generation bug patch (#92)
* SDAP-502: Gridded tile gen squeeze bug patch
* Added unit test
* Changelog typo
---------
Co-authored-by: rileykk <[email protected]>
* Fixed install of kubectl in Collection Manager image build (#99)
* Fixed install of kubectl in Collection Manager image build
* Changelog update
* Changelog update
---------
Co-authored-by: rileykk <[email protected]>
* Update release files
- 1.3.0 header in changelogs
- Removed incubation DISCLAIMER
- Removed incubation language from NOTICE and updated URLs
- Bumped versions
- Made copyright year current
- Poetry re-lock
* Fixed Dockerfile for DISCLAIMER rm
* patch
---------
Co-authored-by: skperez <[email protected]>
Co-authored-by: rileykk <[email protected]>
Co-authored-by: Stepheny Perez <[email protected]>
---
CHANGELOG.md | 18 +-
DISCLAIMER | 6 -
NOTICE | 4 +-
README | 6 +-
VERSION.txt | 2 +-
.../collection_manager/entities/Collection.py | 25 ++-
collection_manager/collection_manager/main.py | 31 ++-
.../services/CollectionProcessor.py | 25 ++-
.../services/CollectionWatcher.py | 9 +-
.../collection_manager/services/S3Observer.py | 23 +-
.../history_manager/SolrIngestionHistory.py | 23 +-
collection_manager/docker/Dockerfile | 6 +-
collection_manager/requirements.txt | 1 -
granule_ingester/docker/Dockerfile | 4 +-
.../GridMultiVariableReadingProcessor.py | 13 +-
.../reading_processors/GridReadingProcessor.py | 21 +-
.../granule_ingester/writers/CassandraStore.py | 14 +-
.../granule_ingester/writers/DataStore.py | 11 +
.../granule_ingester/writers/MetadataStore.py | 13 ++
.../granule_ingester/writers/SolrStore.py | 25 ++-
granule_ingester/poetry.lock | 250 +++++++++++----------
granule_ingester/pyproject.toml | 4 +-
.../test_GridReadingProcessor.py | 55 +++++
23 files changed, 418 insertions(+), 171 deletions(-)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index e8340ad..697f239 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,6 +4,22 @@ All notable changes to this project will be documented in this
file.
The format is based on [Keep a
Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic
Versioning](https://semver.org/spec/v2.0.0.html).
+## [1.3.0] - 2024-06-10
+### Added
+- SDAP-472: Added support for defining Zarr collections in the collection
config
+### Changed
+- Improved Collection Manager logging
+ - Inhibited overly verbose loggers
+ - Logging verbosity configurable by environment
+- Improved concurrency for monitoring S3 collections
+### Deprecated
+### Removed
+### Fixed
+- SDAP-512: Fixed Granule Ingester not closing connections to
Zookeeper/Solr/Cassandra, eventually exhausting network resources and requiring
a restart
+- SDAP-502: Fix for rare bug where gridded tiles generated from inputs where
there is a dimension length where `dimensionLength mod tileSliceLength == 1`
would cause tile generation to fail. This is because `np.squeeze` is used on
the coordinate arrays, which, if the generated tile has only a single lat or
lon, would squeeze the corresponding coordinate into a dimensionless array,
which would raise an error down the line when `len` was called with it. Added a
check for this case that bot [...]
+- Fixed install of `kubectl` in Collection Manager image build
+### Security
+
## [1.2.0] - 2023-11-22
### Added
- SDAP-477: Added preprocessor to properly shape incoming data
@@ -15,7 +31,7 @@ and this project adheres to [Semantic
Versioning](https://semver.org/spec/v2.0.0
### Removed
- SDAP-501: Updated dependencies to remove `chardet`
### Fixed
-- SDAP-488: Workaround to build issue on Apple Silicon (M1/M2). GI image build
installs nexusproto through PyPI instead of building from source. A build arg
`BUILD_NEXUSPROTO` was defined to allow building from source if desired/
+- SDAP-488: Workaround to build issue on Apple Silicon (M1/M2). GI image build
installs nexusproto through PyPI instead of building from source. A build arg
`BUILD_NEXUSPROTO` was defined to allow building from source if desired
### Security
## [1.1.0] - 2023-04-26
diff --git a/DISCLAIMER b/DISCLAIMER
deleted file mode 100644
index 020da1e..0000000
--- a/DISCLAIMER
+++ /dev/null
@@ -1,6 +0,0 @@
-Apache SDAP is an effort undergoing incubation at The Apache Software
Foundation (ASF).
-Incubation is required of all newly accepted projects until a further review
indicates
-that the infrastructure, communications, and decision making process have
stabilized in
-a manner consistent with other successful ASF projects. While incubation
status is not
-necessarily a reflection of the completeness or stability of the code, it does
indicate
-that the project has yet to be fully endorsed by the ASF.
diff --git a/NOTICE b/NOTICE
index ab46d1d..ff5438d 100644
--- a/NOTICE
+++ b/NOTICE
@@ -1,5 +1,5 @@
-Apache SDAP (Incubating)
-Copyright 2017 - 2023 The Apache Software Foundation
+Apache SDAP
+Copyright 2017 - 2024 The Apache Software Foundation
This product includes software developed at
The Apache Software Foundation (http://www.apache.org/).
diff --git a/README b/README
index ca6a1f7..ecbd1de 100644
--- a/README
+++ b/README
@@ -1,4 +1,4 @@
-Apache SDAP release 1.2.0
+Apache SDAP release 1.3.0
This is a source distribution of Apache SDAP - Ingester.
@@ -8,10 +8,10 @@ CHANGELOG.md file.
The LICENSE and NOTICE files contain license information.
You can find instructions how to build the release at
-https://incubator-sdap-nexus.readthedocs.io/en/latest/build.html
+https://sdap-nexus.readthedocs.io/en/latest/build.html
Further information about Apache SDAP is available at its web site,
-https://sdap.incubator.apache.org.
+https://sdap.apache.org.
This product includes test data sourced from NASA's Earth
Observing System Data and Information System (EOSDIS). For more
diff --git a/VERSION.txt b/VERSION.txt
index 867e524..589268e 100644
--- a/VERSION.txt
+++ b/VERSION.txt
@@ -1 +1 @@
-1.2.0
\ No newline at end of file
+1.3.0
\ No newline at end of file
diff --git a/collection_manager/collection_manager/entities/Collection.py
b/collection_manager/collection_manager/entities/Collection.py
index 333f454..ee04f1f 100644
--- a/collection_manager/collection_manager/entities/Collection.py
+++ b/collection_manager/collection_manager/entities/Collection.py
@@ -33,6 +33,7 @@ class CollectionStorageType(Enum):
LOCAL = 1
S3 = 2
REMOTE = 3
+ ZARR = 4
@dataclass(frozen=True)
@@ -49,6 +50,8 @@ class Collection:
preprocess: str = None
processors: str = None
group: str = None
+ store_type: str = None
+ config: str = None
@staticmethod
def __decode_dimension_names(dimension_names_dict):
@@ -76,18 +79,27 @@ class Collection:
"""
Accepting either `variable` or `variables` from the configmap
"""
- logger.debug(f'incoming properties dict: {properties}')
+ # Inhibiting this for now...
+ # logger.debug(f'Incoming properties dict: {properties}')
+
try:
date_to = datetime.fromisoformat(properties['to']) if 'to' in
properties else None
date_from = datetime.fromisoformat(properties['from']) if 'from'
in properties else None
+ store_type = properties.get('storeType')
+
+ slices = properties.get('slices', {}).items()
+
preprocess = json.dumps(properties['preprocess']) if 'preprocess'
in properties else None
extra_processors = json.dumps(properties['processors']) if
'processors' in properties else None
+ config = properties['config'] if 'config' in properties else None
+
+ projection = properties['projection'] if 'projection' in
properties else None
collection = Collection(dataset_id=properties['id'],
- projection=properties['projection'],
+ projection=projection,
dimension_names=frozenset(Collection.__decode_dimension_names(properties['dimensionNames'])),
-
slices=frozenset(properties['slices'].items()),
+ slices=frozenset(slices),
path=properties['path'],
historical_priority=properties['priority'],
forward_processing_priority=properties.get('forward-processing-priority', None),
@@ -95,12 +107,17 @@ class Collection:
date_from=date_from,
preprocess=preprocess,
processors=extra_processors,
- group=properties.get('group'))
+ group=properties.get('group'),
+ store_type=store_type,
+ config=config
+ )
return collection
except KeyError as e:
raise MissingValueCollectionError(missing_value=e.args[0])
def storage_type(self):
+ if self.store_type == 'zarr':
+ return CollectionStorageType.ZARR
if urlparse(self.path).scheme == 's3':
return CollectionStorageType.S3
elif urlparse(self.path).scheme in {'http', 'https'}:
diff --git a/collection_manager/collection_manager/main.py
b/collection_manager/collection_manager/main.py
index 8eead68..4f5acf6 100644
--- a/collection_manager/collection_manager/main.py
+++ b/collection_manager/collection_manager/main.py
@@ -24,8 +24,32 @@ from collection_manager.services.history_manager import (
FileIngestionHistoryBuilder, SolrIngestionHistoryBuilder,
md5sum_from_filepath)
-logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s]
[%(name)s::%(lineno)d] %(message)s")
-logging.getLogger("pika").setLevel(logging.WARNING)
+
+log_level = os.getenv('LOG_LEVEL', 'INFO')
+
+if log_level in ['CRITICAL', 'ERROR', 'WARNING', 'INFO', 'DEBUG']:
+ log_level = getattr(logging, log_level)
+else:
+ try:
+ log_level = int(log_level)
+ except:
+ log_level = logging.INFO
+
+logging.basicConfig(level=log_level, format="%(asctime)s [%(levelname)s]
[%(name)s::%(lineno)d] %(message)s")
+
+SUPPRESS = [
+ 'botocore',
+ 's3transfer',
+ 'urllib3',
+ 'pika',
+ 'boto3',
+ 'aioboto3'
+]
+
+for logger_name in SUPPRESS:
+ logging.getLogger(logger_name).setLevel(logging.WARNING)
+
+
logger = logging.getLogger(__name__)
@@ -96,6 +120,7 @@ async def main():
history_manager_builder=history_manager_builder)
collection_watcher =
CollectionWatcher(collections_path=options.collections_path,
granule_updated_callback=collection_processor.process_granule,
+
dataset_added_callback=collection_processor.add_plugin_collection,
collections_refresh_interval=int(options.refresh),
s3_bucket=options.s3_bucket)
@@ -112,4 +137,4 @@ async def main():
if __name__ == "__main__":
- asyncio.run(main())
+ asyncio.run(main(), debug=False)
diff --git
a/collection_manager/collection_manager/services/CollectionProcessor.py
b/collection_manager/collection_manager/services/CollectionProcessor.py
index 6c129c7..6671a50 100644
--- a/collection_manager/collection_manager/services/CollectionProcessor.py
+++ b/collection_manager/collection_manager/services/CollectionProcessor.py
@@ -16,13 +16,14 @@
import json
import logging
import os.path
-from typing import Dict
+from typing import Dict, Optional
import yaml
from collection_manager.entities import Collection
from collection_manager.services import MessagePublisher
from collection_manager.services.history_manager import (GranuleStatus,
- IngestionHistory)
+ IngestionHistory,
+ SolrIngestionHistory)
from collection_manager.services.history_manager.IngestionHistory import \
IngestionHistoryBuilder
@@ -46,6 +47,7 @@ class CollectionProcessor:
:return: None
"""
if not self._file_supported(granule):
+ logger.warning(f'Tried to process unsupported file {granule}.
Skipping.')
return
history_manager = self._get_history_manager(collection.dataset_id)
@@ -74,12 +76,29 @@ class CollectionProcessor:
await self._publisher.publish_message(body=dataset_config,
priority=use_priority)
await history_manager.push(granule, modified_time)
+ def add_plugin_collection(self, collection: Collection):
+ history_manager = self._get_history_manager(None)
+
+ if isinstance(history_manager, SolrIngestionHistory):
+ collection_config = {
+ 'path': collection.path,
+ 'config': collection.config
+ }
+
+ collection_dimensions = dict(collection.dimension_names)
+
+ collection_config['config']['variables'] =
collection_dimensions['variable']
+ collection_config['config']['coords'] = {dim:
collection_dimensions[dim]
+ for dim in
collection_dimensions if dim != 'variable'}
+
+ history_manager._push_dataset(collection.dataset_id,
collection.store_type, json.dumps(collection_config))
+
@staticmethod
def _file_supported(file_path: str):
ext = os.path.splitext(file_path)[-1]
return ext in SUPPORTED_FILE_EXTENSIONS
- def _get_history_manager(self, dataset_id: str) -> IngestionHistory:
+ def _get_history_manager(self, dataset_id: Optional[str]) ->
IngestionHistory:
if dataset_id not in self._history_manager_cache:
self._history_manager_cache[dataset_id] =
self._history_manager_builder.build(dataset_id=dataset_id)
return self._history_manager_cache[dataset_id]
diff --git
a/collection_manager/collection_manager/services/CollectionWatcher.py
b/collection_manager/collection_manager/services/CollectionWatcher.py
index 063e823..94863fd 100644
--- a/collection_manager/collection_manager/services/CollectionWatcher.py
+++ b/collection_manager/collection_manager/services/CollectionWatcher.py
@@ -42,6 +42,7 @@ class CollectionWatcher:
def __init__(self,
collections_path: str,
granule_updated_callback: Callable[[str, Collection],
Awaitable],
+ dataset_added_callback: Callable[[Collection], None],
s3_bucket: Optional[str] = None,
collections_refresh_interval: float = 30):
if not os.path.isabs(collections_path):
@@ -49,6 +50,7 @@ class CollectionWatcher:
self._collections_path = collections_path
self._granule_updated_callback = granule_updated_callback
+ self._dataset_added_callback = dataset_added_callback
self._collections_refresh_interval = collections_refresh_interval
self._collections_by_dir: Dict[str, Set[Collection]] = defaultdict(set)
@@ -96,13 +98,18 @@ class CollectionWatcher:
with open(self._collections_path, 'r') as f:
collections_yaml = yaml.load(f, Loader=yaml.FullLoader)
self._collections_by_dir.clear()
+ logger.info('Refreshing collection config')
+
for collection_dict in collections_yaml['collections']:
try:
collection = Collection.from_dict(collection_dict)
- if collection.storage_type() !=
CollectionStorageType.REMOTE:
+ if collection.storage_type() == CollectionStorageType.ZARR:
+ self._dataset_added_callback(collection)
+ elif collection.storage_type() !=
CollectionStorageType.REMOTE:
self._validate_collection(collection)
self._collections_by_dir[collection.directory()].add(collection)
except MissingValueCollectionError as e:
+ logger.exception(e)
logger.error(f"A collection is missing
'{e.missing_value}'. Ignoring this collection for now.")
except RelativePathCollectionError as e:
logger.error(f"Relative paths are not allowed for the
'path' property of a collection. "
diff --git a/collection_manager/collection_manager/services/S3Observer.py
b/collection_manager/collection_manager/services/S3Observer.py
index 866e43a..5ee84d6 100644
--- a/collection_manager/collection_manager/services/S3Observer.py
+++ b/collection_manager/collection_manager/services/S3Observer.py
@@ -20,9 +20,12 @@ import os
import time
from dataclasses import dataclass
from typing import Set, Dict, Optional, Callable, Awaitable
+import logging
import aioboto3
+logger = logging.getLogger(__name__)
+
@dataclass
class S3Event:
@@ -102,8 +105,14 @@ class S3Observer:
watch_index = {**watch_index, **new_index}
difference = set(new_cache.items()) - set(self._cache.items())
+ logger.info(f'S3 Poll completed; creating events for {len(difference)}
found files')
+ logger.debug(f'(has_polled = {self._has_polled} || init_scan =
{self._initial_scan}) = {self._has_polled or self._initial_scan}')
+
if self._has_polled or self._initial_scan:
- for (file, modified_date) in difference:
+ for i, (file, modified_date) in enumerate(difference):
+ if i % 100 == 0:
+ logger.debug(f'Iterated over {i} items in diff')
+
watch = watch_index[file]
file_is_new = file not in self._cache
@@ -112,6 +121,10 @@ class S3Observer:
else:
watch.event_handler.on_modified(S3FileModifiedEvent(src_path=file,
modified_time=modified_date))
+ await asyncio.sleep(0)
+
+ logger.info('All S3 events for this poll have been created')
+
self._cache = new_cache
self._has_polled = True
@@ -122,12 +135,20 @@ class S3Observer:
async with aioboto3.resource("s3") as s3:
bucket = await s3.Bucket(self._bucket)
+ n_keys = 0
+
object_key = S3Observer._get_object_key(path)
+ logger.debug(f'Listing objects for bucket {self._bucket} under
path {object_key}')
async for file in bucket.objects.filter(Prefix=object_key):
+ n_keys += 1
new_cache[f"s3://{file.bucket_name}/{file.key}"] = await
file.last_modified
+
end = time.perf_counter()
duration = end - start
+ logger.info(f'Finished listing objects for bucket {self._bucket} under
path {object_key}: Found {n_keys} in '
+ f'{duration} seconds')
+
return new_cache
def _get_object_key(full_path: str):
diff --git
a/collection_manager/collection_manager/services/history_manager/SolrIngestionHistory.py
b/collection_manager/collection_manager/services/history_manager/SolrIngestionHistory.py
index 6aff426..acf91d3 100644
---
a/collection_manager/collection_manager/services/history_manager/SolrIngestionHistory.py
+++
b/collection_manager/collection_manager/services/history_manager/SolrIngestionHistory.py
@@ -15,7 +15,7 @@
import hashlib
import logging
-
+from datetime import datetime
import pysolr
import requests
from collection_manager.services.history_manager.IngestionHistory import
(IngestionHistory, IngestionHistoryBuilder)
@@ -82,6 +82,20 @@ class SolrIngestionHistory(IngestionHistory):
'latest_update_l': self._latest_ingested_file_update}])
self._solr_datasets.commit()
+ @run_in_executor
+ def _push_dataset(self, dataset_id, type, config):
+ if self._solr_datasets:
+ if len(self._solr_datasets.search(q=f'id:{dataset_id}')) == 0:
+ self._solr_datasets.add([{
+ 'id': dataset_id,
+ 'dataset_s': dataset_id,
+ 'latest_update_l': int(datetime.now().timestamp()),
+ 'store_type_s': type,
+ 'config': config,
+ 'source_s': 'collection_config'
+ }])
+ self._solr_datasets.commit()
+
def _get_latest_file_update(self):
results = self._solr_datasets.search(q=f"id:{self._dataset_id}")
if results:
@@ -141,12 +155,15 @@ class SolrIngestionHistory(IngestionHistory):
schema_endpoint =
f"{self._url_prefix}/{self._dataset_collection_name}/schema"
self._add_field(schema_endpoint, "dataset_s", "string")
self._add_field(schema_endpoint, "latest_update_l",
"TrieLongField")
+ self._add_field(schema_endpoint, "store_type_s", "string",
True)
+ self._add_field(schema_endpoint, "source_s", "string", True)
+ self._add_field(schema_endpoint, "config", "text_general",
True)
except requests.exceptions.RequestException as e:
logger.error(f"solr instance unreachable {self._solr_url}")
raise e
- def _add_field(self, schema_url, field_name, field_type):
+ def _add_field(self, schema_url, field_name, field_type, stored=False):
"""
Helper to add a string field in a solr schema
:param schema_url:
@@ -158,7 +175,7 @@ class SolrIngestionHistory(IngestionHistory):
"add-field": {
"name": field_name,
"type": field_type,
- "stored": False
+ "stored": stored
}
}
return self._req_session.post(schema_url,
data=str(add_field_payload).encode('utf-8'))
diff --git a/collection_manager/docker/Dockerfile
b/collection_manager/docker/Dockerfile
index 5c48215..993ce58 100644
--- a/collection_manager/docker/Dockerfile
+++ b/collection_manager/docker/Dockerfile
@@ -15,9 +15,9 @@
FROM python:3.8.13
-RUN apt-get update && apt-get install -y apt-transport-https gnupg2
-RUN curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key
add -
-RUN echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | tee -a
/etc/apt/sources.list.d/kubernetes.list
+RUN apt-get update && apt-get install -y apt-transport-https gnupg2
ca-certificates curl
+RUN mkdir -p -m 755 /etc/apt/keyrings && curl -fsSL
https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key | gpg --dearmor -o
/etc/apt/keyrings/kubernetes-apt-keyring.gpg
+RUN echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg]
https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /' | tee
/etc/apt/sources.list.d/kubernetes.list
RUN apt-get update && apt-get install -y kubectl
COPY VERSION.txt /VERSION.txt
diff --git a/collection_manager/requirements.txt
b/collection_manager/requirements.txt
index fbace6e..d768fec 100644
--- a/collection_manager/requirements.txt
+++ b/collection_manager/requirements.txt
@@ -14,7 +14,6 @@
# limitations under the License.
PyYAML==5.3.1
-pystache==0.5.4
pysolr==3.9.0
watchdog==0.10.2
requests==2.31.0
diff --git a/granule_ingester/docker/Dockerfile
b/granule_ingester/docker/Dockerfile
index 579d510..3bec3d1 100644
--- a/granule_ingester/docker/Dockerfile
+++ b/granule_ingester/docker/Dockerfile
@@ -31,7 +31,7 @@ COPY granule_ingester/granule_ingester /sdap/granule_ingester
COPY granule_ingester/docker/install_nexusproto.sh /sdap/install_nexusproto.sh
COPY granule_ingester/docker/entrypoint.sh /entrypoint.sh
-COPY granule_ingester/README.md README DISCLAIMER LICENSE.txt NOTICE /sdap/
+COPY granule_ingester/README.md README LICENSE.txt NOTICE /sdap/
ENV PYTHONUNBUFFERED=1 \
# prevents python creating .pyc files
@@ -79,4 +79,4 @@ RUN cd /common && python setup.py install
RUN pip install boto3==1.16.10
-ENTRYPOINT ["/bin/bash", "/entrypoint.sh"]
\ No newline at end of file
+ENTRYPOINT ["/bin/bash", "/entrypoint.sh"]
diff --git
a/granule_ingester/granule_ingester/processors/reading_processors/GridMultiVariableReadingProcessor.py
b/granule_ingester/granule_ingester/processors/reading_processors/GridMultiVariableReadingProcessor.py
index 035bb6b..bab04f6 100644
---
a/granule_ingester/granule_ingester/processors/reading_processors/GridMultiVariableReadingProcessor.py
+++
b/granule_ingester/granule_ingester/processors/reading_processors/GridMultiVariableReadingProcessor.py
@@ -55,8 +55,17 @@ class
GridMultiVariableReadingProcessor(TileReadingProcessor):
lat_subset =
ds[self.latitude][type(self)._slices_for_variable(ds[self.latitude],
dimensions_to_slices)]
lon_subset =
ds[self.longitude][type(self)._slices_for_variable(ds[self.longitude],
dimensions_to_slices)]
- lat_subset = np.ma.filled(np.squeeze(lat_subset), np.NaN)
- lon_subset = np.ma.filled(np.squeeze(lon_subset), np.NaN)
+
+ lat_subset = np.squeeze(lat_subset)
+ if lat_subset.shape == ():
+ lat_subset = np.expand_dims(lat_subset, 0)
+
+ lon_subset = np.squeeze(lon_subset)
+ if lon_subset.shape == ():
+ lon_subset = np.expand_dims(lon_subset, 0)
+
+ lat_subset = np.ma.filled(lat_subset, np.NaN)
+ lon_subset = np.ma.filled(lon_subset, np.NaN)
if not isinstance(self.variable, list):
raise ValueError(f'self.variable `{self.variable}` needs to be a
list. use GridReadingProcessor for single band Grid files.')
diff --git
a/granule_ingester/granule_ingester/processors/reading_processors/GridReadingProcessor.py
b/granule_ingester/granule_ingester/processors/reading_processors/GridReadingProcessor.py
index 73969e6..95bb7c6 100644
---
a/granule_ingester/granule_ingester/processors/reading_processors/GridReadingProcessor.py
+++
b/granule_ingester/granule_ingester/processors/reading_processors/GridReadingProcessor.py
@@ -36,15 +36,32 @@ class GridReadingProcessor(TileReadingProcessor):
data_variable = self.variable[0] if isinstance(self.variable, list)
else self.variable
new_tile = nexusproto.GridTile()
+ expand_axes = []
+
lat_subset =
ds[self.latitude][type(self)._slices_for_variable(ds[self.latitude],
dimensions_to_slices)]
lon_subset =
ds[self.longitude][type(self)._slices_for_variable(ds[self.longitude],
dimensions_to_slices)]
- lat_subset = np.ma.filled(np.squeeze(lat_subset), np.NaN)
- lon_subset = np.ma.filled(np.squeeze(lon_subset), np.NaN)
+
+ lat_subset = np.squeeze(lat_subset)
+ if lat_subset.shape == ():
+ lat_subset = np.expand_dims(lat_subset, 0)
+ expand_axes.append(0)
+
+ lon_subset = np.squeeze(lon_subset)
+ if lon_subset.shape == ():
+ lon_subset = np.expand_dims(lon_subset, 0)
+ expand_axes.append(1)
+
+ lat_subset = np.ma.filled(lat_subset, np.NaN)
+ lon_subset = np.ma.filled(lon_subset, np.NaN)
data_subset =
ds[data_variable][type(self)._slices_for_variable(ds[data_variable],
dimensions_to_slices)].data
data_subset = np.array(np.squeeze(data_subset))
+ if len(expand_axes) > 0:
+ data_subset = np.expand_dims(data_subset, tuple(expand_axes))
+
+
if self.depth:
depth_dim, depth_slice =
list(type(self)._slices_for_variable(ds[self.depth],
dimensions_to_slices).items())[0]
diff --git a/granule_ingester/granule_ingester/writers/CassandraStore.py
b/granule_ingester/granule_ingester/writers/CassandraStore.py
index ba2fd0d..297854f 100644
--- a/granule_ingester/granule_ingester/writers/CassandraStore.py
+++ b/granule_ingester/granule_ingester/writers/CassandraStore.py
@@ -86,9 +86,17 @@ class CassandraStore(DataStore):
def connect(self):
self._session = self._get_session()
- def __del__(self):
- if self._session:
- self._session.shutdown()
+ def close(self):
+ session: Session = self._session
+ if session is not None:
+ cluster = session.cluster
+
+ session.shutdown()
+ cluster.shutdown()
+
+ del cluster, session
+
+ self._session = None
@retry(stop=stop_after_attempt(5), wait=wait_exponential(multiplier=1,
min=1, max=12))
async def save_data(self, tile: NexusTile) -> None:
diff --git a/granule_ingester/granule_ingester/writers/DataStore.py
b/granule_ingester/granule_ingester/writers/DataStore.py
index 7ed3d6b..a5e39ab 100644
--- a/granule_ingester/granule_ingester/writers/DataStore.py
+++ b/granule_ingester/granule_ingester/writers/DataStore.py
@@ -23,6 +23,17 @@ from typing import List
class DataStore(HealthCheck, ABC):
+ @abstractmethod
+ def connect(self):
+ ...
+
+ @abstractmethod
+ def close(self):
+ ...
+
+ def __del__(self):
+ self.close()
+
@abstractmethod
def save_data(self, nexus_tile: nexusproto.NexusTile) -> None:
diff --git a/granule_ingester/granule_ingester/writers/MetadataStore.py
b/granule_ingester/granule_ingester/writers/MetadataStore.py
index 8a97566..e1e317b 100644
--- a/granule_ingester/granule_ingester/writers/MetadataStore.py
+++ b/granule_ingester/granule_ingester/writers/MetadataStore.py
@@ -18,6 +18,8 @@ from abc import ABC, abstractmethod
from nexusproto import DataTile_pb2 as nexusproto
from granule_ingester.healthcheck import HealthCheck
+from asyncio import AbstractEventLoop
+
from typing import List
@@ -31,3 +33,14 @@ class MetadataStore(HealthCheck, ABC):
def save_batch(self, tiles: List[nexusproto.NexusTile]) -> None:
pass
+ @abstractmethod
+ def connect(self, loop: AbstractEventLoop = None) -> None:
+ pass
+
+ @abstractmethod
+ def close(self) -> None:
+ pass
+
+ def __del__(self):
+ self.close()
+
diff --git a/granule_ingester/granule_ingester/writers/SolrStore.py
b/granule_ingester/granule_ingester/writers/SolrStore.py
index 2dac038..963e4bc 100644
--- a/granule_ingester/granule_ingester/writers/SolrStore.py
+++ b/granule_ingester/granule_ingester/writers/SolrStore.py
@@ -21,7 +21,7 @@ import logging
from asyncio import AbstractEventLoop
from datetime import datetime
from pathlib import Path
-from typing import Dict, List, Union
+from typing import Dict, List, Union, Tuple, Optional
import pysolr
from kazoo.exceptions import NoNodeError
@@ -51,7 +51,8 @@ class SolrStore(MetadataStore):
self.geo_precision: int = 3
self._collection: str = "nexustiles"
self.log: logging.Logger = logging.getLogger(__name__)
- self._solr = None
+ self._solr: Optional[pysolr.Solr] = None
+ self._zk: Optional[pysolr.ZooKeeper] = None
def _get_collections(self, zk, parent_nodes):
"""
@@ -85,23 +86,33 @@ class SolrStore(MetadataStore):
collections.update(json.loads(zk.zk.get(f"{parent_node}/{c}/state.json")[0].decode("utf-8")))
zk.collections = collections
- def _get_connection(self) -> pysolr.Solr:
+ def _get_connection(self) -> Tuple[pysolr.Solr, Union[pysolr.ZooKeeper,
None]]:
if self._zk_url:
zk = pysolr.ZooKeeper(f"{self._zk_url}")
self._set_solr_status(zk)
- return pysolr.SolrCloud(zk, self._collection, always_commit=True)
+ return pysolr.SolrCloud(zk, self._collection, always_commit=True),
zk
elif self._solr_url:
- return pysolr.Solr(f'{self._solr_url}/solr/{self._collection}',
always_commit=True)
+ return pysolr.Solr(f'{self._solr_url}/solr/{self._collection}',
always_commit=True), None
else:
raise RuntimeError("You must provide either solr_host or
zookeeper_host.")
def connect(self, loop: AbstractEventLoop = None):
- self._solr = self._get_connection()
+ self._solr, self._zk = self._get_connection()
+
+ def close(self):
+ if self._solr is not None:
+ self._solr.get_session().close()
+
+ if self._zk is not None:
+ self._zk.zk.stop()
+ self._zk.zk.close()
async def health_check(self):
try:
- connection = self._get_connection()
+ connection, _ = self._get_connection()
connection.ping()
+
+ self.close()
except pysolr.SolrError:
raise SolrFailedHealthCheckError("Cannot connect to Solr!")
except NoNodeError:
diff --git a/granule_ingester/poetry.lock b/granule_ingester/poetry.lock
index 203bb18..1545e32 100644
--- a/granule_ingester/poetry.lock
+++ b/granule_ingester/poetry.lock
@@ -1,4 +1,4 @@
-# This file is automatically @generated by Poetry 1.8.1 and should not be
changed by hand.
+# This file is automatically @generated by Poetry 1.8.3 and should not be
changed by hand.
[[package]]
name = "aio-pika"
@@ -57,87 +57,87 @@ boto3 = ["boto3 (==1.16.52)"]
[[package]]
name = "aiohttp"
-version = "3.9.3"
+version = "3.9.5"
description = "Async http client/server framework (asyncio)"
optional = false
python-versions = ">=3.8"
files = [
- {file = "aiohttp-3.9.3-cp310-cp310-macosx_10_9_universal2.whl", hash =
"sha256:939677b61f9d72a4fa2a042a5eee2a99a24001a67c13da113b2e30396567db54"},
- {file = "aiohttp-3.9.3-cp310-cp310-macosx_10_9_x86_64.whl", hash =
"sha256:1f5cd333fcf7590a18334c90f8c9147c837a6ec8a178e88d90a9b96ea03194cc"},
- {file = "aiohttp-3.9.3-cp310-cp310-macosx_11_0_arm64.whl", hash =
"sha256:82e6aa28dd46374f72093eda8bcd142f7771ee1eb9d1e223ff0fa7177a96b4a5"},
- {file =
"aiohttp-3.9.3-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
hash =
"sha256:f56455b0c2c7cc3b0c584815264461d07b177f903a04481dfc33e08a89f0c26b"},
- {file =
"aiohttp-3.9.3-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl",
hash =
"sha256:bca77a198bb6e69795ef2f09a5f4c12758487f83f33d63acde5f0d4919815768"},
- {file =
"aiohttp-3.9.3-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash
= "sha256:e083c285857b78ee21a96ba1eb1b5339733c3563f72980728ca2b08b53826ca5"},
- {file =
"aiohttp-3.9.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
hash =
"sha256:ab40e6251c3873d86ea9b30a1ac6d7478c09277b32e14745d0d3c6e76e3c7e29"},
- {file =
"aiohttp-3.9.3-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl",
hash =
"sha256:df822ee7feaaeffb99c1a9e5e608800bd8eda6e5f18f5cfb0dc7eeb2eaa6bbec"},
- {file = "aiohttp-3.9.3-cp310-cp310-musllinux_1_1_aarch64.whl", hash =
"sha256:acef0899fea7492145d2bbaaaec7b345c87753168589cc7faf0afec9afe9b747"},
- {file = "aiohttp-3.9.3-cp310-cp310-musllinux_1_1_i686.whl", hash =
"sha256:cd73265a9e5ea618014802ab01babf1940cecb90c9762d8b9e7d2cc1e1969ec6"},
- {file = "aiohttp-3.9.3-cp310-cp310-musllinux_1_1_ppc64le.whl", hash =
"sha256:a78ed8a53a1221393d9637c01870248a6f4ea5b214a59a92a36f18151739452c"},
- {file = "aiohttp-3.9.3-cp310-cp310-musllinux_1_1_s390x.whl", hash =
"sha256:6b0e029353361f1746bac2e4cc19b32f972ec03f0f943b390c4ab3371840aabf"},
- {file = "aiohttp-3.9.3-cp310-cp310-musllinux_1_1_x86_64.whl", hash =
"sha256:7cf5c9458e1e90e3c390c2639f1017a0379a99a94fdfad3a1fd966a2874bba52"},
- {file = "aiohttp-3.9.3-cp310-cp310-win32.whl", hash =
"sha256:3e59c23c52765951b69ec45ddbbc9403a8761ee6f57253250c6e1536cacc758b"},
- {file = "aiohttp-3.9.3-cp310-cp310-win_amd64.whl", hash =
"sha256:055ce4f74b82551678291473f66dc9fb9048a50d8324278751926ff0ae7715e5"},
- {file = "aiohttp-3.9.3-cp311-cp311-macosx_10_9_universal2.whl", hash =
"sha256:6b88f9386ff1ad91ace19d2a1c0225896e28815ee09fc6a8932fded8cda97c3d"},
- {file = "aiohttp-3.9.3-cp311-cp311-macosx_10_9_x86_64.whl", hash =
"sha256:c46956ed82961e31557b6857a5ca153c67e5476972e5f7190015018760938da2"},
- {file = "aiohttp-3.9.3-cp311-cp311-macosx_11_0_arm64.whl", hash =
"sha256:07b837ef0d2f252f96009e9b8435ec1fef68ef8b1461933253d318748ec1acdc"},
- {file =
"aiohttp-3.9.3-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
hash =
"sha256:dad46e6f620574b3b4801c68255492e0159d1712271cc99d8bdf35f2043ec266"},
- {file =
"aiohttp-3.9.3-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl",
hash =
"sha256:5ed3e046ea7b14938112ccd53d91c1539af3e6679b222f9469981e3dac7ba1ce"},
- {file =
"aiohttp-3.9.3-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash
= "sha256:039df344b45ae0b34ac885ab5b53940b174530d4dd8a14ed8b0e2155b9dddccb"},
- {file =
"aiohttp-3.9.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
hash =
"sha256:7943c414d3a8d9235f5f15c22ace69787c140c80b718dcd57caaade95f7cd93b"},
- {file =
"aiohttp-3.9.3-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl",
hash =
"sha256:84871a243359bb42c12728f04d181a389718710129b36b6aad0fc4655a7647d4"},
- {file = "aiohttp-3.9.3-cp311-cp311-musllinux_1_1_aarch64.whl", hash =
"sha256:5eafe2c065df5401ba06821b9a054d9cb2848867f3c59801b5d07a0be3a380ae"},
- {file = "aiohttp-3.9.3-cp311-cp311-musllinux_1_1_i686.whl", hash =
"sha256:9d3c9b50f19704552f23b4eaea1fc082fdd82c63429a6506446cbd8737823da3"},
- {file = "aiohttp-3.9.3-cp311-cp311-musllinux_1_1_ppc64le.whl", hash =
"sha256:f033d80bc6283092613882dfe40419c6a6a1527e04fc69350e87a9df02bbc283"},
- {file = "aiohttp-3.9.3-cp311-cp311-musllinux_1_1_s390x.whl", hash =
"sha256:2c895a656dd7e061b2fd6bb77d971cc38f2afc277229ce7dd3552de8313a483e"},
- {file = "aiohttp-3.9.3-cp311-cp311-musllinux_1_1_x86_64.whl", hash =
"sha256:1f5a71d25cd8106eab05f8704cd9167b6e5187bcdf8f090a66c6d88b634802b4"},
- {file = "aiohttp-3.9.3-cp311-cp311-win32.whl", hash =
"sha256:50fca156d718f8ced687a373f9e140c1bb765ca16e3d6f4fe116e3df7c05b2c5"},
- {file = "aiohttp-3.9.3-cp311-cp311-win_amd64.whl", hash =
"sha256:5fe9ce6c09668063b8447f85d43b8d1c4e5d3d7e92c63173e6180b2ac5d46dd8"},
- {file = "aiohttp-3.9.3-cp312-cp312-macosx_10_9_universal2.whl", hash =
"sha256:38a19bc3b686ad55804ae931012f78f7a534cce165d089a2059f658f6c91fa60"},
- {file = "aiohttp-3.9.3-cp312-cp312-macosx_10_9_x86_64.whl", hash =
"sha256:770d015888c2a598b377bd2f663adfd947d78c0124cfe7b959e1ef39f5b13869"},
- {file = "aiohttp-3.9.3-cp312-cp312-macosx_11_0_arm64.whl", hash =
"sha256:ee43080e75fc92bf36219926c8e6de497f9b247301bbf88c5c7593d931426679"},
- {file =
"aiohttp-3.9.3-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
hash =
"sha256:52df73f14ed99cee84865b95a3d9e044f226320a87af208f068ecc33e0c35b96"},
- {file =
"aiohttp-3.9.3-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl",
hash =
"sha256:dc9b311743a78043b26ffaeeb9715dc360335e5517832f5a8e339f8a43581e4d"},
- {file =
"aiohttp-3.9.3-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash
= "sha256:b955ed993491f1a5da7f92e98d5dad3c1e14dc175f74517c4e610b1f2456fb11"},
- {file =
"aiohttp-3.9.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
hash =
"sha256:504b6981675ace64c28bf4a05a508af5cde526e36492c98916127f5a02354d53"},
- {file =
"aiohttp-3.9.3-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl",
hash =
"sha256:a6fe5571784af92b6bc2fda8d1925cccdf24642d49546d3144948a6a1ed58ca5"},
- {file = "aiohttp-3.9.3-cp312-cp312-musllinux_1_1_aarch64.whl", hash =
"sha256:ba39e9c8627edc56544c8628cc180d88605df3892beeb2b94c9bc857774848ca"},
- {file = "aiohttp-3.9.3-cp312-cp312-musllinux_1_1_i686.whl", hash =
"sha256:e5e46b578c0e9db71d04c4b506a2121c0cb371dd89af17a0586ff6769d4c58c1"},
- {file = "aiohttp-3.9.3-cp312-cp312-musllinux_1_1_ppc64le.whl", hash =
"sha256:938a9653e1e0c592053f815f7028e41a3062e902095e5a7dc84617c87267ebd5"},
- {file = "aiohttp-3.9.3-cp312-cp312-musllinux_1_1_s390x.whl", hash =
"sha256:c3452ea726c76e92f3b9fae4b34a151981a9ec0a4847a627c43d71a15ac32aa6"},
- {file = "aiohttp-3.9.3-cp312-cp312-musllinux_1_1_x86_64.whl", hash =
"sha256:ff30218887e62209942f91ac1be902cc80cddb86bf00fbc6783b7a43b2bea26f"},
- {file = "aiohttp-3.9.3-cp312-cp312-win32.whl", hash =
"sha256:38f307b41e0bea3294a9a2a87833191e4bcf89bb0365e83a8be3a58b31fb7f38"},
- {file = "aiohttp-3.9.3-cp312-cp312-win_amd64.whl", hash =
"sha256:b791a3143681a520c0a17e26ae7465f1b6f99461a28019d1a2f425236e6eedb5"},
- {file = "aiohttp-3.9.3-cp38-cp38-macosx_10_9_universal2.whl", hash =
"sha256:0ed621426d961df79aa3b963ac7af0d40392956ffa9be022024cd16297b30c8c"},
- {file = "aiohttp-3.9.3-cp38-cp38-macosx_10_9_x86_64.whl", hash =
"sha256:7f46acd6a194287b7e41e87957bfe2ad1ad88318d447caf5b090012f2c5bb528"},
- {file = "aiohttp-3.9.3-cp38-cp38-macosx_11_0_arm64.whl", hash =
"sha256:feeb18a801aacb098220e2c3eea59a512362eb408d4afd0c242044c33ad6d542"},
- {file =
"aiohttp-3.9.3-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
hash =
"sha256:f734e38fd8666f53da904c52a23ce517f1b07722118d750405af7e4123933511"},
- {file =
"aiohttp-3.9.3-cp38-cp38-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl",
hash =
"sha256:b40670ec7e2156d8e57f70aec34a7216407848dfe6c693ef131ddf6e76feb672"},
- {file =
"aiohttp-3.9.3-cp38-cp38-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash =
"sha256:fdd215b7b7fd4a53994f238d0f46b7ba4ac4c0adb12452beee724ddd0743ae5d"},
- {file =
"aiohttp-3.9.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash
= "sha256:017a21b0df49039c8f46ca0971b3a7fdc1f56741ab1240cb90ca408049766168"},
- {file =
"aiohttp-3.9.3-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl",
hash =
"sha256:e99abf0bba688259a496f966211c49a514e65afa9b3073a1fcee08856e04425b"},
- {file = "aiohttp-3.9.3-cp38-cp38-musllinux_1_1_aarch64.whl", hash =
"sha256:648056db9a9fa565d3fa851880f99f45e3f9a771dd3ff3bb0c048ea83fb28194"},
- {file = "aiohttp-3.9.3-cp38-cp38-musllinux_1_1_i686.whl", hash =
"sha256:8aacb477dc26797ee089721536a292a664846489c49d3ef9725f992449eda5a8"},
- {file = "aiohttp-3.9.3-cp38-cp38-musllinux_1_1_ppc64le.whl", hash =
"sha256:522a11c934ea660ff8953eda090dcd2154d367dec1ae3c540aff9f8a5c109ab4"},
- {file = "aiohttp-3.9.3-cp38-cp38-musllinux_1_1_s390x.whl", hash =
"sha256:5bce0dc147ca85caa5d33debc4f4d65e8e8b5c97c7f9f660f215fa74fc49a321"},
- {file = "aiohttp-3.9.3-cp38-cp38-musllinux_1_1_x86_64.whl", hash =
"sha256:4b4af9f25b49a7be47c0972139e59ec0e8285c371049df1a63b6ca81fdd216a2"},
- {file = "aiohttp-3.9.3-cp38-cp38-win32.whl", hash =
"sha256:298abd678033b8571995650ccee753d9458dfa0377be4dba91e4491da3f2be63"},
- {file = "aiohttp-3.9.3-cp38-cp38-win_amd64.whl", hash =
"sha256:69361bfdca5468c0488d7017b9b1e5ce769d40b46a9f4a2eed26b78619e9396c"},
- {file = "aiohttp-3.9.3-cp39-cp39-macosx_10_9_universal2.whl", hash =
"sha256:0fa43c32d1643f518491d9d3a730f85f5bbaedcbd7fbcae27435bb8b7a061b29"},
- {file = "aiohttp-3.9.3-cp39-cp39-macosx_10_9_x86_64.whl", hash =
"sha256:835a55b7ca49468aaaac0b217092dfdff370e6c215c9224c52f30daaa735c1c1"},
- {file = "aiohttp-3.9.3-cp39-cp39-macosx_11_0_arm64.whl", hash =
"sha256:06a9b2c8837d9a94fae16c6223acc14b4dfdff216ab9b7202e07a9a09541168f"},
- {file =
"aiohttp-3.9.3-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
hash =
"sha256:abf151955990d23f84205286938796c55ff11bbfb4ccfada8c9c83ae6b3c89a3"},
- {file =
"aiohttp-3.9.3-cp39-cp39-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl",
hash =
"sha256:59c26c95975f26e662ca78fdf543d4eeaef70e533a672b4113dd888bd2423caa"},
- {file =
"aiohttp-3.9.3-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash =
"sha256:f95511dd5d0e05fd9728bac4096319f80615aaef4acbecb35a990afebe953b0e"},
- {file =
"aiohttp-3.9.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash
= "sha256:595f105710293e76b9dc09f52e0dd896bd064a79346234b521f6b968ffdd8e58"},
- {file =
"aiohttp-3.9.3-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl",
hash =
"sha256:c7c8b816c2b5af5c8a436df44ca08258fc1a13b449393a91484225fcb7545533"},
- {file = "aiohttp-3.9.3-cp39-cp39-musllinux_1_1_aarch64.whl", hash =
"sha256:f1088fa100bf46e7b398ffd9904f4808a0612e1d966b4aa43baa535d1b6341eb"},
- {file = "aiohttp-3.9.3-cp39-cp39-musllinux_1_1_i686.whl", hash =
"sha256:f59dfe57bb1ec82ac0698ebfcdb7bcd0e99c255bd637ff613760d5f33e7c81b3"},
- {file = "aiohttp-3.9.3-cp39-cp39-musllinux_1_1_ppc64le.whl", hash =
"sha256:361a1026c9dd4aba0109e4040e2aecf9884f5cfe1b1b1bd3d09419c205e2e53d"},
- {file = "aiohttp-3.9.3-cp39-cp39-musllinux_1_1_s390x.whl", hash =
"sha256:363afe77cfcbe3a36353d8ea133e904b108feea505aa4792dad6585a8192c55a"},
- {file = "aiohttp-3.9.3-cp39-cp39-musllinux_1_1_x86_64.whl", hash =
"sha256:8e2c45c208c62e955e8256949eb225bd8b66a4c9b6865729a786f2aa79b72e9d"},
- {file = "aiohttp-3.9.3-cp39-cp39-win32.whl", hash =
"sha256:f7217af2e14da0856e082e96ff637f14ae45c10a5714b63c77f26d8884cf1051"},
- {file = "aiohttp-3.9.3-cp39-cp39-win_amd64.whl", hash =
"sha256:27468897f628c627230dba07ec65dc8d0db566923c48f29e084ce382119802bc"},
- {file = "aiohttp-3.9.3.tar.gz", hash =
"sha256:90842933e5d1ff760fae6caca4b2b3edba53ba8f4b71e95dacf2818a2aca06f7"},
+ {file = "aiohttp-3.9.5-cp310-cp310-macosx_10_9_universal2.whl", hash =
"sha256:fcde4c397f673fdec23e6b05ebf8d4751314fa7c24f93334bf1f1364c1c69ac7"},
+ {file = "aiohttp-3.9.5-cp310-cp310-macosx_10_9_x86_64.whl", hash =
"sha256:5d6b3f1fabe465e819aed2c421a6743d8debbde79b6a8600739300630a01bf2c"},
+ {file = "aiohttp-3.9.5-cp310-cp310-macosx_11_0_arm64.whl", hash =
"sha256:6ae79c1bc12c34082d92bf9422764f799aee4746fd7a392db46b7fd357d4a17a"},
+ {file =
"aiohttp-3.9.5-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
hash =
"sha256:4d3ebb9e1316ec74277d19c5f482f98cc65a73ccd5430540d6d11682cd857430"},
+ {file =
"aiohttp-3.9.5-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl",
hash =
"sha256:84dabd95154f43a2ea80deffec9cb44d2e301e38a0c9d331cc4aa0166fe28ae3"},
+ {file =
"aiohttp-3.9.5-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash
= "sha256:c8a02fbeca6f63cb1f0475c799679057fc9268b77075ab7cf3f1c600e81dd46b"},
+ {file =
"aiohttp-3.9.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
hash =
"sha256:c26959ca7b75ff768e2776d8055bf9582a6267e24556bb7f7bd29e677932be72"},
+ {file =
"aiohttp-3.9.5-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl",
hash =
"sha256:714d4e5231fed4ba2762ed489b4aec07b2b9953cf4ee31e9871caac895a839c0"},
+ {file = "aiohttp-3.9.5-cp310-cp310-musllinux_1_1_aarch64.whl", hash =
"sha256:e7a6a8354f1b62e15d48e04350f13e726fa08b62c3d7b8401c0a1314f02e3558"},
+ {file = "aiohttp-3.9.5-cp310-cp310-musllinux_1_1_i686.whl", hash =
"sha256:c413016880e03e69d166efb5a1a95d40f83d5a3a648d16486592c49ffb76d0db"},
+ {file = "aiohttp-3.9.5-cp310-cp310-musllinux_1_1_ppc64le.whl", hash =
"sha256:ff84aeb864e0fac81f676be9f4685f0527b660f1efdc40dcede3c251ef1e867f"},
+ {file = "aiohttp-3.9.5-cp310-cp310-musllinux_1_1_s390x.whl", hash =
"sha256:ad7f2919d7dac062f24d6f5fe95d401597fbb015a25771f85e692d043c9d7832"},
+ {file = "aiohttp-3.9.5-cp310-cp310-musllinux_1_1_x86_64.whl", hash =
"sha256:702e2c7c187c1a498a4e2b03155d52658fdd6fda882d3d7fbb891a5cf108bb10"},
+ {file = "aiohttp-3.9.5-cp310-cp310-win32.whl", hash =
"sha256:67c3119f5ddc7261d47163ed86d760ddf0e625cd6246b4ed852e82159617b5fb"},
+ {file = "aiohttp-3.9.5-cp310-cp310-win_amd64.whl", hash =
"sha256:471f0ef53ccedec9995287f02caf0c068732f026455f07db3f01a46e49d76bbb"},
+ {file = "aiohttp-3.9.5-cp311-cp311-macosx_10_9_universal2.whl", hash =
"sha256:e0ae53e33ee7476dd3d1132f932eeb39bf6125083820049d06edcdca4381f342"},
+ {file = "aiohttp-3.9.5-cp311-cp311-macosx_10_9_x86_64.whl", hash =
"sha256:c088c4d70d21f8ca5c0b8b5403fe84a7bc8e024161febdd4ef04575ef35d474d"},
+ {file = "aiohttp-3.9.5-cp311-cp311-macosx_11_0_arm64.whl", hash =
"sha256:639d0042b7670222f33b0028de6b4e2fad6451462ce7df2af8aee37dcac55424"},
+ {file =
"aiohttp-3.9.5-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
hash =
"sha256:f26383adb94da5e7fb388d441bf09c61e5e35f455a3217bfd790c6b6bc64b2ee"},
+ {file =
"aiohttp-3.9.5-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl",
hash =
"sha256:66331d00fb28dc90aa606d9a54304af76b335ae204d1836f65797d6fe27f1ca2"},
+ {file =
"aiohttp-3.9.5-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash
= "sha256:4ff550491f5492ab5ed3533e76b8567f4b37bd2995e780a1f46bca2024223233"},
+ {file =
"aiohttp-3.9.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
hash =
"sha256:f22eb3a6c1080d862befa0a89c380b4dafce29dc6cd56083f630073d102eb595"},
+ {file =
"aiohttp-3.9.5-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl",
hash =
"sha256:a81b1143d42b66ffc40a441379387076243ef7b51019204fd3ec36b9f69e77d6"},
+ {file = "aiohttp-3.9.5-cp311-cp311-musllinux_1_1_aarch64.whl", hash =
"sha256:f64fd07515dad67f24b6ea4a66ae2876c01031de91c93075b8093f07c0a2d93d"},
+ {file = "aiohttp-3.9.5-cp311-cp311-musllinux_1_1_i686.whl", hash =
"sha256:93e22add827447d2e26d67c9ac0161756007f152fdc5210277d00a85f6c92323"},
+ {file = "aiohttp-3.9.5-cp311-cp311-musllinux_1_1_ppc64le.whl", hash =
"sha256:55b39c8684a46e56ef8c8d24faf02de4a2b2ac60d26cee93bc595651ff545de9"},
+ {file = "aiohttp-3.9.5-cp311-cp311-musllinux_1_1_s390x.whl", hash =
"sha256:4715a9b778f4293b9f8ae7a0a7cef9829f02ff8d6277a39d7f40565c737d3771"},
+ {file = "aiohttp-3.9.5-cp311-cp311-musllinux_1_1_x86_64.whl", hash =
"sha256:afc52b8d969eff14e069a710057d15ab9ac17cd4b6753042c407dcea0e40bf75"},
+ {file = "aiohttp-3.9.5-cp311-cp311-win32.whl", hash =
"sha256:b3df71da99c98534be076196791adca8819761f0bf6e08e07fd7da25127150d6"},
+ {file = "aiohttp-3.9.5-cp311-cp311-win_amd64.whl", hash =
"sha256:88e311d98cc0bf45b62fc46c66753a83445f5ab20038bcc1b8a1cc05666f428a"},
+ {file = "aiohttp-3.9.5-cp312-cp312-macosx_10_9_universal2.whl", hash =
"sha256:c7a4b7a6cf5b6eb11e109a9755fd4fda7d57395f8c575e166d363b9fc3ec4678"},
+ {file = "aiohttp-3.9.5-cp312-cp312-macosx_10_9_x86_64.whl", hash =
"sha256:0a158704edf0abcac8ac371fbb54044f3270bdbc93e254a82b6c82be1ef08f3c"},
+ {file = "aiohttp-3.9.5-cp312-cp312-macosx_11_0_arm64.whl", hash =
"sha256:d153f652a687a8e95ad367a86a61e8d53d528b0530ef382ec5aaf533140ed00f"},
+ {file =
"aiohttp-3.9.5-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
hash =
"sha256:82a6a97d9771cb48ae16979c3a3a9a18b600a8505b1115cfe354dfb2054468b4"},
+ {file =
"aiohttp-3.9.5-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl",
hash =
"sha256:60cdbd56f4cad9f69c35eaac0fbbdf1f77b0ff9456cebd4902f3dd1cf096464c"},
+ {file =
"aiohttp-3.9.5-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash
= "sha256:8676e8fd73141ded15ea586de0b7cda1542960a7b9ad89b2b06428e97125d4fa"},
+ {file =
"aiohttp-3.9.5-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
hash =
"sha256:da00da442a0e31f1c69d26d224e1efd3a1ca5bcbf210978a2ca7426dfcae9f58"},
+ {file =
"aiohttp-3.9.5-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl",
hash =
"sha256:18f634d540dd099c262e9f887c8bbacc959847cfe5da7a0e2e1cf3f14dbf2daf"},
+ {file = "aiohttp-3.9.5-cp312-cp312-musllinux_1_1_aarch64.whl", hash =
"sha256:320e8618eda64e19d11bdb3bd04ccc0a816c17eaecb7e4945d01deee2a22f95f"},
+ {file = "aiohttp-3.9.5-cp312-cp312-musllinux_1_1_i686.whl", hash =
"sha256:2faa61a904b83142747fc6a6d7ad8fccff898c849123030f8e75d5d967fd4a81"},
+ {file = "aiohttp-3.9.5-cp312-cp312-musllinux_1_1_ppc64le.whl", hash =
"sha256:8c64a6dc3fe5db7b1b4d2b5cb84c4f677768bdc340611eca673afb7cf416ef5a"},
+ {file = "aiohttp-3.9.5-cp312-cp312-musllinux_1_1_s390x.whl", hash =
"sha256:393c7aba2b55559ef7ab791c94b44f7482a07bf7640d17b341b79081f5e5cd1a"},
+ {file = "aiohttp-3.9.5-cp312-cp312-musllinux_1_1_x86_64.whl", hash =
"sha256:c671dc117c2c21a1ca10c116cfcd6e3e44da7fcde37bf83b2be485ab377b25da"},
+ {file = "aiohttp-3.9.5-cp312-cp312-win32.whl", hash =
"sha256:5a7ee16aab26e76add4afc45e8f8206c95d1d75540f1039b84a03c3b3800dd59"},
+ {file = "aiohttp-3.9.5-cp312-cp312-win_amd64.whl", hash =
"sha256:5ca51eadbd67045396bc92a4345d1790b7301c14d1848feaac1d6a6c9289e888"},
+ {file = "aiohttp-3.9.5-cp38-cp38-macosx_10_9_universal2.whl", hash =
"sha256:694d828b5c41255e54bc2dddb51a9f5150b4eefa9886e38b52605a05d96566e8"},
+ {file = "aiohttp-3.9.5-cp38-cp38-macosx_10_9_x86_64.whl", hash =
"sha256:0605cc2c0088fcaae79f01c913a38611ad09ba68ff482402d3410bf59039bfb8"},
+ {file = "aiohttp-3.9.5-cp38-cp38-macosx_11_0_arm64.whl", hash =
"sha256:4558e5012ee03d2638c681e156461d37b7a113fe13970d438d95d10173d25f78"},
+ {file =
"aiohttp-3.9.5-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
hash =
"sha256:9dbc053ac75ccc63dc3a3cc547b98c7258ec35a215a92bd9f983e0aac95d3d5b"},
+ {file =
"aiohttp-3.9.5-cp38-cp38-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl",
hash =
"sha256:4109adee842b90671f1b689901b948f347325045c15f46b39797ae1bf17019de"},
+ {file =
"aiohttp-3.9.5-cp38-cp38-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash =
"sha256:a6ea1a5b409a85477fd8e5ee6ad8f0e40bf2844c270955e09360418cfd09abac"},
+ {file =
"aiohttp-3.9.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash
= "sha256:f3c2890ca8c59ee683fd09adf32321a40fe1cf164e3387799efb2acebf090c11"},
+ {file =
"aiohttp-3.9.5-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl",
hash =
"sha256:3916c8692dbd9d55c523374a3b8213e628424d19116ac4308e434dbf6d95bbdd"},
+ {file = "aiohttp-3.9.5-cp38-cp38-musllinux_1_1_aarch64.whl", hash =
"sha256:8d1964eb7617907c792ca00b341b5ec3e01ae8c280825deadbbd678447b127e1"},
+ {file = "aiohttp-3.9.5-cp38-cp38-musllinux_1_1_i686.whl", hash =
"sha256:d5ab8e1f6bee051a4bf6195e38a5c13e5e161cb7bad83d8854524798bd9fcd6e"},
+ {file = "aiohttp-3.9.5-cp38-cp38-musllinux_1_1_ppc64le.whl", hash =
"sha256:52c27110f3862a1afbcb2af4281fc9fdc40327fa286c4625dfee247c3ba90156"},
+ {file = "aiohttp-3.9.5-cp38-cp38-musllinux_1_1_s390x.whl", hash =
"sha256:7f64cbd44443e80094309875d4f9c71d0401e966d191c3d469cde4642bc2e031"},
+ {file = "aiohttp-3.9.5-cp38-cp38-musllinux_1_1_x86_64.whl", hash =
"sha256:8b4f72fbb66279624bfe83fd5eb6aea0022dad8eec62b71e7bf63ee1caadeafe"},
+ {file = "aiohttp-3.9.5-cp38-cp38-win32.whl", hash =
"sha256:6380c039ec52866c06d69b5c7aad5478b24ed11696f0e72f6b807cfb261453da"},
+ {file = "aiohttp-3.9.5-cp38-cp38-win_amd64.whl", hash =
"sha256:da22dab31d7180f8c3ac7c7635f3bcd53808f374f6aa333fe0b0b9e14b01f91a"},
+ {file = "aiohttp-3.9.5-cp39-cp39-macosx_10_9_universal2.whl", hash =
"sha256:1732102949ff6087589408d76cd6dea656b93c896b011ecafff418c9661dc4ed"},
+ {file = "aiohttp-3.9.5-cp39-cp39-macosx_10_9_x86_64.whl", hash =
"sha256:c6021d296318cb6f9414b48e6a439a7f5d1f665464da507e8ff640848ee2a58a"},
+ {file = "aiohttp-3.9.5-cp39-cp39-macosx_11_0_arm64.whl", hash =
"sha256:239f975589a944eeb1bad26b8b140a59a3a320067fb3cd10b75c3092405a1372"},
+ {file =
"aiohttp-3.9.5-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
hash =
"sha256:3b7b30258348082826d274504fbc7c849959f1989d86c29bc355107accec6cfb"},
+ {file =
"aiohttp-3.9.5-cp39-cp39-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl",
hash =
"sha256:cd2adf5c87ff6d8b277814a28a535b59e20bfea40a101db6b3bdca7e9926bc24"},
+ {file =
"aiohttp-3.9.5-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash =
"sha256:e9a3d838441bebcf5cf442700e3963f58b5c33f015341f9ea86dcd7d503c07e2"},
+ {file =
"aiohttp-3.9.5-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash
= "sha256:9e3a1ae66e3d0c17cf65c08968a5ee3180c5a95920ec2731f53343fac9bad106"},
+ {file =
"aiohttp-3.9.5-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.manylinux_2_17_i686.manylinux2014_i686.whl",
hash =
"sha256:9c69e77370cce2d6df5d12b4e12bdcca60c47ba13d1cbbc8645dd005a20b738b"},
+ {file = "aiohttp-3.9.5-cp39-cp39-musllinux_1_1_aarch64.whl", hash =
"sha256:0cbf56238f4bbf49dab8c2dc2e6b1b68502b1e88d335bea59b3f5b9f4c001475"},
+ {file = "aiohttp-3.9.5-cp39-cp39-musllinux_1_1_i686.whl", hash =
"sha256:d1469f228cd9ffddd396d9948b8c9cd8022b6d1bf1e40c6f25b0fb90b4f893ed"},
+ {file = "aiohttp-3.9.5-cp39-cp39-musllinux_1_1_ppc64le.whl", hash =
"sha256:45731330e754f5811c314901cebdf19dd776a44b31927fa4b4dbecab9e457b0c"},
+ {file = "aiohttp-3.9.5-cp39-cp39-musllinux_1_1_s390x.whl", hash =
"sha256:3fcb4046d2904378e3aeea1df51f697b0467f2aac55d232c87ba162709478c46"},
+ {file = "aiohttp-3.9.5-cp39-cp39-musllinux_1_1_x86_64.whl", hash =
"sha256:8cf142aa6c1a751fcb364158fd710b8a9be874b81889c2bd13aa8893197455e2"},
+ {file = "aiohttp-3.9.5-cp39-cp39-win32.whl", hash =
"sha256:7b179eea70833c8dee51ec42f3b4097bd6370892fa93f510f76762105568cf09"},
+ {file = "aiohttp-3.9.5-cp39-cp39-win_amd64.whl", hash =
"sha256:38d80498e2e169bc61418ff36170e0aad0cd268da8b38a17c4cf29d254a8b3f1"},
+ {file = "aiohttp-3.9.5.tar.gz", hash =
"sha256:edea7d15772ceeb29db4aff55e482d4bcfb6ae160ce144f2682de02f6d693551"},
]
[package.dependencies]
@@ -314,43 +314,51 @@ graph = ["gremlinpython (==3.3.4)"]
[[package]]
name = "certifi"
-version = "2024.2.2"
+version = "2024.6.2"
description = "Python package for providing Mozilla's CA Bundle."
optional = false
python-versions = ">=3.6"
files = [
- {file = "certifi-2024.2.2-py3-none-any.whl", hash =
"sha256:dc383c07b76109f368f6106eee2b593b04a011ea4d55f652c6ca24a754d1cdd1"},
- {file = "certifi-2024.2.2.tar.gz", hash =
"sha256:0569859f95fc761b18b45ef421b1290a0f65f147e92a1e5eb3e635f9a5e4e66f"},
+ {file = "certifi-2024.6.2-py3-none-any.whl", hash =
"sha256:ddc6c8ce995e6987e7faf5e3f1b02b302836a0e5d98ece18392cb1a36c72ad56"},
+ {file = "certifi-2024.6.2.tar.gz", hash =
"sha256:3cd43f1c6fa7dedc5899d69d3ad0398fd018ad1a17fba83ddaf78aa46c747516"},
]
[[package]]
name = "cftime"
-version = "1.6.3"
+version = "1.6.4"
description = "Time-handling functionality from netcdf4-python"
optional = false
python-versions = ">=3.8"
files = [
- {file = "cftime-1.6.3-cp310-cp310-macosx_10_9_x86_64.whl", hash =
"sha256:b62d42546fa5c914dfea5b15a9aaed2087ea1211cc36d08c374502ef95892038"},
- {file = "cftime-1.6.3-cp310-cp310-macosx_11_0_arm64.whl", hash =
"sha256:eb6dd70b2ccabfe1a14b7fbb0bbdce0418e71697094373c0d573c880790fa291"},
- {file =
"cftime-1.6.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash
= "sha256:f9878bfd8c1c3f24184ecbd528f739ba46ebaceaf1c8a24d348d7befb117a285"},
- {file = "cftime-1.6.3-cp310-cp310-win_amd64.whl", hash =
"sha256:3cf6e216a4c06f9a628cdf8e9c9d5e8097fb3eb02dd087dd14ab3b18478a7271"},
- {file = "cftime-1.6.3-cp311-cp311-macosx_10_9_x86_64.whl", hash =
"sha256:8d2c01456d9d7b46aa710a41d1c711a50d5ea259aff4a987d0e973d1093bc922"},
- {file = "cftime-1.6.3-cp311-cp311-macosx_11_0_arm64.whl", hash =
"sha256:80eb1170ce1639016f55760847f4aadd04b0312496c5bac2797e930914bba48d"},
- {file =
"cftime-1.6.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash
= "sha256:d87dadd0824262bdd7493babd2a44447da0a22175ded8ae9e060a3aebec7c5d7"},
- {file = "cftime-1.6.3-cp311-cp311-win_amd64.whl", hash =
"sha256:0a38eb9f5c733a23e1714bd3ef2762ed5acee34f127670f8fb4ad6464946f6b3"},
- {file = "cftime-1.6.3-cp312-cp312-macosx_10_9_x86_64.whl", hash =
"sha256:2d113a01ab924445e61d65c26bbd95bc08e4a22878d3b947064bba056c884c4a"},
- {file = "cftime-1.6.3-cp312-cp312-macosx_11_0_arm64.whl", hash =
"sha256:5f11685663a6af97418908060492a07663c16d42519c139ca03c2ffb1377fd25"},
- {file =
"cftime-1.6.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash
= "sha256:a98abb1d46d118e52b0611ce668a0b714b407be26177ef0581ecf5e95f894725"},
- {file = "cftime-1.6.3-cp312-cp312-win_amd64.whl", hash =
"sha256:4d6fbd5f41b322cfa7b0ac3aaadeceb4450100a164b5bccbbb9e7c5048489a88"},
- {file = "cftime-1.6.3-cp38-cp38-macosx_10_9_x86_64.whl", hash =
"sha256:bedb577bc8b8f3f10f5336c0792e5dae88605781890f50f36b45bb46907968e8"},
- {file = "cftime-1.6.3-cp38-cp38-macosx_11_0_arm64.whl", hash =
"sha256:022dabf1610cdd04a693e730fa8f71d307059717f29dba921e7486e553412bb4"},
- {file =
"cftime-1.6.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash =
"sha256:bbf782ab4ac0605bdec2b941952c897595613203942b7f8c2fccd17efa5147df"},
- {file = "cftime-1.6.3-cp38-cp38-win_amd64.whl", hash =
"sha256:9eb177a02db7cd84aa6962278e4bd2d3106a545de82e6aacd9404f1e153661db"},
- {file = "cftime-1.6.3-cp39-cp39-macosx_10_9_x86_64.whl", hash =
"sha256:3b86be8c2f254147be4ba88f12099466dde457a4a3a21de6c69d52a7224c13ae"},
- {file = "cftime-1.6.3-cp39-cp39-macosx_11_0_arm64.whl", hash =
"sha256:523b9a6bf03f5e36407979e248381d0fcab2d225b915bbde77d00c6dde192b90"},
- {file =
"cftime-1.6.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash =
"sha256:8a14d2c7d22fd2a6dfa6ad563283b6d6679f1df95e0ed8d14b8f284dad402887"},
- {file = "cftime-1.6.3-cp39-cp39-win_amd64.whl", hash =
"sha256:d9b00c2844c7a1701d8ede5336b6321dfee256ceab81a34a1aff0483d56891a6"},
- {file = "cftime-1.6.3.tar.gz", hash =
"sha256:d0a6b29f72a13f08e008b9becff247cc75c84acb213332ede18879c5b6aa4dfd"},
+ {file = "cftime-1.6.4-cp310-cp310-macosx_10_9_x86_64.whl", hash =
"sha256:ee70074df4bae0d9ee98f201cf5f11fd302791cf1cdeb73c34f685d6b632e17d"},
+ {file = "cftime-1.6.4-cp310-cp310-macosx_11_0_arm64.whl", hash =
"sha256:e5456fd58d4cc6b8d7b4932b749617ee142b62a52bc5d8e3c282ce69ce3a20ba"},
+ {file =
"cftime-1.6.4-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
hash =
"sha256:1289e08617be350a6b26c6e4352a0cb088625ac33d25e95110df549c26d6ab8e"},
+ {file =
"cftime-1.6.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash
= "sha256:18b132d9225b4a109929866200846c72302316db9069e2de3ec8d8ec377f567f"},
+ {file = "cftime-1.6.4-cp310-cp310-musllinux_1_2_x86_64.whl", hash =
"sha256:ca1a264570e68fbb611bba251641b8efd0cf88c0ad2dcab5fa784df264232b75"},
+ {file = "cftime-1.6.4-cp310-cp310-win_amd64.whl", hash =
"sha256:6fc82928cbf477bebf233f41914e64bff7b9e894c7f0c34170784a48250f8da7"},
+ {file = "cftime-1.6.4-cp311-cp311-macosx_10_9_x86_64.whl", hash =
"sha256:c1558d9b477bd29626cd8bfc89e736635f72887d1a993e2834ab579bba7abb8c"},
+ {file = "cftime-1.6.4-cp311-cp311-macosx_11_0_arm64.whl", hash =
"sha256:03494e7b66a2fbb6b04e364ab67185130dee0ced660abac5c1559070571d143d"},
+ {file =
"cftime-1.6.4-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
hash =
"sha256:4dcb2a01d4e614437582af33b36db4fb441b7666758482864827a1f037d2b639"},
+ {file =
"cftime-1.6.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash
= "sha256:0b47bf25195fb3889bbae34df0e80957eb69c48f66902f5d538c7a8ec34253f6"},
+ {file = "cftime-1.6.4-cp311-cp311-musllinux_1_2_x86_64.whl", hash =
"sha256:d4f2cc0d5c6ffba9c5b0fd1ecd0c7c1c426d0be7b8de1480e2a9fb857c1905e9"},
+ {file = "cftime-1.6.4-cp311-cp311-win_amd64.whl", hash =
"sha256:76b8f1e5d1e424accdf760a43e0a1793a7b640bab83cb067273d5c9dbb336c44"},
+ {file = "cftime-1.6.4-cp312-cp312-macosx_10_9_x86_64.whl", hash =
"sha256:3c349a91fa7ac9ec50118b04a8746bdea967bd2fc525d87c776003040b8d3392"},
+ {file = "cftime-1.6.4-cp312-cp312-macosx_11_0_arm64.whl", hash =
"sha256:588d073400798adc24ece759cd1cb24ef730f55d1f70e31a898e7686f9d763d8"},
+ {file =
"cftime-1.6.4-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
hash =
"sha256:6e07b91b488570573bbeb6f815656a8974d13d15b2279c82de2927f4f692bbcd"},
+ {file =
"cftime-1.6.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash
= "sha256:f92f2e405eeda47b30ab6231d8b7d136a55f21034d394f93ade322d356948654"},
+ {file = "cftime-1.6.4-cp312-cp312-musllinux_1_2_x86_64.whl", hash =
"sha256:567574df94d0de1101bb5da76e7fbc6eabfddeeb2eb8cf83286b3599a136bbf7"},
+ {file = "cftime-1.6.4-cp312-cp312-win_amd64.whl", hash =
"sha256:5b5ad7559a16bedadb66af8e417b6805f758acb57aa38d2730844dfc63a1e667"},
+ {file = "cftime-1.6.4-cp38-cp38-macosx_10_9_x86_64.whl", hash =
"sha256:c072fe9e09925af66a9473edf5752ca1890ba752e7c1935d1f0245ad48f0977c"},
+ {file = "cftime-1.6.4-cp38-cp38-macosx_11_0_arm64.whl", hash =
"sha256:c05a71389f53d6340cb365b60f028c08268c72401660b9ef76108dee9f1cb5b2"},
+ {file =
"cftime-1.6.4-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash
= "sha256:0edeb1cb019d8155b2513cffb96749c0d7d459370e69bdf03077e0bee214aed8"},
+ {file =
"cftime-1.6.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash =
"sha256:a8f05d5d6bb4137f9783fa61ad330030fcea8dcc6946dea69a27774edbe480e7"},
+ {file = "cftime-1.6.4-cp38-cp38-win_amd64.whl", hash =
"sha256:b32ac1278a2a111b066d5a1e6e5ce6f38c4c505993a6a3130873b56f99d7b56f"},
+ {file = "cftime-1.6.4-cp39-cp39-macosx_10_9_x86_64.whl", hash =
"sha256:c20f03e12af39534c3450bba376272803bfb850b5ce6433c839bfaa99f8d835a"},
+ {file = "cftime-1.6.4-cp39-cp39-macosx_11_0_arm64.whl", hash =
"sha256:90609b3c1a31a756a68ecdbc961a4ce0b22c1620f166c8dabfa3a4c337ac8b9e"},
+ {file =
"cftime-1.6.4-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash
= "sha256:bbe11ad73b2a0ddc79995da21459fc2a3fd6b1593ca73f00a60e4d81c3e230f3"},
+ {file =
"cftime-1.6.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash =
"sha256:25f043703e785de0bd7cd8222c0a53317e9aeb3dfc062588b05e6f3ebb007468"},
+ {file = "cftime-1.6.4-cp39-cp39-musllinux_1_2_x86_64.whl", hash =
"sha256:f9acc272df1022f24fe7dbe9de43fa5d8271985161df14549e4d8d28c90dc9ea"},
+ {file = "cftime-1.6.4-cp39-cp39-win_amd64.whl", hash =
"sha256:e8467b6fbf8dbfe0be8c04d61180765fdd3b9ab0fe51313a0bbf87e63634a3d8"},
]
[package.dependencies]
@@ -482,13 +490,13 @@ files = [
[[package]]
name = "elastic-transport"
-version = "8.12.0"
+version = "8.13.1"
description = "Transport classes and utilities shared among Python Elastic
client libraries"
optional = false
python-versions = ">=3.7"
files = [
- {file = "elastic-transport-8.12.0.tar.gz", hash =
"sha256:48839b942fcce199eece1558ecea6272e116c58da87ca8d495ef12eb61effaf7"},
- {file = "elastic_transport-8.12.0-py3-none-any.whl", hash =
"sha256:87d9dc9dee64a05235e7624ed7e6ab6e5ca16619aa7a6d22e853273b9f1cfbee"},
+ {file = "elastic_transport-8.13.1-py3-none-any.whl", hash =
"sha256:5d4bb6b8e9d74a9c16de274e91a5caf65a3a8d12876f1e99152975e15b2746fe"},
+ {file = "elastic_transport-8.13.1.tar.gz", hash =
"sha256:16339d392b4bbe86ad00b4bdeecff10edf516d32bc6c16053846625f2c6ea250"},
]
[package.dependencies]
@@ -496,7 +504,7 @@ certifi = "*"
urllib3 = ">=1.26.2,<3"
[package.extras]
-develop = ["aiohttp", "furo", "mock", "pytest", "pytest-asyncio",
"pytest-cov", "pytest-httpserver", "pytest-mock", "requests", "sphinx (>2)",
"sphinx-autodoc-typehints", "trustme"]
+develop = ["aiohttp", "furo", "httpx", "mock", "opentelemetry-api",
"opentelemetry-sdk", "orjson", "pytest", "pytest-asyncio", "pytest-cov",
"pytest-httpserver", "pytest-mock", "requests", "respx", "sphinx (>2)",
"sphinx-autodoc-typehints", "trustme"]
[[package]]
name = "elasticsearch"
@@ -630,13 +638,13 @@ six = "*"
[[package]]
name = "idna"
-version = "3.6"
+version = "3.7"
description = "Internationalized Domain Names in Applications (IDNA)"
optional = false
python-versions = ">=3.5"
files = [
- {file = "idna-3.6-py3-none-any.whl", hash =
"sha256:c05567e9c24a6b9faaa835c4821bad0590fbb9d5779e7caa6e1cc4978e7eb24f"},
- {file = "idna-3.6.tar.gz", hash =
"sha256:9ecdbbd083b06798ae1e86adcbfe8ab1479cf864e4ee30fe4e46a003d12491ca"},
+ {file = "idna-3.7-py3-none-any.whl", hash =
"sha256:82fee1fc78add43492d3a1898bfa6d8a904cc97d8427f683ed8e798d07761aa0"},
+ {file = "idna-3.7.tar.gz", hash =
"sha256:028ff3aadf0609c1fd278d8ea3089299412a7a8b9bd005dd08b9f8285bcb5cfc"},
]
[[package]]
@@ -853,13 +861,13 @@ files = [
[[package]]
name = "packaging"
-version = "23.2"
+version = "24.1"
description = "Core utilities for Python packages"
optional = false
-python-versions = ">=3.7"
+python-versions = ">=3.8"
files = [
- {file = "packaging-23.2-py3-none-any.whl", hash =
"sha256:8c491190033a9af7e1d931d0b5dacc2ef47509b34dd0de67ed209b5203fc88c7"},
- {file = "packaging-23.2.tar.gz", hash =
"sha256:048fb0e9405036518eaaf48a55953c750c11e1a1b68e0dd1a9d62ed0c092cfc5"},
+ {file = "packaging-24.1-py3-none-any.whl", hash =
"sha256:5b8f2217dbdbd2f7f384c41c628544e6d52f2d0f53c6d0c3ea61aa5d1d7ff124"},
+ {file = "packaging-24.1.tar.gz", hash =
"sha256:026ed72c8ed3fcce5bf8950572258698927fd1dbda10a5e981cdf0ac37f4f002"},
]
[[package]]
@@ -988,13 +996,13 @@ files = [
[[package]]
name = "requests"
-version = "2.31.0"
+version = "2.32.3"
description = "Python HTTP for Humans."
optional = false
-python-versions = ">=3.7"
+python-versions = ">=3.8"
files = [
- {file = "requests-2.31.0-py3-none-any.whl", hash =
"sha256:58cd2187c01e70e6e26505bca751777aa9f2ee0b7f4300988b709f44e013003f"},
- {file = "requests-2.31.0.tar.gz", hash =
"sha256:942c5a758f98d790eaed1a29cb6eefc7ffb0d1cf7af05c3d2791656dbd6ad1e1"},
+ {file = "requests-2.32.3-py3-none-any.whl", hash =
"sha256:70761cfe03c773ceb22aa2f671b4757976145175cdfca038c02654d061d6dcc6"},
+ {file = "requests-2.32.3.tar.gz", hash =
"sha256:55365417734eb18255590a9ff9eb97e9e1da868d4ccd6402399eaf68af20a760"},
]
[package.dependencies]
@@ -1100,13 +1108,13 @@ doc = ["reno", "sphinx", "tornado (>=4.5)"]
[[package]]
name = "typing-extensions"
-version = "4.10.0"
+version = "4.12.2"
description = "Backported and Experimental Type Hints for Python 3.8+"
optional = false
python-versions = ">=3.8"
files = [
- {file = "typing_extensions-4.10.0-py3-none-any.whl", hash =
"sha256:69b1a937c3a517342112fb4c6df7e72fc39a38e7891a5730ed4985b5214b5475"},
- {file = "typing_extensions-4.10.0.tar.gz", hash =
"sha256:b0abd7c89e8fb96f98db18d86106ff1d90ab692004eb746cf6eda2682f91b3cb"},
+ {file = "typing_extensions-4.12.2-py3-none-any.whl", hash =
"sha256:04e5ca0351e0f3f85c6853954072df659d0d13fac324d0072316b67d7794700d"},
+ {file = "typing_extensions-4.12.2.tar.gz", hash =
"sha256:1a7ead55c7e559dd4dee8856e3a88b41225abfe1ce8df57b7c13915fe121ffb8"},
]
[[package]]
diff --git a/granule_ingester/pyproject.toml b/granule_ingester/pyproject.toml
index 516e955..b90b01c 100644
--- a/granule_ingester/pyproject.toml
+++ b/granule_ingester/pyproject.toml
@@ -15,9 +15,9 @@
[tool.poetry]
name = "sdap_granule_ingester"
-version = "1.2.0"
+version = "1.3.0"
description = "Ingester component for Apache SDAP"
-authors = ["SDAP PPMC <[email protected]>"]
+authors = ["SDAP PMC <[email protected]>"]
license = "Apache-2.0"
readme = "README.md"
packages = [
diff --git
a/granule_ingester/tests/reading_processors/test_GridReadingProcessor.py
b/granule_ingester/tests/reading_processors/test_GridReadingProcessor.py
index ef78394..3123cc4 100644
--- a/granule_ingester/tests/reading_processors/test_GridReadingProcessor.py
+++ b/granule_ingester/tests/reading_processors/test_GridReadingProcessor.py
@@ -123,6 +123,61 @@ class TestReadMurData(unittest.TestCase):
masked_data =
np.ma.masked_invalid(from_shaped_array(output_tile.tile.grid_tile.variable_data))
self.assertEqual(50, np.ma.count(masked_data))
+ def test_single_width_coords(self):
+ reading_processor = GridReadingProcessor(['analysed_sst'], 'lat',
'lon', time='time')
+ granule_path = path.join(path.dirname(__file__),
'../granules/not_empty_mur.nc4')
+
+ input_tile = nexusproto.NexusTile()
+ input_tile.summary.granule = granule_path
+
+ dimensions_to_slices = {
+ 'time': slice(0, 1),
+ 'lat': slice(50, 51),
+ 'lon': slice(0, 5)
+ }
+ with xr.open_dataset(granule_path) as ds:
+ output_tile = reading_processor._generate_tile(ds,
dimensions_to_slices, input_tile)
+
+ self.assertEqual(granule_path, output_tile.summary.granule,
granule_path)
+ self.assertEqual(1451638800, output_tile.tile.grid_tile.time)
+ self.assertEqual([1, 5],
output_tile.tile.grid_tile.variable_data.shape)
+ self.assertEqual([1], output_tile.tile.grid_tile.latitude.shape)
+ self.assertEqual([5], output_tile.tile.grid_tile.longitude.shape)
+
+ input_tile = nexusproto.NexusTile()
+ input_tile.summary.granule = granule_path
+
+ dimensions_to_slices = {
+ 'time': slice(0, 1),
+ 'lat': slice(0, 10),
+ 'lon': slice(50, 51)
+ }
+ with xr.open_dataset(granule_path) as ds:
+ output_tile = reading_processor._generate_tile(ds,
dimensions_to_slices, input_tile)
+
+ self.assertEqual(granule_path, output_tile.summary.granule,
granule_path)
+ self.assertEqual(1451638800, output_tile.tile.grid_tile.time)
+ self.assertEqual([10, 1],
output_tile.tile.grid_tile.variable_data.shape)
+ self.assertEqual([10], output_tile.tile.grid_tile.latitude.shape)
+ self.assertEqual([1], output_tile.tile.grid_tile.longitude.shape)
+
+ input_tile = nexusproto.NexusTile()
+ input_tile.summary.granule = granule_path
+
+ dimensions_to_slices = {
+ 'time': slice(0, 1),
+ 'lat': slice(50, 51),
+ 'lon': slice(50, 51),
+ }
+ with xr.open_dataset(granule_path) as ds:
+ output_tile = reading_processor._generate_tile(ds,
dimensions_to_slices, input_tile)
+
+ self.assertEqual(granule_path, output_tile.summary.granule,
granule_path)
+ self.assertEqual(1451638800, output_tile.tile.grid_tile.time)
+ self.assertEqual([1, 1],
output_tile.tile.grid_tile.variable_data.shape)
+ self.assertEqual([1], output_tile.tile.grid_tile.latitude.shape)
+ self.assertEqual([1], output_tile.tile.grid_tile.longitude.shape)
+
class TestReadCcmpData(unittest.TestCase):
