This is an automated email from the ASF dual-hosted git repository.
eamonford pushed a change to branch dev
in repository https://gitbox.apache.org/repos/asf/incubator-sdap-ingester.git.
at 43cced9 Fix bad merge
This branch includes the following new commits:
new 33b9a5b files added
new 9cda328 update for deployment in test
new 82b249a update path to ease deployment in dev for now
new 3b44ccc new updates for tests in dev env
new 718e060 add contraints on group to match docker name constraints
new 80cb2ee remove & in command line, wrong syntax
new 2838d37 revise command line syntax with Popen
new 699f2db have path to runjob as in dev to ease deployment for now
new 7b45f2e add forgotten quotes
new bbe0eaa manage stdout properly with Popen
new b3084aa manage stdout and stderr in Popen
new add5c05 Check that logs directory exists
new fc9ccd1 wait for subprocess to be finished
new 8dd3269 update names of test files
new 8c12a55 manage logfile closing
new 4aae536 manage subprocess a synchronous process, kubernetes being in
charge of queuing
new e60ea32 manage work directory for runjob subprocesses'
new 245b3fd adjust work directories
new 7c8e031 correct typo
new 34651c6 log actual command line
new fb623e5 have absolute path in granule file list
new 963c99a change the names of config and granules list files with
suffixes to avoid conflicts in configmap names
new 0d781e6 change extension from yaml to yml
new cfb09da enable to parse nfs mount points to use them in kubernetes
new 211ee1b more log to debug
new 2ebb43b validation bugs
new bb2b562 organize as package with command line
new fabc3d4 rename main script
new 867e5d0 reorganize script and validation test
new 338b161 correct reference to package function
new b1d6e0e update doc
new 316d592 integration of single granule run, externalisation of
configuration
new 7e51330 add documentation on configuration
new 8d97fae review documentation
new bff0289 reorganized configuration
new 467424c change configuration path to user directory
new c3f31f5 have configuration in sys.prefix
new 6f34d30 version 0.1.0 with token saved in config directory
new 1cfd8d1 integration of granule processing in same python process,
create history of ingested granules
new 5dd07cc add feature o read local collection configuration file in
yaml format
new 27cd174 do not reingest files which are in the ingestion history
new 5772db2 manage priority among collections to be harvested
new 0f388e7 add documentation and file for pypi publication
new 486924f change github action event specification
new ff32e97 correct github action syntax
new f4866e6 add build and test steps in github action
new 57c1d2f update requirements
new 3e1d67b add install before test so that configuration files are
deployed
new b2c58b8 add github action steps to initialize credentials for tests
with google api
new 04e2d79 correct python shell syntax in github action
new acf6e05 update python shell syntax in github action
new e529573 add option -s for pytest because error logs in github action
advised to do so
new f9e3549 remove validation test with google api
new 82cc460 add logs
new 552626d add lfs support for github action checkout
new 0e00024 add step to generate package in github action
new c2325cd update wrong syntax in github action
new 1976605 change development status
new b07f30c make bdist instead of sdist
new d544f0c only support one python version in github action, does not
impact the supported version where the package runs
new ad9938d correct syntax error in github action
new 6796465 correct stuff in README file
new 0947ae3 bug corrections, documentation review for 0.2.0rc2
new 9253288 started solr ingestion history and automated dev build github
action
new c89d046 rename dev build release action and syntax error correction
new 9895535 remove test on solr history for test in github actions to work
new 56e9503 remove tests on solr history so that test a valid in github
action environment
new 0f86a58 deactivate tests with proper unittest decorator
new 80d9d31 use python snapshot release command instead of the default
which is for maven project
new 0ebe778 add lines at the end of github actions so that last line is
run
new 4a15fce change the suffix for a dev version
new 8e7b5db remove useless comment
new 0411f32 detail on dev buid in README
new 5ded066 start docker configuration
new 16efecf new history manager plugged to ingestion process
new 9ea22bc new test with kubernetes ingestion integration
new f7bf694 add feature to reconstruct nfs client path to be able to
compute md5sum from outside the kubernetes pod
new f20a219 deactivate tests which requires a local solr
new b7ff511 start implementation of ingestion cache with update time
management (instead of md5sum)
new 617d3a3 use update timestamp as a reference for history ingested
files, deduplicate history
new efa61fe ingestion history keys are now based on update time of the
files
new dc74eb8 skip history purge unit test
new 45d7c88 new command line to launch ingestion from a single command
new d97483c more log while reading config
new 87162fd add session attribute to module
new b063cb3 clean one pep style warning
new bb54159 add sleep temporarilly to check how volume is claimed in
kubernetes
new 6c6b158 add log message before sleep
new c26e41f started to prepare kubernetes configMap instead of native
configuration files
new 0205881 use the kubernetes configMap path for .ini config file
new e5dfc99 integration of collections ingester in kubernetes as a job
new fcb427d manage the deployment specfic confguration as an argument of
command run_collections
new 632dd1a add time range criteria on single collection ingestion
new 621a25a add time slides to single collection command line, accept
directory instead of regexp for the file path
new 40da4ff merge to prepare release 0.3.0
new 45e21d6 remove obsolete test file
new 52a93b9 add option --forward-processing
new 276e0e8 integrate data directory to kubernetes pods
new 164f462 have dev suffix for the version
new a333355 more log when getting colletion config file
new a1b4102 manage data volume in granule ingester pod
new 51e155b complete management of data volume for granule ingester
new 97811ad temporarilly for test, recreate data volume in granule
ingester jobs
new 4315361 add log (was print before, did not show when run in kubernete)
new 5135e1f end of day, correct bad log syntax
new 224a8a5 put back the print which sends the k8s job template
new ce670c9 correct the granule job template
new 6eb572c define data volume as shared k8s volume used by both
collection and granule ingester
new 4847cfa usage shared volume claim for data storage
new 51d7944 integrate solr history manager, latest ingested data
management remains to be done
new 5fd2eaf replace history configuration with solr
new f51ecb9 correct bugs after test on solr history
new 3b78099 add forward-processing capabilities on solr history
new ec5240f make solr schema update work
new 1df86b0 made solr history instanciation more robust by catching
excepions
new 1660d64 made solr history instanciation more robust by catching
excepions
new e0c0b70 change default conf to solr pods access
new 688cd58 force buffering to 1 for ingestion history file
new 1fedc94 correct typo in README
new 5f18529 correction on buffering history on files, started package for
order/config service
new d707d7d started implementation of order service
new f4fc5b1 start git synchornization function for configuration
new 8e2a167 synchronize ingestion orders with git repository
new b2a409c simplified obsolete unit test
new 8894a59 remove .idea from repository
new cd15dd4 Code restructuring, add rabbitmq support
new 423ad00 moved files
new f7f0ce7 a lot of cleanup and restructuring
new 01f1773 fix paths
new 57e6260 manage ingestion order list configured on a git repository
new ceae47c Publish messages to RabbitMQ, plus lots of refactoring
new 090576e clean more unused file, some information on how to launch in
readme, to be reviewed
new 84218f0 update the README with rabbitmq prerequisite command
new c07eb3d update the docker command to launch rabbitmq for development
prerequisite
new 1635e56 Code restructuring, add rabbitmq support
new 62b5c1a Add scheduler
new e4413a5 cleanup merge
new 06b43f7 cleanup
new 4c6aa4e removed flask
new f02ae2f remove ingestion order references
new ce0d737 Merge branch 'master' of
https://github.com/tloubrieu-jpl/incubator-sdap-nexus-ingestion-manager into dev
new 70b1455 initiate a configMap object with unit test (with bug)
new 11cccba add config_operator working on local dir or remote git repo
new 5dcb8af make the config-operator robust to syntactically wrong files,
by default synchornization is deactivated
new e936306 start to implement config-operator containers
new 825c93f move files in specific subdirectory, start k8s deployment,
does not work
new a2843b2 integrates with kubernetes
new 10ba61c run config operator as a k8s operator
new 17bd28c apply change requests by pull request #3 review
new a61fc11 some more cleaining in unit test and README
new 969adf6 allow multiple collections to use the same directory
new 43cced9 Fix bad merge
The 152 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
