(airavata-mft) 02/02: feat(bootstrap): when start mft, quit if consul does not start after 20 second wait
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
commit 3ccbfa2a5bc30e50bb385297ce82e809331967bf
Author: Shivang Mishra
AuthorDate: Wed Apr 3 14:03:42 2024 +0530
feat(bootstrap): when start mft, quit if consul does not start after 20
second wait
---
python-cli/mft_cli/airavata_mft_cli/bootstrap.py | 18 ++
1 file changed, 10 insertions(+), 8 deletions(-)
diff --git a/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
b/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
index 5102ba4..d130711 100644
--- a/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
+++ b/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
@@ -106,10 +106,10 @@ def validate_java_availability(required_version):
print("Java is either not installed or path hasn't been set properly")
raise typer.Exit()
-def is_port_open(port):
+def is_consul_running():
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
-return s.connect_ex(('localhost', port)) == 0
-
+return s.connect_ex(('localhost', 8500)) == 0
+
def start_mft():
print("Setting up MFT Services")
@@ -164,13 +164,15 @@ def start_mft():
zip_path = os.path.join(os.path.expanduser('~'),
".mft/Standalone-Service-0.01-bin.zip")
download_and_unarchive(url, zip_path)
- while not is_port_open(8500):
-print("Waiting for Consul to start...")
+ for _ in range(20):
+if is_consul_running():
+ restart_service(path + "/bin", "standalone-service-daemon.sh")
+ print("MFT Started")
+ return
time.sleep(1)
+ print("Consul is not running. Quitting...")
+ raise typer.Exit()
- restart_service(path + "/bin", "standalone-service-daemon.sh")
-
- print("MFT Started")
def stop_mft():
(airavata-mft) 01/02: wait for consul before starting mft
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
commit 419b55a761484549a2501c02da6291599b50f98a
Author: Shivang Mishra
AuthorDate: Mon Apr 1 11:23:38 2024 +0530
wait for consul before starting mft
---
python-cli/mft_cli/airavata_mft_cli/bootstrap.py | 8
1 file changed, 8 insertions(+)
diff --git a/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
b/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
index a2d9085..5102ba4 100644
--- a/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
+++ b/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
@@ -27,6 +27,7 @@ from subprocess import Popen
from pathlib import Path
from sys import platform
import shutil
+import socket
import time
def download_and_unarchive(url, download_path, extract_dir =
os.path.join(os.path.expanduser('~'), ".mft/")):
@@ -105,6 +106,9 @@ def validate_java_availability(required_version):
print("Java is either not installed or path hasn't been set properly")
raise typer.Exit()
+def is_port_open(port):
+ with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
+return s.connect_ex(('localhost', port)) == 0
def start_mft():
print("Setting up MFT Services")
@@ -160,6 +164,10 @@ def start_mft():
zip_path = os.path.join(os.path.expanduser('~'),
".mft/Standalone-Service-0.01-bin.zip")
download_and_unarchive(url, zip_path)
+ while not is_port_open(8500):
+print("Waiting for Consul to start...")
+time.sleep(1)
+
restart_service(path + "/bin", "standalone-service-daemon.sh")
print("MFT Started")
(airavata-mft) branch master updated (517b496 -> 3ccbfa2)
This is an automated email from the ASF dual-hosted git repository. dimuthuupe pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airavata-mft.git from 517b496 workflow on publishing a release new 419b55a wait for consul before starting mft new 3ccbfa2 feat(bootstrap): when start mft, quit if consul does not start after 20 second wait The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: python-cli/mft_cli/airavata_mft_cli/bootstrap.py | 16 +--- 1 file changed, 13 insertions(+), 3 deletions(-)
(airavata-cerebrum) branch main updated (d41d5c3 -> 0aec9bf)
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airavata-cerebrum.git
from d41d5c3 Merge pull request #11 from srirampc/main
add aa285d2 Added functionality to compute layer-specific ratios
add 8b60d02 Merge branch 'apache:main' into main
new 0aec9bf Merge pull request #12 from srirampc/main
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
atlas/abc_mouse.py | 203 +-
atlas/mouse_brain.py | 46 +
.../{merfish2model.ipynb => atlas2cellprop.ipynb} | 1982
3 files changed, 1462 insertions(+), 769 deletions(-)
create mode 100644 atlas/mouse_brain.py
copy notebooks/{merfish2model.ipynb => atlas2cellprop.ipynb} (55%)
(airavata-cerebrum) 01/01: Merge pull request #12 from srirampc/main
This is an automated email from the ASF dual-hosted git repository. dimuthuupe pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airavata-cerebrum.git commit 0aec9bf02ddc4eb73c7218898623ebd41f1b4035 Merge: d41d5c3 8b60d02 Author: Dimuthu Wannipurage AuthorDate: Tue Mar 26 12:09:57 2024 -0400 Merge pull request #12 from srirampc/main Functionality to compute layer-wise specific ratios for a region atlas/abc_mouse.py | 203 - atlas/mouse_brain.py | 46 + notebooks/atlas2cellprop.ipynb | 1975 3 files changed, 2204 insertions(+), 20 deletions(-)
(airavata-cerebrum) 01/01: Merge pull request #11 from srirampc/main
This is an automated email from the ASF dual-hosted git repository. dimuthuupe pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/airavata-cerebrum.git commit d41d5c3f61eadb377c7009b875b750d60ee5113c Merge: 302dafb 26ed807 Author: Dimuthu Wannipurage AuthorDate: Sat Mar 16 21:44:16 2024 -0400 Merge pull request #11 from srirampc/main Allen Brain Cell Atlas (ABC) Mouse MERFISH Data Analysis atlas/__init__.py |0 atlas/abc_mouse.py| 406 +++ atlas/bicn.py | 28 + notebooks/merfish2model.ipynb | 1491 + requirements.txt |7 +- 5 files changed, 1931 insertions(+), 1 deletion(-)
(airavata-cerebrum) branch main updated (302dafb -> d41d5c3)
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/airavata-cerebrum.git
from 302dafb Create .asf.yaml
add b46d894 Added some utility function to download data from cellxgene
add 43195b7 added method to retreive the dataset data frame
add a9bfb64 Merge branch 'apache:main' into main
add b86ee8e rename to cellxgene
add 9608202 rename folder
add 4cade83 added abc mouse and bicn interfaces
add 62836cd remove download notebook and add it to model
add 9354f5f Added documentation for the python functions
add 0f10cca added download part in the notebook
add 8f9dab6 move jupyter notebooks to the notebooks folder
add d13751d added init py
add a8b3ff5 removed cellxgene routines
add 26ed807 update requirements with scanpy, anndata etc.
new d41d5c3 Merge pull request #11 from srirampc/main
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
{cybershuttle => atlas}/__init__.py |0
atlas/abc_mouse.py | 406 ++
atlas/bicn.py | 28 +
notebooks/merfish2model.ipynb | 1491 +++
requirements.txt|7 +-
5 files changed, 1931 insertions(+), 1 deletion(-)
copy {cybershuttle => atlas}/__init__.py (100%)
create mode 100644 atlas/abc_mouse.py
create mode 100644 atlas/bicn.py
create mode 100644 notebooks/merfish2model.ipynb
(airavata-django-portal) branch master updated: Update externals_login_form.html
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-django-portal.git
The following commit(s) were added to refs/heads/master by this push:
new afd1fb90 Update externals_login_form.html
afd1fb90 is described below
commit afd1fb90ef064c70dcfc91e34e678ff6d5af66f5
Author: Johnsonjesuraj
AuthorDate: Thu Feb 29 12:29:53 2024 +0530
Update externals_login_form.html
Added Alt Tag information to resolve the Image missing a text alternative -
Siteimprove
---
.../templates/django_airavata_auth/partials/externals_login_form.html | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/django_airavata/apps/auth/templates/django_airavata_auth/partials/externals_login_form.html
b/django_airavata/apps/auth/templates/django_airavata_auth/partials/externals_login_form.html
index 617aeec4..e1948609 100644
---
a/django_airavata/apps/auth/templates/django_airavata_auth/partials/externals_login_form.html
+++
b/django_airavata/apps/auth/templates/django_airavata_auth/partials/externals_login_form.html
@@ -16,7 +16,7 @@
{% endif %}
class="btn btn-primary btn-block mt-3">
{% if external.logo %}
-
+
{% endif %}
Sign in with {{ external.name }}
(airavata-mft) tag v0.0.3 created (now 517b496)
This is an automated email from the ASF dual-hosted git repository. dimuthuupe pushed a change to tag v0.0.3 in repository https://gitbox.apache.org/repos/asf/airavata-mft.git at 517b496 (commit) No new revisions were added by this update.
(airavata-mft) branch master updated: workflow on publishing a release
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
The following commit(s) were added to refs/heads/master by this push:
new 517b496 workflow on publishing a release
517b496 is described below
commit 517b49666414e04c3d84280361f5e4f54bcbfd8a
Author: nandarshan
AuthorDate: Fri Feb 16 14:48:04 2024 -0500
workflow on publishing a release
---
.github/workflows/release_on_tag_push.yml | 40 ++-
1 file changed, 13 insertions(+), 27 deletions(-)
diff --git a/.github/workflows/release_on_tag_push.yml
b/.github/workflows/release_on_tag_push.yml
index 3227938..cd984b5 100644
--- a/.github/workflows/release_on_tag_push.yml
+++ b/.github/workflows/release_on_tag_push.yml
@@ -1,9 +1,8 @@
name: GitHub Actions Demo
run-name: ${{ github.actor }} is testing out GitHub Actions
on:
- push:
-tags:
- - '*'
+ release:
+types: [published]
jobs:
Explore-GitHub-Actions:
@@ -13,6 +12,10 @@ jobs:
uses: actions/checkout@v4
- run: echo "The ${{ github.repository }} repository has been cloned to
the runner."
+ - name: Print the tag name
+run: |
+ echo ${{ github.ref }}
+
- name: List files in the repository
run: |
ls ${{ github.workspace }}
@@ -24,31 +27,14 @@ jobs:
run: |
ls ${{ github.workspace }}
- - name: Create Release to get an id
-id: create_release_id
-uses: actions/create-release@v1
-env:
- GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-with:
- tag_name: ${{ github.ref }}
- release_name: ${{ github.ref }}
-
- - name: Upload Zip File 1
-uses: actions/upload-release-asset@v1
+ - name: Upload file 1
+run: |
+ gh release upload ${{github.event.release.tag_name}}
agent/service/target/MFT-Agent-0.01-bin.zip
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-with:
- upload_url: ${{ steps.create_release_id.outputs.upload_url }}
- asset_path: agent/service/target/MFT-Agent-0.01-bin.zip
- asset_name: MFT-Agent-0.01-bin.zip
- asset_content_type: application/zip
- - name: Upload Zip File 2
-uses: actions/upload-release-asset@v1
+ - name: Upload file 2
+run: |
+ gh release upload ${{github.event.release.tag_name}}
standalone-service/target/Standalone-Service-0.01-bin.zip
env:
- GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-with:
- upload_url: ${{ steps.create_release_id.outputs.upload_url }}
- asset_path: standalone-service/target/Standalone-Service-0.01-bin.zip
- asset_name: Standalone-Service-0.01-bin.zip
- asset_content_type: application/zip
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
\ No newline at end of file
(airavata-mft) branch master updated: grpcio issue
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
The following commit(s) were added to refs/heads/master by this push:
new 5434571 grpcio issue
5434571 is described below
commit 543457177756aa57163617201d538aed5861f62d
Author: nandarshan
AuthorDate: Wed Feb 21 09:33:03 2024 -0500
grpcio issue
---
python-cli/mft_cli/pyproject.toml | 6 --
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/python-cli/mft_cli/pyproject.toml
b/python-cli/mft_cli/pyproject.toml
index 0f43495..e3d68f1 100644
--- a/python-cli/mft_cli/pyproject.toml
+++ b/python-cli/mft_cli/pyproject.toml
@@ -39,12 +39,14 @@ python = "^3.10"
typer = {extras = ["all"], version = "^0.7.0"}
pick = {version= "2.2.0"}
grpcio= [
-{version="1.46.3", markers = "platform_machine != 'arm64'"},
+{version="1.46.3", markers = "platform_machine != 'arm64'", python =
"<3.11"},
+{version="1.60.0", markers = "platform_machine != 'arm64'", python =
">=3.11"},
{version="1.60.0", markers = "platform_machine == 'arm64'", python =
">=3.11"},
{version="1.47.0rc1", markers = "platform_machine == 'arm64'", python =
"<3.11"},
]
grpcio-tools = [
-{version="1.46.3", markers = "platform_machine != 'arm64'"},
+{version="1.46.3", markers = "platform_machine != 'arm64'", python =
"<3.11"},
+{version="1.60.0", markers = "platform_machine != 'arm64'", python =
">=3.11"},
{version="1.60.0", markers = "platform_machine == 'arm64'", python =
">=3.11"},
{version="1.47.0rc1", markers = "platform_machine == 'arm64'", python =
"<3.11"}
]
(airavata-mft) branch master updated: Supporting python version specific grpc io installation in M2 macs
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
The following commit(s) were added to refs/heads/master by this push:
new 72ecfd9 Supporting python version specific grpc io installation in M2
macs
72ecfd9 is described below
commit 72ecfd9cc0ea32a27561b54c2863c074f89339e7
Author: Dimuthu Wannipurage
AuthorDate: Wed Feb 21 08:31:00 2024 -0500
Supporting python version specific grpc io installation in M2 macs
---
python-cli/README.md | 2 +-
python-cli/mft_cli/pyproject.toml | 14 +++---
2 files changed, 12 insertions(+), 4 deletions(-)
diff --git a/python-cli/README.md b/python-cli/README.md
index 539309d..43ece75 100644
--- a/python-cli/README.md
+++ b/python-cli/README.md
@@ -39,7 +39,7 @@ poetry install
mft --help
```
-To publish the ditribution to pypi
+To publish the distribution to pypi
```
poetry publish --build
```
\ No newline at end of file
diff --git a/python-cli/mft_cli/pyproject.toml
b/python-cli/mft_cli/pyproject.toml
index 924f904..0f43495 100644
--- a/python-cli/mft_cli/pyproject.toml
+++ b/python-cli/mft_cli/pyproject.toml
@@ -18,7 +18,7 @@
[tool.poetry]
name = "airavata-mft-cli"
-version = "0.1.18"
+version = "0.1.22"
description = "Command Line Client for Apache Airavata MFT data transfer
software"
authors = [
"Dimuthu Wannipurage ",
@@ -38,8 +38,16 @@ mft = "airavata_mft_cli.main:app"
python = "^3.10"
typer = {extras = ["all"], version = "^0.7.0"}
pick = {version= "2.2.0"}
-grpcio= [{version="1.46.3", markers = "platform_machine !=
'arm64'"},{version="1.47.0rc1", markers = "platform_machine == 'arm64'"}]
-grpcio-tools = [{version="1.46.3", markers = "platform_machine !=
'arm64'"},{version="1.47.0rc1", markers = "platform_machine == 'arm64'"}]
+grpcio= [
+{version="1.46.3", markers = "platform_machine != 'arm64'"},
+{version="1.60.0", markers = "platform_machine == 'arm64'", python =
">=3.11"},
+{version="1.47.0rc1", markers = "platform_machine == 'arm64'", python =
"<3.11"},
+]
+grpcio-tools = [
+{version="1.46.3", markers = "platform_machine != 'arm64'"},
+{version="1.60.0", markers = "platform_machine == 'arm64'", python =
">=3.11"},
+{version="1.47.0rc1", markers = "platform_machine == 'arm64'", python =
"<3.11"}
+]
airavata-mft-sdk = "0.0.1a34"
pandas = "^2.0.3"
(airavata-mft) tag v0.0.2 created (now 33acf8c)
This is an automated email from the ASF dual-hosted git repository. dimuthuupe pushed a change to tag v0.0.2 in repository https://gitbox.apache.org/repos/asf/airavata-mft.git at 33acf8c (commit) No new revisions were added by this update.
(airavata-mft) tag v0.0.2 deleted (was 484f599)
This is an automated email from the ASF dual-hosted git repository. dimuthuupe pushed a change to tag v0.0.2 in repository https://gitbox.apache.org/repos/asf/airavata-mft.git *** WARNING: tag v0.0.2 was deleted! *** was 484f599 on tag create yaml The revisions that were on this tag are still contained in other references; therefore, this change does not discard any commits from the repository.
(airavata-mft) branch master updated: on tag create workflow for new pr
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
The following commit(s) were added to refs/heads/master by this push:
new 33acf8c on tag create workflow for new pr
33acf8c is described below
commit 33acf8c606363b1672732374343a9bbddd5b40d8
Author: nandarshan
AuthorDate: Fri Feb 2 11:03:10 2024 -0500
on tag create workflow for new pr
---
.github/workflows/release_on_tag_push.yml | 43 ++-
1 file changed, 25 insertions(+), 18 deletions(-)
diff --git a/.github/workflows/release_on_tag_push.yml
b/.github/workflows/release_on_tag_push.yml
index 24394df..3227938 100644
--- a/.github/workflows/release_on_tag_push.yml
+++ b/.github/workflows/release_on_tag_push.yml
@@ -24,24 +24,31 @@ jobs:
run: |
ls ${{ github.workspace }}
- - name: Upload more zips
-uses: svenstaro/upload-release-action@v2
+ - name: Create Release to get an id
+id: create_release_id
+uses: actions/create-release@v1
+env:
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
- repo_token: ${{ secrets.GITHUB_TOKEN }}
- file: agent/service/target/MFT-Agent-0.01-bin.zip
- asset-name: MFT-Agent-0.01-bin.zip
- tag: ${{ github.ref }}
- overwrite: true
- body: "Uploading more zips"
+ tag_name: ${{ github.ref }}
+ release_name: ${{ github.ref }}
- - name: Upload more zips
-uses: svenstaro/upload-release-action@v2
+ - name: Upload Zip File 1
+uses: actions/upload-release-asset@v1
+env:
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
- repo_token: ${{ secrets.GITHUB_TOKEN }}
- file: standalone-service/target/Standalone-Service-0.01-bin.zip
- asset-name: Standalone-Service-0.01-bin.zip
- tag: ${{ github.ref }}
- overwrite: true
- body: "Uploading more zips"
-
-
+ upload_url: ${{ steps.create_release_id.outputs.upload_url }}
+ asset_path: agent/service/target/MFT-Agent-0.01-bin.zip
+ asset_name: MFT-Agent-0.01-bin.zip
+ asset_content_type: application/zip
+
+ - name: Upload Zip File 2
+uses: actions/upload-release-asset@v1
+env:
+ GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+with:
+ upload_url: ${{ steps.create_release_id.outputs.upload_url }}
+ asset_path: standalone-service/target/Standalone-Service-0.01-bin.zip
+ asset_name: Standalone-Service-0.01-bin.zip
+ asset_content_type: application/zip
(airavata-mft) tag v0.0.2 created (now 484f599)
This is an automated email from the ASF dual-hosted git repository. dimuthuupe pushed a change to tag v0.0.2 in repository https://gitbox.apache.org/repos/asf/airavata-mft.git at 484f599 (commit) No new revisions were added by this update.
(airavata-mft) branch master updated: on tag create yaml
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
The following commit(s) were added to refs/heads/master by this push:
new 484f599 on tag create yaml
484f599 is described below
commit 484f5997c9478ce09df1da929ad3daca659ad318
Author: nandarshan
AuthorDate: Wed Jan 31 14:59:57 2024 -0500
on tag create yaml
---
.github/workflows/release_on_tag_push.yml | 47 +++
1 file changed, 47 insertions(+)
diff --git a/.github/workflows/release_on_tag_push.yml
b/.github/workflows/release_on_tag_push.yml
new file mode 100644
index 000..24394df
--- /dev/null
+++ b/.github/workflows/release_on_tag_push.yml
@@ -0,0 +1,47 @@
+name: GitHub Actions Demo
+run-name: ${{ github.actor }} is testing out GitHub Actions
+on:
+ push:
+tags:
+ - '*'
+
+jobs:
+ Explore-GitHub-Actions:
+runs-on: ubuntu-latest
+steps:
+ - name: Check out repository code
+uses: actions/checkout@v4
+ - run: echo "The ${{ github.repository }} repository has been cloned to
the runner."
+
+ - name: List files in the repository
+run: |
+ ls ${{ github.workspace }}
+
+ - run: echo "This job's status is ${{ job.status }}."
+ - run: mvn clean install
+
+ - name: List files in the repository
+run: |
+ ls ${{ github.workspace }}
+
+ - name: Upload more zips
+uses: svenstaro/upload-release-action@v2
+with:
+ repo_token: ${{ secrets.GITHUB_TOKEN }}
+ file: agent/service/target/MFT-Agent-0.01-bin.zip
+ asset-name: MFT-Agent-0.01-bin.zip
+ tag: ${{ github.ref }}
+ overwrite: true
+ body: "Uploading more zips"
+
+ - name: Upload more zips
+uses: svenstaro/upload-release-action@v2
+with:
+ repo_token: ${{ secrets.GITHUB_TOKEN }}
+ file: standalone-service/target/Standalone-Service-0.01-bin.zip
+ asset-name: Standalone-Service-0.01-bin.zip
+ tag: ${{ github.ref }}
+ overwrite: true
+ body: "Uploading more zips"
+
+
(airavata-custos) branch master updated: changed the docker build library to io.fabric8:docker-maven-plugin
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-custos.git
The following commit(s) were added to refs/heads/master by this push:
new ecc57c310 changed the docker build library to
io.fabric8:docker-maven-plugin
ecc57c310 is described below
commit ecc57c3108e1f86cc27cb920a1dd777103eaccea
Author: lahiruj
AuthorDate: Tue Jan 16 19:24:19 2024 -0500
changed the docker build library to io.fabric8:docker-maven-plugin
---
custos-rest-proxy/pom.xml | 73 --
.../custos-core-services-server/pom.xml| 27
.../agent-profile-core-service/pom.xml | 4 +-
.../cluster-management-core-service/pom.xml| 4 +-
.../credential-store-core-service/pom.xml | 4 +-
.../custos-core-services/custos-logging/pom.xml| 4 +-
.../custos-messaging-core-service/pom.xml | 4 +-
.../federated-authentication-core-service/pom.xml | 4 +-
.../iam-admin-core-service/pom.xml | 4 +-
.../identity-core-service/pom.xml | 4 +-
.../resource-secret-core-service/pom.xml | 4 +-
.../sharing-core-service/pom.xml | 4 +-
.../tenant-profile-core-service/pom.xml| 4 +-
.../user-profile-core-service/pom.xml | 4 +-
.../custos-grpc-web-proxy/pom.xml | 4 +-
.../custos-keycloak/pom.xml| 4 +-
.../custos-integration-services-server/pom.xml | 27
.../agent-management-service/pom.xml | 4 +-
.../custos-integration-services-swagger/pom.xml| 4 +-
.../group-management-service/pom.xml | 4 +-
.../identity-management-service/pom.xml| 4 +-
.../log-management-service/pom.xml | 4 +-
.../resource-secret-management-service/pom.xml | 4 +-
.../scim-service/pom.xml | 4 +-
.../sharing-management-service/pom.xml | 4 +-
.../tenant-management-service/pom.xml | 4 +-
.../user-management-service/pom.xml| 4 +-
custos-utilities/ide-integration/pom.xml | 27
pom.xml| 11 +++-
29 files changed, 145 insertions(+), 116 deletions(-)
diff --git a/custos-rest-proxy/pom.xml b/custos-rest-proxy/pom.xml
index 4af91e582..2db1982ec 100644
--- a/custos-rest-proxy/pom.xml
+++ b/custos-rest-proxy/pom.xml
@@ -37,42 +37,47 @@
-
-org.springframework.boot
-spring-boot-maven-plugin
-
-
-
-
org.apache.custos.ide.integration.CustosServer
-
-
-
-
-
-com.spotify
-dockerfile-maven-plugin
-${docker.plugin.version}
-
-
-container
-
-build
-push
-
-
-
+
+org.springframework.boot
+spring-boot-maven-plugin
+
+
-true
-docker.io
-
${docker.image.prefix}/custos-rest-proxy
-latest
-
true
-
-
target/${project.build.finalName}.jar
-
-false
+
org.apache.custos.ide.integration.CustosServer
-
+
+
+
+
+io.fabric8
+docker-maven-plugin
+
+
+container
+install
+
+build
+push
+
+
+
+
+
+
+
${docker.image.prefix}/custos-rest-proxy:latest
+
+
${project.basedir}
+true
+
+
target/${project.build.finalName}.jar
[airavata-mft] 01/03: Enabling path style Access to custom S3 endpoint
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
commit 0d6c3b5ccaada5ad58ab3ae73f2e2e8aeafd721b
Author: DImuthuUpe
AuthorDate: Sat Sep 2 07:55:51 2023 +0530
Enabling path style Access to custom S3 endpoint
---
python-cli/mft_cli/airavata_mft_cli/storage/s3.py | 5 ++-
python-cli/mft_cli/pyproject.toml | 4 +--
python-sdk/setup.cfg | 2 +-
.../src/airavata_mft_sdk/s3/S3Storage_pb2.py | 38 +++---
.../server/backend/sql/entity/S3StorageEntity.java | 11 +++
.../stub/src/main/proto/s3/S3Storage.proto | 3 ++
.../apache/airavata/mft/transport/s3/S3Util.java | 12 +--
7 files changed, 49 insertions(+), 26 deletions(-)
diff --git a/python-cli/mft_cli/airavata_mft_cli/storage/s3.py
b/python-cli/mft_cli/airavata_mft_cli/storage/s3.py
index 770d2fd..c6651b8 100644
--- a/python-cli/mft_cli/airavata_mft_cli/storage/s3.py
+++ b/python-cli/mft_cli/airavata_mft_cli/storage/s3.py
@@ -43,6 +43,8 @@ def handle_add_storage():
options = ["Through AWS Cli config file", "Enter manually" ]
option, index = pick(options, "How do you want to load credentials",
indicator="=>")
+enable_path_style_access = False
+
if index == 1: # Manual configuration
client_id = typer.prompt("Access Key ID")
client_secret = typer.prompt("Secret Access Key")
@@ -59,6 +61,7 @@ def handle_add_storage():
else: # If endpoint is a S3 compatible endpoint
endpoint = typer.prompt("What is the S3 endpoint URL?")
region = typer.prompt("What is the region of the bucket?")
+enable_path_style_access = typer.confirm("Enable Path Style
Access?", False)
else: # Loading credentials from the aws cli config file
config = configparser.RawConfigParser()
@@ -93,7 +96,7 @@ def handle_add_storage():
s3_secret = S3Credential_pb2.S3Secret(accessKey=client_id,
secretKey=client_secret, sessionToken = session_token)
secret_wrapper = MFTAgentStubs_pb2.SecretWrapper(s3=s3_secret)
-s3_storage = S3Storage_pb2.S3Storage(endpoint=endpoint, region=region)
+s3_storage = S3Storage_pb2.S3Storage(endpoint=endpoint, region=region,
enablePathStyleAccess=enable_path_style_access)
storage_wrapper = MFTAgentStubs_pb2.StorageWrapper(s3=s3_storage)
direct_req = MFTAgentStubs_pb2.GetResourceMetadataRequest(resourcePath="",
secret=secret_wrapper, storage=storage_wrapper)
diff --git a/python-cli/mft_cli/pyproject.toml
b/python-cli/mft_cli/pyproject.toml
index 9ec3419..924f904 100644
--- a/python-cli/mft_cli/pyproject.toml
+++ b/python-cli/mft_cli/pyproject.toml
@@ -18,7 +18,7 @@
[tool.poetry]
name = "airavata-mft-cli"
-version = "0.1.17"
+version = "0.1.18"
description = "Command Line Client for Apache Airavata MFT data transfer
software"
authors = [
"Dimuthu Wannipurage ",
@@ -40,7 +40,7 @@ typer = {extras = ["all"], version = "^0.7.0"}
pick = {version= "2.2.0"}
grpcio= [{version="1.46.3", markers = "platform_machine !=
'arm64'"},{version="1.47.0rc1", markers = "platform_machine == 'arm64'"}]
grpcio-tools = [{version="1.46.3", markers = "platform_machine !=
'arm64'"},{version="1.47.0rc1", markers = "platform_machine == 'arm64'"}]
-airavata-mft-sdk = "0.0.1a33"
+airavata-mft-sdk = "0.0.1a34"
pandas = "^2.0.3"
[build-system]
diff --git a/python-sdk/setup.cfg b/python-sdk/setup.cfg
index 9d72928..2f1caa5 100644
--- a/python-sdk/setup.cfg
+++ b/python-sdk/setup.cfg
@@ -16,7 +16,7 @@
# under the License.
[metadata]
name = airavata_mft_sdk
-version = 0.0.1-alpha33
+version = 0.0.1-alpha34
author = Airavata MFT Developers
author_email = d...@airavata.apache.org
description = Python SDK for Apache Airavata Managed File Transfers (MFT)
diff --git a/python-sdk/src/airavata_mft_sdk/s3/S3Storage_pb2.py
b/python-sdk/src/airavata_mft_sdk/s3/S3Storage_pb2.py
index f510529..3d3fcef 100644
--- a/python-sdk/src/airavata_mft_sdk/s3/S3Storage_pb2.py
+++ b/python-sdk/src/airavata_mft_sdk/s3/S3Storage_pb2.py
@@ -14,7 +14,7 @@ _sym_db = _symbol_database.Default()
-DESCRIPTOR =
_descriptor_pool.Default().AddSerializedFile(b'\n\x12s3/S3Storage.proto\x12\x31org.apache.airavata.mft.resource.stubs.s3.storage\"r\n\tS3Storage\x12\x11\n\tstorageId\x18\x01
\x01(\t\x12\x12\n\nbucketName\x18\x02 \x01(\t\x12\x0e\n\x06region\x18\x03
\x01(\t\x12\x10\n\x08\x65ndpoint\x18\x04 \x01(\t\x12\x0e\n\x06useTLS\x18\x05
\x01(\x08\x12\x0c\n\x04name\x18\x06
\x01(\t\"5\n\x14S3StorageListRequest\x12\x0e\n\x06offset\x18\x01
\x01(\x05\x12\r\n\x
[airavata-mft] 02/03: Support to describe storage from cli
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
commit c3784404192c9ac01bcdb83de82d2e411f93a6ab
Author: DImuthuUpe
AuthorDate: Thu Oct 12 01:42:35 2023 -0400
Support to describe storage from cli
---
python-cli/mft_cli/airavata_mft_cli/base.py | 6 ++
python-cli/mft_cli/airavata_mft_cli/operations.py | 5 +
2 files changed, 11 insertions(+)
diff --git a/python-cli/mft_cli/airavata_mft_cli/base.py
b/python-cli/mft_cli/airavata_mft_cli/base.py
index 553747f..924cca5 100644
--- a/python-cli/mft_cli/airavata_mft_cli/base.py
+++ b/python-cli/mft_cli/airavata_mft_cli/base.py
@@ -44,6 +44,12 @@ def copy_list(source_storage_id, dest_storage_id, list_file):
except Exception as e:
exception_handler(e)
+@app.command("describe")
+def describe(storage_name):
+ try:
+operations.describe_storage(storage_name)
+ except Exception as e:
+exception_handler(e)
@app.command("init")
def init_mft():
bootstrap.start_mft()
diff --git a/python-cli/mft_cli/airavata_mft_cli/operations.py
b/python-cli/mft_cli/airavata_mft_cli/operations.py
index 8ed867d..50276ba 100644
--- a/python-cli/mft_cli/airavata_mft_cli/operations.py
+++ b/python-cli/mft_cli/airavata_mft_cli/operations.py
@@ -61,6 +61,11 @@ def fetch_storage_and_secret_ids(storage_name):
return sec_resp.storageId, sec_resp.secretId
+def describe_storage(storage_name):
+storage_id, secret_id = fetch_storage_and_secret_ids(storage_name)
+print("Storage id: " + storage_id)
+print("Secret id: " + secret_id)
+
def get_resource_metadata(storage_path, recursive_search = False):
storage_name = storage_path.split("/")[0]
resource_path = storage_path[len(storage_name) +1 :]
[airavata-mft] 03/03: Handling S3 bucket listing errors on cli
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
commit c7c17da816a1499ea270f6ede9dc80db1cb92282
Author: DImuthuUpe
AuthorDate: Thu Oct 12 01:43:05 2023 -0400
Handling S3 bucket listing errors on cli
---
python-cli/mft_cli/airavata_mft_cli/storage/s3.py | 14 --
1 file changed, 8 insertions(+), 6 deletions(-)
diff --git a/python-cli/mft_cli/airavata_mft_cli/storage/s3.py
b/python-cli/mft_cli/airavata_mft_cli/storage/s3.py
index c6651b8..31a2a97 100644
--- a/python-cli/mft_cli/airavata_mft_cli/storage/s3.py
+++ b/python-cli/mft_cli/airavata_mft_cli/storage/s3.py
@@ -101,14 +101,16 @@ def handle_add_storage():
direct_req = MFTAgentStubs_pb2.GetResourceMetadataRequest(resourcePath="",
secret=secret_wrapper, storage=storage_wrapper)
resource_medata_req =
MFTTransferApi_pb2.FetchResourceMetadataRequest(directRequest = direct_req)
-metadata_resp = client.transfer_api.resourceMetadata(resource_medata_req)
-
bucket_options = ["Manually Enter"]
-bucket_list = metadata_resp.directory.directories
-if len(bucket_list) > 0:
-for b in bucket_list:
-bucket_options.append(b.friendlyName)
+try:
+metadata_resp =
client.transfer_api.resourceMetadata(resource_medata_req)
+bucket_list = metadata_resp.directory.directories
+if len(bucket_list) > 0:
+for b in bucket_list:
+bucket_options.append(b.friendlyName)
+except:
+print("Failed to fetch bucket list. Add the bucket name manually")
title = "Select the Bucket: "
selected_bucket, index = pick(bucket_options, title, indicator="=>")
[airavata-mft] branch master updated (d8cf348 -> c7c17da)
This is an automated email from the ASF dual-hosted git repository. dimuthuupe pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airavata-mft.git from d8cf348 Minor optimizations new 0d6c3b5 Enabling path style Access to custom S3 endpoint new c378440 Support to describe storage from cli new c7c17da Handling S3 bucket listing errors on cli The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: python-cli/mft_cli/airavata_mft_cli/base.py| 6 python-cli/mft_cli/airavata_mft_cli/operations.py | 5 +++ python-cli/mft_cli/airavata_mft_cli/storage/s3.py | 19 +++ python-cli/mft_cli/pyproject.toml | 4 +-- python-sdk/setup.cfg | 2 +- .../src/airavata_mft_sdk/s3/S3Storage_pb2.py | 38 +++--- .../server/backend/sql/entity/S3StorageEntity.java | 11 +++ .../stub/src/main/proto/s3/S3Storage.proto | 3 ++ .../apache/airavata/mft/transport/s3/S3Util.java | 12 +-- 9 files changed, 68 insertions(+), 32 deletions(-)
[airavata-mft] branch ISSUE-108-Path-style updated (23ecf9d -> 252f355)
This is an automated email from the ASF dual-hosted git repository. dimuthuupe pushed a change to branch ISSUE-108-Path-style in repository https://gitbox.apache.org/repos/asf/airavata-mft.git from 23ecf9d Enabling path style Access to custom S3 endpoint add ad83517 Support to describe storage from cli add 252f355 Handling S3 bucket listing errors on cli No new revisions were added by this update. Summary of changes: python-cli/mft_cli/airavata_mft_cli/base.py | 6 ++ python-cli/mft_cli/airavata_mft_cli/operations.py | 5 + python-cli/mft_cli/airavata_mft_cli/storage/s3.py | 14 -- 3 files changed, 19 insertions(+), 6 deletions(-)
[airavata-mft] 01/01: Enabling path style Access to custom S3 endpoint
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch ISSUE-108-Path-style
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
commit 23ecf9dd2cc4e15f25df45c90285014bedbf07af
Author: DImuthuUpe
AuthorDate: Sat Sep 2 07:55:51 2023 +0530
Enabling path style Access to custom S3 endpoint
---
python-cli/mft_cli/airavata_mft_cli/storage/s3.py | 5 ++-
python-cli/mft_cli/pyproject.toml | 4 +--
python-sdk/setup.cfg | 2 +-
.../src/airavata_mft_sdk/s3/S3Storage_pb2.py | 38 +++---
.../server/backend/sql/entity/S3StorageEntity.java | 11 +++
.../stub/src/main/proto/s3/S3Storage.proto | 3 ++
.../apache/airavata/mft/transport/s3/S3Util.java | 12 +--
7 files changed, 49 insertions(+), 26 deletions(-)
diff --git a/python-cli/mft_cli/airavata_mft_cli/storage/s3.py
b/python-cli/mft_cli/airavata_mft_cli/storage/s3.py
index 770d2fd..c6651b8 100644
--- a/python-cli/mft_cli/airavata_mft_cli/storage/s3.py
+++ b/python-cli/mft_cli/airavata_mft_cli/storage/s3.py
@@ -43,6 +43,8 @@ def handle_add_storage():
options = ["Through AWS Cli config file", "Enter manually" ]
option, index = pick(options, "How do you want to load credentials",
indicator="=>")
+enable_path_style_access = False
+
if index == 1: # Manual configuration
client_id = typer.prompt("Access Key ID")
client_secret = typer.prompt("Secret Access Key")
@@ -59,6 +61,7 @@ def handle_add_storage():
else: # If endpoint is a S3 compatible endpoint
endpoint = typer.prompt("What is the S3 endpoint URL?")
region = typer.prompt("What is the region of the bucket?")
+enable_path_style_access = typer.confirm("Enable Path Style
Access?", False)
else: # Loading credentials from the aws cli config file
config = configparser.RawConfigParser()
@@ -93,7 +96,7 @@ def handle_add_storage():
s3_secret = S3Credential_pb2.S3Secret(accessKey=client_id,
secretKey=client_secret, sessionToken = session_token)
secret_wrapper = MFTAgentStubs_pb2.SecretWrapper(s3=s3_secret)
-s3_storage = S3Storage_pb2.S3Storage(endpoint=endpoint, region=region)
+s3_storage = S3Storage_pb2.S3Storage(endpoint=endpoint, region=region,
enablePathStyleAccess=enable_path_style_access)
storage_wrapper = MFTAgentStubs_pb2.StorageWrapper(s3=s3_storage)
direct_req = MFTAgentStubs_pb2.GetResourceMetadataRequest(resourcePath="",
secret=secret_wrapper, storage=storage_wrapper)
diff --git a/python-cli/mft_cli/pyproject.toml
b/python-cli/mft_cli/pyproject.toml
index 9ec3419..924f904 100644
--- a/python-cli/mft_cli/pyproject.toml
+++ b/python-cli/mft_cli/pyproject.toml
@@ -18,7 +18,7 @@
[tool.poetry]
name = "airavata-mft-cli"
-version = "0.1.17"
+version = "0.1.18"
description = "Command Line Client for Apache Airavata MFT data transfer
software"
authors = [
"Dimuthu Wannipurage ",
@@ -40,7 +40,7 @@ typer = {extras = ["all"], version = "^0.7.0"}
pick = {version= "2.2.0"}
grpcio= [{version="1.46.3", markers = "platform_machine !=
'arm64'"},{version="1.47.0rc1", markers = "platform_machine == 'arm64'"}]
grpcio-tools = [{version="1.46.3", markers = "platform_machine !=
'arm64'"},{version="1.47.0rc1", markers = "platform_machine == 'arm64'"}]
-airavata-mft-sdk = "0.0.1a33"
+airavata-mft-sdk = "0.0.1a34"
pandas = "^2.0.3"
[build-system]
diff --git a/python-sdk/setup.cfg b/python-sdk/setup.cfg
index 9d72928..2f1caa5 100644
--- a/python-sdk/setup.cfg
+++ b/python-sdk/setup.cfg
@@ -16,7 +16,7 @@
# under the License.
[metadata]
name = airavata_mft_sdk
-version = 0.0.1-alpha33
+version = 0.0.1-alpha34
author = Airavata MFT Developers
author_email = d...@airavata.apache.org
description = Python SDK for Apache Airavata Managed File Transfers (MFT)
diff --git a/python-sdk/src/airavata_mft_sdk/s3/S3Storage_pb2.py
b/python-sdk/src/airavata_mft_sdk/s3/S3Storage_pb2.py
index f510529..3d3fcef 100644
--- a/python-sdk/src/airavata_mft_sdk/s3/S3Storage_pb2.py
+++ b/python-sdk/src/airavata_mft_sdk/s3/S3Storage_pb2.py
@@ -14,7 +14,7 @@ _sym_db = _symbol_database.Default()
-DESCRIPTOR =
_descriptor_pool.Default().AddSerializedFile(b'\n\x12s3/S3Storage.proto\x12\x31org.apache.airavata.mft.resource.stubs.s3.storage\"r\n\tS3Storage\x12\x11\n\tstorageId\x18\x01
\x01(\t\x12\x12\n\nbucketName\x18\x02 \x01(\t\x12\x0e\n\x06region\x18\x03
\x01(\t\x12\x10\n\x08\x65ndpoint\x18\x04 \x01(\t\x12\x0e\n\x06useTLS\x18\x05
\x01(\x08\x12\x0c\n\x04name\x18\x06
\x01(\t\"5\n\x14S3StorageListRequest\x12\x0e\n\x06offset\x18\x01
\x01(\x
[airavata-mft] branch ISSUE-108-Path-style created (now 23ecf9d)
This is an automated email from the ASF dual-hosted git repository. dimuthuupe pushed a change to branch ISSUE-108-Path-style in repository https://gitbox.apache.org/repos/asf/airavata-mft.git at 23ecf9d Enabling path style Access to custom S3 endpoint This branch includes the following new commits: new 23ecf9d Enabling path style Access to custom S3 endpoint The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[airavata-mft] 02/02: Minor optimizations
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
commit d8cf348886735fe92d16f1c0b2c5d83e7741ce5a
Author: DImuthuUpe
AuthorDate: Fri Sep 1 23:40:51 2023 +0530
Minor optimizations
---
.../java/org/apache/airavata/mft/agent/TransportMediator.java| 9 ++---
.../apache/airavata/mft/transport/s3/S3OutgoingConnector.java| 8
2 files changed, 10 insertions(+), 7 deletions(-)
diff --git
a/agent/service/src/main/java/org/apache/airavata/mft/agent/TransportMediator.java
b/agent/service/src/main/java/org/apache/airavata/mft/agent/TransportMediator.java
index 42c0630..0f55b4a 100644
---
a/agent/service/src/main/java/org/apache/airavata/mft/agent/TransportMediator.java
+++
b/agent/service/src/main/java/org/apache/airavata/mft/agent/TransportMediator.java
@@ -280,9 +280,12 @@ public class TransportMediator {
uploader.uploadChunk(chunkIdx, startPos, endPos,
inputStream);
} else {
String tempFile = tempDataDir + File.separator +
UUID.randomUUID().toString() + "-" + chunkIdx;
-downloader.downloadChunk(chunkIdx, startPos, endPos,
tempFile);
-uploader.uploadChunk(chunkIdx, startPos, endPos, tempFile);
-new File(tempFile).delete();
+try {
+downloader.downloadChunk(chunkIdx, startPos, endPos,
tempFile);
+uploader.uploadChunk(chunkIdx, startPos, endPos,
tempFile);
+} finally {
+new File(tempFile).delete();
+}
}
return chunkIdx;
} catch (Exception e) {
diff --git
a/transport/s3-transport/src/main/java/org/apache/airavata/mft/transport/s3/S3OutgoingConnector.java
b/transport/s3-transport/src/main/java/org/apache/airavata/mft/transport/s3/S3OutgoingConnector.java
index 6cc17d4..cc3d6d2 100644
---
a/transport/s3-transport/src/main/java/org/apache/airavata/mft/transport/s3/S3OutgoingConnector.java
+++
b/transport/s3-transport/src/main/java/org/apache/airavata/mft/transport/s3/S3OutgoingConnector.java
@@ -92,10 +92,10 @@ public class S3OutgoingConnector implements
OutgoingChunkedConnector {
.withUploadId(initResponse.getUploadId())
.withPartNumber(chunkId + 1)
.withFileOffset(0)
-//.withMD5Digest(Md5Utils.md5AsBase64(new
File(uploadFile)))
-//.withFile(file)
-.withInputStream(new BufferedInputStream(new
FileInputStream(file), Math.min(16 * 1024 * 1024, (int) ( endByte -
startByte
-.withPartSize(file.length());
+.withFile(file)
+.withMD5Digest(Md5Utils.md5AsBase64(new File(uploadFile)))
+//.withInputStream(new BufferedInputStream(new
FileInputStream(file), Math.min(16 * 1024 * 1024, (int) ( endByte -
startByte
+.withPartSize(endByte - startByte);
UploadPartResult uploadResult = s3Client.uploadPart(uploadRequest);
this.partETags.add(uploadResult.getPartETag());
[airavata-mft] branch master updated (54be23e -> d8cf348)
This is an automated email from the ASF dual-hosted git repository. dimuthuupe pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airavata-mft.git from 54be23e Supporting storage removal from cli new f4acebe Avoiding writing additional byte in SwiftIncomingConnector new d8cf348 Minor optimizations The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../java/org/apache/airavata/mft/agent/TransportMediator.java| 9 ++--- .../apache/airavata/mft/transport/s3/S3OutgoingConnector.java| 8 .../airavata/mft/transport/swift/SwiftIncomingConnector.java | 4 ++-- 3 files changed, 12 insertions(+), 9 deletions(-)
[airavata-mft] 01/02: Avoiding writing additional byte in SwiftIncomingConnector
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
commit f4acebe3e6ab0b7a4987f33fe84345b0867c7aef
Author: DImuthuUpe
AuthorDate: Fri Sep 1 23:39:40 2023 +0530
Avoiding writing additional byte in SwiftIncomingConnector
---
.../apache/airavata/mft/transport/swift/SwiftIncomingConnector.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/transport/swift-transport/src/main/java/org/apache/airavata/mft/transport/swift/SwiftIncomingConnector.java
b/transport/swift-transport/src/main/java/org/apache/airavata/mft/transport/swift/SwiftIncomingConnector.java
index 4c9ab88..aa2631f 100644
---
a/transport/swift-transport/src/main/java/org/apache/airavata/mft/transport/swift/SwiftIncomingConnector.java
+++
b/transport/swift-transport/src/main/java/org/apache/airavata/mft/transport/swift/SwiftIncomingConnector.java
@@ -66,7 +66,7 @@ public class SwiftIncomingConnector implements
IncomingChunkedConnector {
public void downloadChunk(int chunkId, long startByte, long endByte,
String downloadFile) throws Exception {
SwiftObject swiftObject = objectApi.get(
resourcePath,
-GetOptions.Builder.range(startByte, endByte));
+GetOptions.Builder.range(startByte, endByte - 1));
InputStream inputStream = swiftObject.getPayload().openStream();
@@ -85,7 +85,7 @@ public class SwiftIncomingConnector implements
IncomingChunkedConnector {
SwiftObject swiftObject = objectApi.get(
resourcePath,
-GetOptions.Builder.range(startByte, endByte));
+GetOptions.Builder.range(startByte, endByte - 1));
return swiftObject.getPayload().openStream();
}
[airavata-mft] 02/02: Supporting storage removal from cli
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
commit 54be23eb6c302f3fc37d7fc3e6bee3bb590ba0cf
Author: DImuthuUpe
AuthorDate: Fri Sep 1 21:12:12 2023 +0530
Supporting storage removal from cli
---
python-cli/mft_cli/airavata_mft_cli/storage/__init__.py| 14 ++
python-cli/mft_cli/pyproject.toml | 2 +-
.../resource/server/backend/sql/SQLResourceBackend.java| 4 ++--
.../backend/sql/repository/ResolveStorageRepository.java | 2 ++
.../backend/sql/repository/StorageSecretRepository.java| 2 +-
.../server/handler/StorageCommonServiceHandler.java| 7 +--
6 files changed, 25 insertions(+), 6 deletions(-)
diff --git a/python-cli/mft_cli/airavata_mft_cli/storage/__init__.py
b/python-cli/mft_cli/airavata_mft_cli/storage/__init__.py
index b2dd0ff..ac1110d 100644
--- a/python-cli/mft_cli/airavata_mft_cli/storage/__init__.py
+++ b/python-cli/mft_cli/airavata_mft_cli/storage/__init__.py
@@ -60,6 +60,20 @@ def add_storage():
except Exception as e:
exception_handler(e)
+@app.command("remove")
+def remove_storage(storage_id):
+client = mft_client.MFTClient(transfer_api_port =
configcli.transfer_api_port,
+transfer_api_secured =
configcli.transfer_api_secured,
+resource_service_host =
configcli.resource_service_host,
+resource_service_port =
configcli.resource_service_port,
+resource_service_secured =
configcli.resource_service_secured,
+secret_service_host =
configcli.secret_service_host,
+secret_service_port =
configcli.secret_service_port)
+delete_request =
StorageCommon_pb2.SecretForStorageDeleteRequest(storageId=storage_id)
+delete_response = client.common_api.deleteSecretsForStorage(delete_request)
+console = Console()
+console.print("Storage removed: " + str(delete_response.status))
+
@app.command("list")
def list_storage():
try:
diff --git a/python-cli/mft_cli/pyproject.toml
b/python-cli/mft_cli/pyproject.toml
index 40b726f..9ec3419 100644
--- a/python-cli/mft_cli/pyproject.toml
+++ b/python-cli/mft_cli/pyproject.toml
@@ -18,7 +18,7 @@
[tool.poetry]
name = "airavata-mft-cli"
-version = "0.1.14"
+version = "0.1.17"
description = "Command Line Client for Apache Airavata MFT data transfer
software"
authors = [
"Dimuthu Wannipurage ",
diff --git
a/services/resource-service/server/src/main/java/org/apache/airavata/mft/resource/server/backend/sql/SQLResourceBackend.java
b/services/resource-service/server/src/main/java/org/apache/airavata/mft/resource/server/backend/sql/SQLResourceBackend.java
index 4751130..d46b8c6 100644
---
a/services/resource-service/server/src/main/java/org/apache/airavata/mft/resource/server/backend/sql/SQLResourceBackend.java
+++
b/services/resource-service/server/src/main/java/org/apache/airavata/mft/resource/server/backend/sql/SQLResourceBackend.java
@@ -125,8 +125,8 @@ public class SQLResourceBackend implements ResourceBackend {
@Override
public boolean deleteSecretForStorage(SecretForStorageDeleteRequest
request) throws Exception {
-resourceSecretRepository.deleteByStorageId(request.getStorageId());
-return true;
+resolveStorageRepository.deleteByStorageId(request.getStorageId());
+return
resourceSecretRepository.deleteByStorageId(request.getStorageId()) > 0;
}
@Override
diff --git
a/services/resource-service/server/src/main/java/org/apache/airavata/mft/resource/server/backend/sql/repository/ResolveStorageRepository.java
b/services/resource-service/server/src/main/java/org/apache/airavata/mft/resource/server/backend/sql/repository/ResolveStorageRepository.java
index 4f9e380..dcc1e1e 100644
---
a/services/resource-service/server/src/main/java/org/apache/airavata/mft/resource/server/backend/sql/repository/ResolveStorageRepository.java
+++
b/services/resource-service/server/src/main/java/org/apache/airavata/mft/resource/server/backend/sql/repository/ResolveStorageRepository.java
@@ -30,4 +30,6 @@ public interface ResolveStorageRepository extends
CrudRepository getByStorageId(String storageID);
List getByStorageName(String storageName);
List
getByStorageType(ResolveStorageEntity.StorageType storageType);
+
+int deleteByStorageId(String storageId);
}
diff --git
a/services/resource-service/server/src/main/java/org/apache/airavata/mft/resource/server/backend/sql/repository/StorageSecretRepository.java
b/services/resource-service/server/src/main/java/org/apache/airavata/mft/resource/server/backend/sql/repository/StorageSecretRe
[airavata-mft] 01/02: Minor refactoring
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
commit 233a59783e396752518bbfadc5cd2c13b4f08a33
Author: DImuthuUpe
AuthorDate: Fri Sep 1 21:11:29 2023 +0530
Minor refactoring
---
python-cli/mft_cli/airavata_mft_cli/storage/swift.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/python-cli/mft_cli/airavata_mft_cli/storage/swift.py
b/python-cli/mft_cli/airavata_mft_cli/storage/swift.py
index df32346..2bcd41b 100644
--- a/python-cli/mft_cli/airavata_mft_cli/storage/swift.py
+++ b/python-cli/mft_cli/airavata_mft_cli/storage/swift.py
@@ -37,7 +37,7 @@ def handle_add_storage():
user_name = typer.prompt("User Name")
password = typer.prompt("Password")
-tenant_name = typer.prompt("Tenant Name")
+tenant_name = typer.prompt("Project Name")
project_domain = typer.prompt("Project Domain Name", "Default")
user_domain = typer.prompt("User Domain Name", "Default")
v3_sec =
SwiftCredential_pb2.SwiftV3AuthSecret(userDomainName=user_domain,
userName=user_name,
[airavata-mft] branch master updated (8fd067e -> 54be23e)
This is an automated email from the ASF dual-hosted git repository. dimuthuupe pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airavata-mft.git from 8fd067e Updating cli version new 233a597 Minor refactoring new 54be23e Supporting storage removal from cli The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: python-cli/mft_cli/airavata_mft_cli/storage/__init__.py| 14 ++ python-cli/mft_cli/airavata_mft_cli/storage/swift.py | 2 +- python-cli/mft_cli/pyproject.toml | 2 +- .../resource/server/backend/sql/SQLResourceBackend.java| 4 ++-- .../backend/sql/repository/ResolveStorageRepository.java | 2 ++ .../backend/sql/repository/StorageSecretRepository.java| 2 +- .../server/handler/StorageCommonServiceHandler.java| 7 +-- 7 files changed, 26 insertions(+), 7 deletions(-)
[airavata-mft] 01/02: Minor property fix
This is an automated email from the ASF dual-hosted git repository. dimuthuupe pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/airavata-mft.git commit 497c3f858bad9f46cc7b7a8b1297500a7333c18e Author: DImuthuUpe AuthorDate: Fri Sep 1 14:26:45 2023 +0530 Minor property fix --- .../src/main/resources/distribution/conf/application.properties | 4 1 file changed, 4 insertions(+) diff --git a/standalone-service/src/main/resources/distribution/conf/application.properties b/standalone-service/src/main/resources/distribution/conf/application.properties index ffdbf58..09e5678 100644 --- a/standalone-service/src/main/resources/distribution/conf/application.properties +++ b/standalone-service/src/main/resources/distribution/conf/application.properties @@ -21,6 +21,10 @@ spring.jpa.hibernate.ddl-auto=update consul.host=localhost grpc.port=7003 +resource.service.host=localhost +resource.service.port=7003 +secret.service.host=localhost +secret.service.port=7003 consul.port=8500 agent.id=local-agent agent.concurrent.transfers=8
[airavata-mft] branch master updated (06736e8 -> 8fd067e)
This is an automated email from the ASF dual-hosted git repository. dimuthuupe pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airavata-mft.git from 06736e8 Fetching Swfit Metadata object by reference new 497c3f8 Minor property fix new 8fd067e Updating cli version The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: python-cli/mft_cli/pyproject.toml | 2 +- .../src/main/resources/distribution/conf/application.properties | 4 2 files changed, 5 insertions(+), 1 deletion(-)
[airavata-mft] 02/02: Updating cli version
This is an automated email from the ASF dual-hosted git repository. dimuthuupe pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/airavata-mft.git commit 8fd067ec8f4f2e1dcb8f0a3c314fd2b8c707e62d Author: DImuthuUpe AuthorDate: Fri Sep 1 14:27:10 2023 +0530 Updating cli version --- python-cli/mft_cli/pyproject.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/python-cli/mft_cli/pyproject.toml b/python-cli/mft_cli/pyproject.toml index 5d54cc2..40b726f 100644 --- a/python-cli/mft_cli/pyproject.toml +++ b/python-cli/mft_cli/pyproject.toml @@ -18,7 +18,7 @@ [tool.poetry] name = "airavata-mft-cli" -version = "0.1.13" +version = "0.1.14" description = "Command Line Client for Apache Airavata MFT data transfer software" authors = [ "Dimuthu Wannipurage ",
[airavata-mft] 01/02: Loading resources as stream in TransportClassLoader
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
commit a764ea4897147858c57868f8e37bffa751833df5
Author: DImuthuUpe
AuthorDate: Fri Sep 1 14:13:16 2023 +0530
Loading resources as stream in TransportClassLoader
---
.../mft/agent/transport/TransportClassLoader.java | 18 +++---
1 file changed, 15 insertions(+), 3 deletions(-)
diff --git
a/agent/service/src/main/java/org/apache/airavata/mft/agent/transport/TransportClassLoader.java
b/agent/service/src/main/java/org/apache/airavata/mft/agent/transport/TransportClassLoader.java
index bcc2e94..2f3613f 100644
---
a/agent/service/src/main/java/org/apache/airavata/mft/agent/transport/TransportClassLoader.java
+++
b/agent/service/src/main/java/org/apache/airavata/mft/agent/transport/TransportClassLoader.java
@@ -23,6 +23,7 @@ import org.slf4j.LoggerFactory;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
+import java.io.InputStream;
import java.net.URL;
import java.net.URLClassLoader;
import java.nio.file.Files;
@@ -38,7 +39,7 @@ public class TransportClassLoader extends URLClassLoader {
private final ClassLoader systemClassLoader;
-private final Map classes;
+private final Map content;
public Map loadClassesFromFile(final Path path) throws
IOException {
try {
@@ -65,6 +66,8 @@ public class TransportClassLoader extends URLClassLoader {
result.putAll(unzipRecursively(os));
} else if
(entry.getName().toLowerCase().endsWith(".class")) {
result.put(entry.getName().replaceAll("/",
".").substring(0, entry.getName().length() - 6), os.toByteArray());
+} else {
+result.put(entry.getName(), os.toByteArray());
}
}
}
@@ -76,12 +79,12 @@ public class TransportClassLoader extends URLClassLoader {
public TransportClassLoader(URL[] urls, ClassLoader parent, Path
connectorPath) throws IOException {
super(urls, parent);
systemClassLoader = getSystemClassLoader();
-classes = loadClassesFromFile(connectorPath);
+content = loadClassesFromFile(connectorPath);
}
@Override
public final Class findClass(String name) throws ClassNotFoundException
{
-final byte[] bytes = classes.get(name);
+final byte[] bytes = content.get(name);
if(bytes != null) {
logger.debug("Found class {}", name);
return defineClass(name, bytes, 0, bytes.length);
@@ -120,4 +123,13 @@ public class TransportClassLoader extends URLClassLoader {
}
return loadedClass;
}
+
+@Override
+public InputStream getResourceAsStream(String name) {
+if (content.containsKey(name)) {
+return new ByteArrayInputStream(content.get(name));
+} else {
+return super.getResourceAsStream(name);
+}
+}
}
[airavata-mft] branch master updated (b5de3f8 -> 06736e8)
This is an automated email from the ASF dual-hosted git repository. dimuthuupe pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airavata-mft.git from b5de3f8 Disabling small file optimization for s3 outgoing new a764ea4 Loading resources as stream in TransportClassLoader new 06736e8 Fetching Swfit Metadata object by reference The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../mft/agent/transport/TransportClassLoader.java | 18 +++--- .../apache/airavata/mft/transport/swift/SwiftUtil.java | 7 --- 2 files changed, 19 insertions(+), 6 deletions(-)
[airavata-mft] 02/02: Fetching Swfit Metadata object by reference
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
commit 06736e859af9d86093ab41bf6d70f682aa99bd5c
Author: DImuthuUpe
AuthorDate: Fri Sep 1 14:26:16 2023 +0530
Fetching Swfit Metadata object by reference
---
.../java/org/apache/airavata/mft/transport/swift/SwiftUtil.java| 7 ---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git
a/transport/swift-transport/src/main/java/org/apache/airavata/mft/transport/swift/SwiftUtil.java
b/transport/swift-transport/src/main/java/org/apache/airavata/mft/transport/swift/SwiftUtil.java
index 66c24ac..f5b22b4 100644
---
a/transport/swift-transport/src/main/java/org/apache/airavata/mft/transport/swift/SwiftUtil.java
+++
b/transport/swift-transport/src/main/java/org/apache/airavata/mft/transport/swift/SwiftUtil.java
@@ -26,6 +26,7 @@ import
org.apache.airavata.mft.credential.stubs.swift.SwiftV3AuthSecret;
import org.jclouds.ContextBuilder;
import org.jclouds.openstack.keystone.config.KeystoneProperties;
import org.jclouds.openstack.swift.v1.SwiftApi;
+import org.jclouds.openstack.swift.v1.SwiftApiMetadata;
// https://jclouds.apache.org/guides/openstack/
public class SwiftUtil {
@@ -75,13 +76,13 @@ public class SwiftUtil {
SwiftApi swiftApi;
-String provider = "openstack-swift";
+//String provider = "openstack-swift";
Properties overrides = new Properties();
switch (swiftSecret.getSecretCase()) {
case V2AUTHSECRET:
SwiftV2AuthSecret v2AuthSecret = swiftSecret.getV2AuthSecret();
overrides.put(KeystoneProperties.KEYSTONE_VERSION, "2");
- swiftApi = ContextBuilder.newBuilder(provider)
+ swiftApi = ContextBuilder.newBuilder(new SwiftApiMetadata())
.endpoint(swiftSecret.getEndpoint())
.credentials(v2AuthSecret.getTenant() + ":" +
v2AuthSecret.getUserName(),
v2AuthSecret.getPassword())
@@ -100,7 +101,7 @@ public class SwiftUtil {
overrides.put(KeystoneProperties.PROJECT_DOMAIN_NAME,
v3AuthSecret.getProjectDomainName());
}
- swiftApi = ContextBuilder.newBuilder(provider)
+ swiftApi = ContextBuilder.newBuilder(new SwiftApiMetadata())
.endpoint(swiftSecret.getEndpoint())
.credentials(v3AuthSecret.getUserDomainName() + ":" +
v3AuthSecret.getUserName(),
v3AuthSecret.getPassword())
[airavata-mft] branch master updated: Disabling small file optimization for s3 outgoing
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
The following commit(s) were added to refs/heads/master by this push:
new b5de3f8 Disabling small file optimization for s3 outgoing
b5de3f8 is described below
commit b5de3f802878c9d11b16a92b4be069565524eff5
Author: DImuthuUpe
AuthorDate: Wed Aug 30 19:12:21 2023 +0530
Disabling small file optimization for s3 outgoing
---
.../org/apache/airavata/mft/transport/s3/S3OutgoingConnector.java | 8
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git
a/transport/s3-transport/src/main/java/org/apache/airavata/mft/transport/s3/S3OutgoingConnector.java
b/transport/s3-transport/src/main/java/org/apache/airavata/mft/transport/s3/S3OutgoingConnector.java
index d9f1844..6cc17d4 100644
---
a/transport/s3-transport/src/main/java/org/apache/airavata/mft/transport/s3/S3OutgoingConnector.java
+++
b/transport/s3-transport/src/main/java/org/apache/airavata/mft/transport/s3/S3OutgoingConnector.java
@@ -70,15 +70,15 @@ public class S3OutgoingConnector implements
OutgoingChunkedConnector {
s3Client = S3Util.getInstance().leaseS3Client(s3Secret, s3Storage);
-if (cc.getChunkSize() < cc.getMetadata().getFile().getResourceSize()) {
+//if (cc.getChunkSize() <
cc.getMetadata().getFile().getResourceSize()) {
InitiateMultipartUploadRequest initRequest = new
InitiateMultipartUploadRequest(s3Storage.getBucketName(),
resourcePath);
initResponse = s3Client.initiateMultipartUpload(initRequest);
logger.info("Initialized multipart upload for file {} in bucket
{}",
resourcePath, s3Storage.getBucketName());
-} else {
-logger.info("Using non-multipart upload for file {} in bucket {}",
resourcePath, s3Storage.getBucketName());
-}
+//} else {
+//logger.info("Using non-multipart upload for file {} in bucket
{}", resourcePath, s3Storage.getBucketName());
+//}
}
@Override
[airavata-mft] branch master updated: Fixing complete logic for S3 non multipart uploads
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
The following commit(s) were added to refs/heads/master by this push:
new fa14a25 Fixing complete logic for S3 non multipart uploads
fa14a25 is described below
commit fa14a256f1987a320b589b919738c4615ab919a5
Author: DImuthuUpe
AuthorDate: Wed Aug 30 17:56:50 2023 +0530
Fixing complete logic for S3 non multipart uploads
---
.../airavata/mft/transport/s3/S3OutgoingConnector.java | 12 +++-
1 file changed, 7 insertions(+), 5 deletions(-)
diff --git
a/transport/s3-transport/src/main/java/org/apache/airavata/mft/transport/s3/S3OutgoingConnector.java
b/transport/s3-transport/src/main/java/org/apache/airavata/mft/transport/s3/S3OutgoingConnector.java
index 6dfa022..d9f1844 100644
---
a/transport/s3-transport/src/main/java/org/apache/airavata/mft/transport/s3/S3OutgoingConnector.java
+++
b/transport/s3-transport/src/main/java/org/apache/airavata/mft/transport/s3/S3OutgoingConnector.java
@@ -130,11 +130,13 @@ public class S3OutgoingConnector implements
OutgoingChunkedConnector {
@Override
public void complete() throws Exception {
-CompleteMultipartUploadRequest compRequest = new
CompleteMultipartUploadRequest(s3Storage.getBucketName(),
-resourcePath, initResponse.getUploadId(), partETags);
-s3Client.completeMultipartUpload(compRequest);
-logger.info("Completing the upload for file {} in bucket {}",
resourcePath,
-s3Storage.getBucketName());
+
+if (initResponse != null) {
+CompleteMultipartUploadRequest compRequest = new
CompleteMultipartUploadRequest(s3Storage.getBucketName(),
+resourcePath, initResponse.getUploadId(), partETags);
+s3Client.completeMultipartUpload(compRequest);
+}
+logger.info("Completed the upload for file {} in bucket {}",
resourcePath, s3Storage.getBucketName());
}
@Override
[airavata-mft] branch master updated: Fixing a minor logging issues
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
The following commit(s) were added to refs/heads/master by this push:
new c69e14b Fixing a minor logging issues
c69e14b is described below
commit c69e14b409add099d151434b9add711e41f41b20
Author: DImuthuUpe
AuthorDate: Sat Aug 19 00:44:25 2023 +0530
Fixing a minor logging issues
---
.../src/main/java/org/apache/airavata/mft/controller/MFTController.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/controller/src/main/java/org/apache/airavata/mft/controller/MFTController.java
b/controller/src/main/java/org/apache/airavata/mft/controller/MFTController.java
index 110f6d2..f945366 100644
---
a/controller/src/main/java/org/apache/airavata/mft/controller/MFTController.java
+++
b/controller/src/main/java/org/apache/airavata/mft/controller/MFTController.java
@@ -104,13 +104,13 @@ public class MFTController implements CommandLineRunner {
String transferId = key.substring(key.lastIndexOf("/") + 1);
Optional decodedValue = value.getValueAsBytes();
decodedValue.ifPresent(v -> {
-logger.info("Received transfer request : {} with id {}", v,
transferId);
TransferApiRequest transferRequest;
try {
TransferApiRequest.Builder builder =
TransferApiRequest.newBuilder();
builder.mergeFrom(v);
transferRequest = builder.build();
+logger.info("Received transfer request : {} with id {}",
transferRequest, transferId);
} catch (IOException e) {
logger.error("Failed to parse the transfer request {}", v,
e);
return;
[airavata-mft] branch master updated (0092f38 -> a93ad01)
This is an automated email from the ASF dual-hosted git repository. dimuthuupe pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airavata-mft.git from 0092f38 Updating build scripts add a93ad01 Adding missing config files to the standalone distribution No new revisions were added by this update. Summary of changes: .../src/main/assembly/standalone-service-bin-assembly.xml | 14 ++ .../resources/distribution/conf/application.properties | 7 ++- .../conf/resource-service-application.properties | 4 .../conf/secret-service-application.properties | 4 4 files changed, 12 insertions(+), 17 deletions(-) copy transport/dropbox-transport/src/main/resources/application.properties => standalone-service/src/main/resources/distribution/conf/resource-service-application.properties (87%) copy transport/gcp-transport/src/main/resources/application.properties => standalone-service/src/main/resources/distribution/conf/secret-service-application.properties (91%)
[airavata-mft] branch master updated: Updating build scripts
This is an automated email from the ASF dual-hosted git repository. dimuthuupe pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/airavata-mft.git The following commit(s) were added to refs/heads/master by this push: new 0092f38 Updating build scripts 0092f38 is described below commit 0092f38d604ce6050fd4d4eedda232e12e574e64 Author: DImuthuUpe AuthorDate: Sat Aug 5 04:14:47 2023 -0400 Updating build scripts --- scripts/build.sh | 6 -- scripts/log.sh | 23 --- scripts/start-mft.sh | 6 +- scripts/stop-mft.sh | 6 +- 4 files changed, 10 insertions(+), 31 deletions(-) diff --git a/scripts/build.sh b/scripts/build.sh index 2d495f6..a857f5b 100755 --- a/scripts/build.sh +++ b/scripts/build.sh @@ -21,7 +21,8 @@ cd ../ mvn clean install rm -rf airavata-mft mkdir -p airavata-mft -cp agent/target/MFT-Agent-0.01-bin.zip airavata-mft/ +cp agent/service/target/MFT-Agent-0.01-bin.zip airavata-mft/ +cp standalone-service/target/Standalone-Service-0.01-bin.zip airavata-mft/ cp controller/target/MFT-Controller-0.01-bin.zip airavata-mft/ cp services/resource-service/server/target/Resource-Service-0.01-bin.zip airavata-mft/ cp services/secret-service/server/target/Secret-Service-0.01-bin.zip airavata-mft/ @@ -31,10 +32,11 @@ cp scripts/start-mft.sh airavata-mft/ cp scripts/stop-mft.sh airavata-mft/ cp scripts/start-consul.sh airavata-mft/ cp scripts/stop-consul.sh airavata-mft/ -cp command-line/target/mft-command-line-0.01-SNAPSHOT-jar-with-dependencies.jar airavata-mft/mft-client.jar unzip -o airavata-mft/MFT-Agent-0.01-bin.zip -d airavata-mft/ rm airavata-mft/MFT-Agent-0.01-bin.zip +unzip -o airavata-mft/Standalone-Service-0.01-bin.zip -d airavata-mft/ +rm airavata-mft/Standalone-Service-0.01-bin.zip unzip -o airavata-mft/MFT-Controller-0.01-bin.zip -d airavata-mft/ rm airavata-mft/MFT-Controller-0.01-bin.zip unzip -o airavata-mft/Resource-Service-0.01-bin.zip -d airavata-mft/ diff --git a/scripts/log.sh b/scripts/log.sh index 50e5a95..35d7dba 100755 --- a/scripts/log.sh +++ b/scripts/log.sh @@ -18,20 +18,8 @@ # under the License. case $1 in -agent) -tail -100f ../airavata-mft/MFT-Agent-0.01/logs/airavata.log -;; -resource) -tail -100f ../airavata-mft/Resource-Service-0.01/logs/airavata.log -;; -secret) -tail -100f ../airavata-mft/Secret-Service-0.01/logs/airavata.log -;; -api) -tail -100f ../airavata-mft/API-Service-0.01/logs/airavata.log -;; -controller) -tail -100f ../airavata-mft/MFT-Controller-0.01/logs/airavata.log +mft) +tail -100f ../airavata-mft/Standalone-Service-0.01/logs/airavata.log ;; consul) tail -100f ../airavata-mft/consul/consul.log @@ -40,11 +28,8 @@ case $1 in echo "Usage: log.sh" echo "command options:" -echo " agentView logs of MFT Agent" -echo " resource View logs of MFT Resource Service" -echo " secret View logs of MFT Secret Service" -echo " api View logs of MFT API Service" -echo " controller View logs of MFT Controller" +echo " mft View logs of MFT" +echo " consul View logs of Consul" echo " -h Display this help and exit" shift exit 0 diff --git a/scripts/start-mft.sh b/scripts/start-mft.sh index 64215dd..8966a69 100755 --- a/scripts/start-mft.sh +++ b/scripts/start-mft.sh @@ -17,8 +17,4 @@ # specific language governing permissions and limitations # under the License. -./../airavata-mft/API-Service-0.01/bin/api-service-daemon.sh start -./../airavata-mft/MFT-Controller-0.01/bin/controller-daemon.sh start -./../airavata-mft/Resource-Service-0.01/bin/resource-service-daemon.sh start -./../airavata-mft/Secret-Service-0.01/bin/secret-service-daemon.sh start -./../airavata-mft/MFT-Agent-0.01/bin/agent-daemon.sh start \ No newline at end of file +./../airavata-mft/Standalone-Service-0.01/bin/standalone-service-daemon.sh start \ No newline at end of file diff --git a/scripts/stop-mft.sh b/scripts/stop-mft.sh index f9110ec..02ee1f4 100755 --- a/scripts/stop-mft.sh +++ b/scripts/stop-mft.sh @@ -17,8 +17,4 @@ # specific language governing permissions and limitations # under the License. -./../airavata-mft/API-Service-0.01/bin/api-service-daemon.sh stop -./../airavata-mft/MFT-Controller-0.01/bin/controller-daemon.sh stop -./../airavata-mft/Resource-Service-0.01/bin/resource-service-daemon.sh stop -./../airavata-mft/Secret-Service-0.01/bin/secret-service-daemon.sh stop -./../airavata-mft/MFT-Agent-0.01/bin/agent-daemon.sh stop \ No newline at end of file +./../airavata-mft/Standalone-Service-0.01/bin/standalone-service-daemon.sh stop \ No newline at end of file
[airavata-mft] branch master updated: Temporarily removing the agent starting class
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
The following commit(s) were added to refs/heads/master by this push:
new 190493a Temporarily removing the agent starting class
190493a is described below
commit 190493a5129161ab7a79bfff70dab9ed5cca13ba
Author: DImuthuUpe
AuthorDate: Fri Jul 21 14:03:49 2023 -0400
Temporarily removing the agent starting class
---
.../standalone/server/AgentServiceApplication.java | 37 --
1 file changed, 37 deletions(-)
diff --git
a/standalone-service/src/main/java/org/apache/airavata/mft/standalone/server/AgentServiceApplication.java
b/standalone-service/src/main/java/org/apache/airavata/mft/standalone/server/AgentServiceApplication.java
deleted file mode 100644
index bfa79ee..000
---
a/standalone-service/src/main/java/org/apache/airavata/mft/standalone/server/AgentServiceApplication.java
+++ /dev/null
@@ -1,37 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.airavata.mft.standalone.server;
-
-import org.springframework.boot.SpringApplication;
-import org.springframework.boot.autoconfigure.SpringBootApplication;
-import org.springframework.boot.autoconfigure.jdbc.DataSourceAutoConfiguration;
-import
org.springframework.boot.autoconfigure.jdbc.DataSourceTransactionManagerAutoConfiguration;
-import
org.springframework.boot.autoconfigure.orm.jpa.HibernateJpaAutoConfiguration;
-import org.springframework.context.annotation.ComponentScan;
-
-@ComponentScan(basePackages = {"org.apache.airavata.mft.agent"})
-@SpringBootApplication(exclude = {
-DataSourceAutoConfiguration.class,
-DataSourceTransactionManagerAutoConfiguration.class,
-HibernateJpaAutoConfiguration.class
-})
-public class AgentServiceApplication {
-public static void main(String[] args) {
-SpringApplication.run(AgentServiceApplication.class, args);
-}
-}
[airavata-mft] branch master updated: Configuration to overwrite already copied data
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
The following commit(s) were added to refs/heads/master by this push:
new 74c3ba5 Configuration to overwrite already copied data
74c3ba5 is described below
commit 74c3ba5d448cfbe2ef399599846ff5b91f3f077b
Author: DImuthuUpe
AuthorDate: Wed Jul 19 17:09:25 2023 -0400
Configuration to overwrite already copied data
---
.../java/org/apache/airavata/mft/agent/TransferOrchestrator.java | 5 -
agent/service/src/main/resources/agent-application.properties| 4 +++-
.../airavata/mft/standalone/server/AgentServiceApplication.java | 9 +
standalone-service/src/main/resources/application.properties | 3 ++-
.../src/main/resources/distribution/conf/application.properties | 3 ++-
5 files changed, 20 insertions(+), 4 deletions(-)
diff --git
a/agent/service/src/main/java/org/apache/airavata/mft/agent/TransferOrchestrator.java
b/agent/service/src/main/java/org/apache/airavata/mft/agent/TransferOrchestrator.java
index ecbc15e..77a1a30 100644
---
a/agent/service/src/main/java/org/apache/airavata/mft/agent/TransferOrchestrator.java
+++
b/agent/service/src/main/java/org/apache/airavata/mft/agent/TransferOrchestrator.java
@@ -66,6 +66,9 @@ public class TransferOrchestrator {
@org.springframework.beans.factory.annotation.Value("${agent.id}")
private String agentId;
+
@org.springframework.beans.factory.annotation.Value("${overwrite.existing}")
+private Boolean overwriteExisting;
+
@Autowired
private MFTConsulClient mftConsulClient;
@@ -142,7 +145,7 @@ public class TransferOrchestrator {
MetadataCollector dstMetadataCollector =
dstMetadataCollectorOp.orElseThrow(() -> new Exception("Could not find a
metadata collector for destination"));
dstMetadataCollector.init(destStorage, destSecret);
-if
(dstMetadataCollector.isAvailable(endpointPath.getDestinationPath())) {
+if (!overwriteExisting &&
dstMetadataCollector.isAvailable(endpointPath.getDestinationPath())) {
ResourceMetadata destinationMetadata =
dstMetadataCollector.getResourceMetadata(endpointPath.getDestinationPath(),
false);
if (destinationMetadata.getMetadataCase() ==
ResourceMetadata.MetadataCase.FILE &&
destinationMetadata.getFile().getResourceSize() ==
srcMetadata.getFile().getResourceSize()) {
diff --git a/agent/service/src/main/resources/agent-application.properties
b/agent/service/src/main/resources/agent-application.properties
index 045b6ce..900b65d 100644
--- a/agent/service/src/main/resources/agent-application.properties
+++ b/agent/service/src/main/resources/agent-application.properties
@@ -44,4 +44,6 @@ agent.transport.directory=plugins
#agent.transport.local.dma = true
agent.transport.local.buffLen = 16777216
agent.transport.s3.chunkSize = 10
-agent.transport.s3.multipart = true
\ No newline at end of file
+agent.transport.s3.multipart = true
+
+overwrite.existing=true
\ No newline at end of file
diff --git
a/standalone-service/src/main/java/org/apache/airavata/mft/standalone/server/AgentServiceApplication.java
b/standalone-service/src/main/java/org/apache/airavata/mft/standalone/server/AgentServiceApplication.java
index fbb2754..bfa79ee 100644
---
a/standalone-service/src/main/java/org/apache/airavata/mft/standalone/server/AgentServiceApplication.java
+++
b/standalone-service/src/main/java/org/apache/airavata/mft/standalone/server/AgentServiceApplication.java
@@ -18,9 +18,18 @@
package org.apache.airavata.mft.standalone.server;
import org.springframework.boot.SpringApplication;
+import org.springframework.boot.autoconfigure.SpringBootApplication;
+import org.springframework.boot.autoconfigure.jdbc.DataSourceAutoConfiguration;
+import
org.springframework.boot.autoconfigure.jdbc.DataSourceTransactionManagerAutoConfiguration;
+import
org.springframework.boot.autoconfigure.orm.jpa.HibernateJpaAutoConfiguration;
import org.springframework.context.annotation.ComponentScan;
@ComponentScan(basePackages = {"org.apache.airavata.mft.agent"})
+@SpringBootApplication(exclude = {
+DataSourceAutoConfiguration.class,
+DataSourceTransactionManagerAutoConfiguration.class,
+HibernateJpaAutoConfiguration.class
+})
public class AgentServiceApplication {
public static void main(String[] args) {
SpringApplication.run(AgentServiceApplication.class, args);
diff --git a/standalone-service/src/main/resources/application.properties
b/standalone-service/src/main/resources/application.properties
index fea9489..2b363e9 100644
--- a/standalone-service/src/main/resources/application.properties
+++ b/standalone-service/src/main/resources/application.properties
@@ -30,4 +30,5 @@ agent.concurrent.transfers=
[airavata-mft] branch master updated: Ability start the agent through standalone service
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
The following commit(s) were added to refs/heads/master by this push:
new cae3db4 Ability start the agent through standalone service
cae3db4 is described below
commit cae3db4e82e34ea0bdc1efddf92f9b691d33cb06
Author: DImuthuUpe
AuthorDate: Wed Jul 19 15:51:32 2023 -0400
Ability start the agent through standalone service
---
.../standalone/server/AgentServiceApplication.java | 28 ++
.../src/main/resources/application.properties | 4
.../distribution/bin/standalone-service-daemon.sh | 12 ++
.../distribution/conf/application.properties | 4
4 files changed, 48 insertions(+)
diff --git
a/standalone-service/src/main/java/org/apache/airavata/mft/standalone/server/AgentServiceApplication.java
b/standalone-service/src/main/java/org/apache/airavata/mft/standalone/server/AgentServiceApplication.java
new file mode 100644
index 000..fbb2754
--- /dev/null
+++
b/standalone-service/src/main/java/org/apache/airavata/mft/standalone/server/AgentServiceApplication.java
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.airavata.mft.standalone.server;
+
+import org.springframework.boot.SpringApplication;
+import org.springframework.context.annotation.ComponentScan;
+
+@ComponentScan(basePackages = {"org.apache.airavata.mft.agent"})
+public class AgentServiceApplication {
+public static void main(String[] args) {
+SpringApplication.run(AgentServiceApplication.class, args);
+}
+}
diff --git a/standalone-service/src/main/resources/application.properties
b/standalone-service/src/main/resources/application.properties
index 91e8595..fea9489 100644
--- a/standalone-service/src/main/resources/application.properties
+++ b/standalone-service/src/main/resources/application.properties
@@ -26,4 +26,8 @@ resource.service.port=7003
secret.service.host=localhost
secret.service.port=7003
agent.id=local-agent
+agent.concurrent.transfers=8
+agent.concurrent.chunked.threads=20
+agent.chunk.size=10
+agent.chunk.streaming.enabled=true
agent.transport.directory=plugins
\ No newline at end of file
diff --git
a/standalone-service/src/main/resources/distribution/bin/standalone-service-daemon.sh
b/standalone-service/src/main/resources/distribution/bin/standalone-service-daemon.sh
index 5bd7322..425f0f0 100644
---
a/standalone-service/src/main/resources/distribution/bin/standalone-service-daemon.sh
+++
b/standalone-service/src/main/resources/distribution/bin/standalone-service-daemon.sh
@@ -48,6 +48,17 @@ case $1 in
echo "$SERVICE_NAME is already running ..."
fi
;;
+start-agent)
+echo "Starting MFT Agent ..."
+if [ ! -f $PID_PATH_NAME ]; then
+nohup java ${JAVA_OPTS} -classpath "${AIRAVATA_CLASSPATH}" \
+org.apache.airavata.mft.standalone.server.AgentServiceApplication
${AIRAVATA_COMMAND} $* > $LOG_FILE 2>&1 &
+echo $! > $PID_PATH_NAME
+echo "MFT Agent started ..."
+else
+echo "MFT Agent is already running ..."
+fi
+;;
stop)
if [ -f $PID_PATH_NAME ]; then
PID=$(cat $PID_PATH_NAME);
@@ -102,6 +113,7 @@ case $1 in
echo "command options:"
echo " start Start server in daemon mode"
+echo " start-agent Start MFT Agent in daemon mode"
echo " stopStop server running in daemon mode"
echo " restart Restart server in daemon mode"
echo " -log Where to redirect stdout/stderr
(defaults to $DEFAULT_LOG_FILE)"
diff --git
a/standalone-service/src/main/resources/distribution/conf/application.properties
b/standalone-service/src/main/resources/distribution/conf/application.properties
index 91a37e7..5eaaf45 100644
---
a/standalone-service/src/main/resources/distribution/c
[airavata-mft] branch master updated: Supporting bulk transfers at cli
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
The following commit(s) were added to refs/heads/master by this push:
new 1a5958e Supporting bulk transfers at cli
1a5958e is described below
commit 1a5958ee5a193e3f84f776756e92db654a5a30d1
Author: DImuthuUpe
AuthorDate: Wed Jul 19 15:50:31 2023 -0400
Supporting bulk transfers at cli
---
python-cli/mft_cli/airavata_mft_cli/base.py| 7 ++
python-cli/mft_cli/airavata_mft_cli/operations.py | 83 +-
.../mft_cli/airavata_mft_cli/storage/http.py | 8 +--
python-cli/mft_cli/pyproject.toml | 3 +-
4 files changed, 94 insertions(+), 7 deletions(-)
diff --git a/python-cli/mft_cli/airavata_mft_cli/base.py
b/python-cli/mft_cli/airavata_mft_cli/base.py
index 1d1e2d3..553747f 100644
--- a/python-cli/mft_cli/airavata_mft_cli/base.py
+++ b/python-cli/mft_cli/airavata_mft_cli/base.py
@@ -37,6 +37,13 @@ def copy(source, destination):
except Exception as e:
exception_handler(e)
+@app.command("cp-list")
+def copy_list(source_storage_id, dest_storage_id, list_file):
+ try:
+operations.copy_list(source_storage_id, dest_storage_id, list_file)
+ except Exception as e:
+exception_handler(e)
+
@app.command("init")
def init_mft():
bootstrap.start_mft()
diff --git a/python-cli/mft_cli/airavata_mft_cli/operations.py
b/python-cli/mft_cli/airavata_mft_cli/operations.py
index 40fbe64..8ed867d 100644
--- a/python-cli/mft_cli/airavata_mft_cli/operations.py
+++ b/python-cli/mft_cli/airavata_mft_cli/operations.py
@@ -26,6 +26,7 @@ import time
import sys
sys.path.append('.')
from . import config as configcli
+import pandas as pd
def fetch_storage_and_secret_ids(storage_name):
client = mft_client.MFTClient(transfer_api_port =
configcli.transfer_api_port,
@@ -193,13 +194,93 @@ def copy(source, destination):
completed = 0
failed = 0
+ progress_percentage = 0
with typer.progressbar(length=100) as progress:
while 1:
state_resp = client.transfer_api.getTransferStateSummary(state_request)
- progress.update(int(state_resp.percentage * 100))
+ progress_percentage = int(state_resp.percentage * 100)
+ progress.update(progress_percentage - prev_percentage)
+ prev_percentage = progress_percentage
+
+ if (state_resp.percentage == 1.0):
+completed = len(state_resp.completed)
+failed = len(state_resp.failed)
+break
+
+ if (state_resp.state == "FAILED"):
+print("Transfer failed. Reason: " + state_resp.description)
+raise typer.Abort()
+ time.sleep(1)
+
+ print(f"Processed {completed + failed} files. Completed {completed}, Failed
{failed}.")
+
+def copy_list(source_storage_id, dest_storage_id, list_file):
+
+ source_storage_id, source_secret_id =
fetch_storage_and_secret_ids(source_storage_id)
+ dest_storage_id, dest_secret_id =
fetch_storage_and_secret_ids(dest_storage_id)
+
+ ## TODO : Check agent availability and deploy cloud agents if required
+
+ file_list = []
+ endpoint_paths = []
+ total_volume = 0
+
+ transfer_request = MFTTransferApi_pb2.TransferApiRequest(sourceStorageId =
source_storage_id,
+ sourceSecretId =
source_secret_id,
+
destinationStorageId = dest_storage_id,
+ destinationSecretId
= dest_secret_id,
+
optimizeTransferPath = False)
+ columns=['source', 'destination']
+ df = pd.read_csv(list_file, header=None, dtype=str, names=columns)
+ for i in range(len(df)):
+source_path = df['source'][i]
+destination_path = df['destination'][i]
+
+endpoint_paths.append(MFTTransferApi_pb2.EndpointPaths(sourcePath =
source_path,
+destinationPath = destination_path))
+
+ transfer_request.endpointPaths.extend(endpoint_paths)
+
+ confirm = typer.confirm("Total number of " + str(len(endpoint_paths)) +
+ " files to be transferred. Do you want to start the
transfer? ", True)
+
+ if not confirm:
+ raise typer.Abort()
+
+ client = mft_client.MFTClient(transfer_api_port =
configcli.transfer_api_port,
+transfer_api_secured =
configcli.transfer_api_secured,
+resource_service_host =
configcli.resource_service_host,
+resource_service_port =
configcli.resource_service_port,
+resource_service_secured =
configcli.resource_service_secured,
+secret_service_host =
configcli.secret_service_host,
+se
[airavata-mft] branch master updated: HTTP Transport implementation
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
The following commit(s) were added to refs/heads/master by this push:
new 3c1ddf6 HTTP Transport implementation
3c1ddf6 is described below
commit 3c1ddf69b817046331b3fdbe1d457bf1e02917c4
Author: DImuthuUpe
AuthorDate: Wed Jul 19 11:03:42 2023 -0400
HTTP Transport implementation
---
.../mft/agent/transport/ConnectorResolver.java | 3 +
.../agent/transport/MetadataCollectorResolver.java | 3 +
agent/stub/pom.xml | 2 +
agent/stub/src/main/proto/MFTAgentStubs.proto | 4 +
.../airavata/mft/api/handler/MFTApiHandler.java| 16 ++
.../airavata/mft/controller/RequestBuilder.java| 44 +++--
.../mft_cli/airavata_mft_cli/storage/__init__.py | 7 +-
.../mft_cli/airavata_mft_cli/storage/http.py | 80 +
python-cli/mft_cli/airavata_mft_cli/util.py| 8 +-
python-cli/mft_cli/pyproject.toml | 4 +-
python-sdk/setup.cfg | 2 +-
.../src/airavata_mft_sdk/MFTAgentStubs_pb2.py | 40 +++--
.../airavata_mft_sdk/common/StorageCommon_pb2.py | 13 +-
python-sdk/src/airavata_mft_sdk/generate-stubs.sh | 7 +
.../http/HTTPStorageService_pb2.py | 29 +++
.../http/HTTPStorageService_pb2_grpc.py| 199 +
.../src/airavata_mft_sdk/http/HTTPStorage_pb2.py | 115
.../airavata_mft_sdk/http/HTTPStorage_pb2_grpc.py | 4 +
.../airavata_mft_sdk/http/HttpCredential_pb2.py| 116
.../http/HttpCredential_pb2_grpc.py| 4 +
.../airavata_mft_sdk/http/HttpSecretService_pb2.py | 29 +++
.../http/HttpSecretService_pb2_grpc.py | 165 +
python-sdk/src/airavata_mft_sdk/http/__init__.py | 0
python-sdk/src/airavata_mft_sdk/mft_client.py | 4 +
.../mft/resource/client/StorageServiceClient.java | 5 +
.../resource/server/backend/ResourceBackend.java | 8 +
.../server/backend/sql/SQLResourceBackend.java | 46 -
.../backend/sql/entity/GenericResourceEntity.java | 2 +-
.../backend/sql/entity/HttpStorageEntity.java | 48 +
.../backend/sql/entity/ResolveStorageEntity.java | 2 +-
.../sql/repository/HttpStorageRepository.java | 30
.../server/handler/HttpServiceHandler.java | 107 +++
.../stub/src/main/proto/common/StorageCommon.proto | 1 +
.../stub/src/main/proto/http/HTTPStorage.proto | 66 +++
.../src/main/proto/http/HTTPStorageService.proto | 42 +
.../mft/secret/client/SecretServiceClient.java | 5 +
.../mft/secret/server/backend/SecretBackend.java | 6 +
.../server/backend/sql/SQLSecretBackend.java | 117
.../sql/entity/http/BasicAuthSecretEntity.java | 48 +
.../backend/sql/entity/http/HttpSecretEntity.java | 51 ++
.../sql/entity/http/TokenAuthSecretEntity.java | 36
.../repository/http/BasicAuthSecretRepository.java | 28 +++
.../sql/repository/http/HttpSecretRepository.java | 27 +++
.../repository/http/TokenAuthSecretRepository.java | 27 +++
.../secret/server/handler/HttpServiceHandler.java | 103 +++
.../stub/src/main/proto/http/HttpCredential.proto | 68 +++
.../src/main/proto/http/HttpSecretService.proto| 37
.../src/main/resources/application.properties | 2 +-
transport/http-transport/pom.xml | 74
.../src/main/assembly/transport-assembly.xml | 53 ++
.../http/HttpIncomingStreamingConnector.java | 108 +++
.../mft/trnasport/http/HttpMetadataCollector.java | 87 +
transport/pom.xml | 1 +
53 files changed, 2085 insertions(+), 48 deletions(-)
diff --git
a/agent/service/src/main/java/org/apache/airavata/mft/agent/transport/ConnectorResolver.java
b/agent/service/src/main/java/org/apache/airavata/mft/agent/transport/ConnectorResolver.java
index ba2186f..44b1464 100644
---
a/agent/service/src/main/java/org/apache/airavata/mft/agent/transport/ConnectorResolver.java
+++
b/agent/service/src/main/java/org/apache/airavata/mft/agent/transport/ConnectorResolver.java
@@ -48,6 +48,9 @@ public final class ConnectorResolver {
case "LOCAL":
className =
"org.apache.airavata.mft.transport.local.LocalIncomingStreamingConnector";
break;
+case "HTTP":
+className =
"org.apache.airavata.mft.trnasport.http.HttpIncomingStreamingConnector";
+break;
}
if (className != null) {
diff --git
a/agent/service/src/main/java/org/apache/airavata/mft/agent/transport/MetadataCollectorResolver.java
b/agent/service/src/main/java/org/apache/airavata/mft/agent/transport/MetadataCollectorRes
[airavata-mft] branch master updated: Adding missing bin files for Agent
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
The following commit(s) were added to refs/heads/master by this push:
new 6c09eb8 Adding missing bin files for Agent
6c09eb8 is described below
commit 6c09eb88c74535ae5ad5fd3583c8f54127415946
Author: DImuthuUpe
AuthorDate: Wed Jul 12 02:20:22 2023 -0400
Adding missing bin files for Agent
---
.../resources/distribution/bin/agent-daemon.sh | 113 +
.../src/main/resources/distribution/bin/agent.sh | 71 +
.../src/main/resources/distribution/bin/setenv.sh | 46 +
3 files changed, 230 insertions(+)
diff --git a/agent/service/src/main/resources/distribution/bin/agent-daemon.sh
b/agent/service/src/main/resources/distribution/bin/agent-daemon.sh
new file mode 100644
index 000..20272a2
--- /dev/null
+++ b/agent/service/src/main/resources/distribution/bin/agent-daemon.sh
@@ -0,0 +1,113 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+. `dirname $0`/setenv.sh
+# Capture user's working dir before changing directory
+CWD="$PWD"
+cd ${AIRAVATA_HOME}/bin
+LOGO_FILE="logo.txt"
+
+JAVA_OPTS="-Dspring.config.location=${AIRAVATA_HOME}/conf/
-Dairavata.home=${AIRAVATA_HOME}
-Dlog4j.configurationFile=file:${AIRAVATA_HOME}/conf/log4j2.xml"
+AIRAVATA_COMMAND=""
+EXTRA_ARGS=""
+SERVERS=""
+LOGO=true
+IS_SUBSET=false
+SUBSET=""
+DEFAULT_LOG_FILE="${AIRAVATA_HOME}/logs/airavata-daemon.out"
+LOG_FILE=$DEFAULT_LOG_FILE
+
+SERVICE_NAME="MFT Agent"
+PID_PATH_NAME="${AIRAVATA_HOME}/bin/service-pid"
+
+case $1 in
+start)
+echo "Starting $SERVICE_NAME ..."
+if [ ! -f $PID_PATH_NAME ]; then
+nohup java ${JAVA_OPTS} -classpath "${AIRAVATA_CLASSPATH}" \
+org.apache.airavata.mft.agent.MFTAgent ${AIRAVATA_COMMAND} $* >
$LOG_FILE 2>&1 &
+echo $! > $PID_PATH_NAME
+echo "$SERVICE_NAME started ..."
+else
+echo "$SERVICE_NAME is already running ..."
+fi
+;;
+stop)
+if [ -f $PID_PATH_NAME ]; then
+PID=$(cat $PID_PATH_NAME);
+echo "$SERVICE_NAME stoping ..."
+kill $PID;
+RETRY=0
+while kill -0 $PID 2> /dev/null; do
+echo "Waiting for the process $PID to be stopped"
+RETRY=`expr ${RETRY} + 1`
+if [ "${RETRY}" -gt "20" ]
+then
+echo "Forcefully killing the process as it is not
responding ..."
+kill -9 $PID
+fi
+sleep 1
+done
+echo "$SERVICE_NAME stopped ..."
+rm $PID_PATH_NAME
+else
+echo "$SERVICE_NAME is not running ..."
+fi
+;;
+restart)
+if [ -f $PID_PATH_NAME ]; then
+PID=$(cat $PID_PATH_NAME);
+echo "$SERVICE_NAME stopping ...";
+kill $PID;
+RETRY=0
+while kill -0 $PID 2> /dev/null; do
+echo "Waiting for the process $PID to be stopped"
+RETRY=`expr ${RETRY} + 1`
+if [ "${RETRY}" -gt "20" ]
+then
+echo "Forcefully killing the process as it is not
responding ..."
+kill -9 $PID
+fi
+sleep 1
+done
+echo "$SERVICE_NAME stopped ...";
+rm $PID_PATH_NAME
+echo "$SERVICE_NAME starting ..."
+nohup java ${JAVA_OPTS} -classpath "${AIRAVATA_CLASSPATH}" \
+org.apache.airavata.mft.agent.MFTAgent ${AIRAVATA_COMMAND} $* >
$LOG_FILE 2>&1 &
+echo $! > $PID_PATH_NAME
+echo "$SERVICE_NAME started ..."
+else
+echo &
[airavata-mft] 01/03: Handling scp secrets with no public key
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
commit e6492987f6e14f668238a2980efb5c43189d712f
Author: DImuthuUpe
AuthorDate: Wed Jun 28 20:14:05 2023 -0400
Handling scp secrets with no public key
---
.../org/apache/airavata/mft/transport/scp/SCPIncomingConnector.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/transport/scp-transport/src/main/java/org/apache/airavata/mft/transport/scp/SCPIncomingConnector.java
b/transport/scp-transport/src/main/java/org/apache/airavata/mft/transport/scp/SCPIncomingConnector.java
index dea4c6e..1157214 100644
---
a/transport/scp-transport/src/main/java/org/apache/airavata/mft/transport/scp/SCPIncomingConnector.java
+++
b/transport/scp-transport/src/main/java/org/apache/airavata/mft/transport/scp/SCPIncomingConnector.java
@@ -58,7 +58,7 @@ public final class SCPIncomingConnector implements
IncomingStreamingConnector {
scpStorage.getHost(),
scpStorage.getPort(),
scpSecret.getPrivateKey().getBytes(),
-scpSecret.getPublicKey().getBytes(),
+scpSecret.getPublicKey().equals("")? null :
scpSecret.getPublicKey().getBytes(),
scpSecret.getPassphrase().equals("")? null :
scpSecret.getPassphrase().getBytes());
if (session == null) {
@@ -94,7 +94,7 @@ public final class SCPIncomingConnector implements
IncomingStreamingConnector {
while (true) {
int c = checkAck(in);
-if (c != 'C') {
+if (c != 'C' && c != 'c' ) {
break;
}
[airavata-mft] branch master updated (e530d65 -> db231dd)
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
from e530d65 grpcio grpcio tools issue 104
new e649298 Handling scp secrets with no public key
new e0c5226 Adding local outgoing streaming connector
new db231dd Terminating pending transfers if an exception was captured
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../mft/agent/transport/ConnectorResolver.java | 3 +++
.../airavata/mft/controller/MFTController.java | 16 ---
...r.java => LocalOutgoingStreamingConnector.java} | 24 +-
.../mft/transport/scp/SCPIncomingConnector.java| 4 ++--
4 files changed, 28 insertions(+), 19 deletions(-)
copy
transport/local-transport/src/main/java/org/apache/airavata/mft/transport/local/{LocalIncomingStreamingConnector.java
=> LocalOutgoingStreamingConnector.java} (70%)
[airavata-mft] 03/03: Terminating pending transfers if an exception was captured
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
commit db231ddc775e67dbab55d35ebaab539a5a79d3f4
Author: DImuthuUpe
AuthorDate: Wed Jun 28 20:15:43 2023 -0400
Terminating pending transfers if an exception was captured
---
.../apache/airavata/mft/controller/MFTController.java| 16 +---
1 file changed, 13 insertions(+), 3 deletions(-)
diff --git
a/controller/src/main/java/org/apache/airavata/mft/controller/MFTController.java
b/controller/src/main/java/org/apache/airavata/mft/controller/MFTController.java
index eff5d04..110f6d2 100644
---
a/controller/src/main/java/org/apache/airavata/mft/controller/MFTController.java
+++
b/controller/src/main/java/org/apache/airavata/mft/controller/MFTController.java
@@ -304,17 +304,27 @@ public class MFTController implements CommandLineRunner {
if (value.getValueAsString().isPresent()) {
logger.debug("Pending " + value.getKey() + " : " +
value.getValueAsString().get());
+String transferId =
value.getKey().substring(value.getKey().lastIndexOf("/") + 1);
try {
TransferApiRequest.Builder builder =
TransferApiRequest.newBuilder();
JsonFormat.parser().merge(value.getValueAsString().get(),
builder);
TransferApiRequest transferRequest = builder.build();
-
-String transferId =
value.getKey().substring(value.getKey().lastIndexOf("/") + 1);
AgentTransferRequest.Builder agentTransferRequest =
requestBuilder.prepareAgentTransferRequest(transferRequest);
pathOptimizer.handleTransferRequest(transferId,
transferRequest, agentTransferRequest, value.getKey());
} catch (Exception e) {
-logger.error("Failed to process pending transfer in key
{}", value.getKey(), e);
+logger.error("Failed to process pending transfer in key
{}. Deleting from queue", value.getKey(), e);
+try {
+mftConsulClient.saveTransferState(transferId, null,
new TransferState()
+.setUpdateTimeMils(System.currentTimeMillis())
+.setState("FAILED").setPercentage(0)
+.setPublisher("controller")
+.setDescription("Failed to process pending
transfer "));
+
mftConsulClient.getKvClient().deleteKey(value.getKey());
+} catch (Exception ex) {
+logger.warn("Failed to update state of transfer {} to
FAILED", transferId);
+// Ignore
+}
}
}
});
[airavata-mft] 02/03: Adding local outgoing streaming connector
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
commit e0c5226b58a94a57f79d1bee3f4d216489301a32
Author: DImuthuUpe
AuthorDate: Wed Jun 28 20:14:56 2023 -0400
Adding local outgoing streaming connector
---
.../mft/agent/transport/ConnectorResolver.java | 3 ++
.../local/LocalOutgoingStreamingConnector.java | 53 ++
2 files changed, 56 insertions(+)
diff --git
a/agent/service/src/main/java/org/apache/airavata/mft/agent/transport/ConnectorResolver.java
b/agent/service/src/main/java/org/apache/airavata/mft/agent/transport/ConnectorResolver.java
index 0dee097..ba2186f 100644
---
a/agent/service/src/main/java/org/apache/airavata/mft/agent/transport/ConnectorResolver.java
+++
b/agent/service/src/main/java/org/apache/airavata/mft/agent/transport/ConnectorResolver.java
@@ -75,6 +75,9 @@ public final class ConnectorResolver {
case "GCS":
className =
"org.apache.airavata.mft.transport.gcp.GCSOutgoingStreamingConnector";
break;
+case "LOCAL":
+className =
"org.apache.airavata.mft.transport.local.LocalOutgoingStreamingConnector";
+break;
}
diff --git
a/transport/local-transport/src/main/java/org/apache/airavata/mft/transport/local/LocalOutgoingStreamingConnector.java
b/transport/local-transport/src/main/java/org/apache/airavata/mft/transport/local/LocalOutgoingStreamingConnector.java
new file mode 100644
index 000..87d0563
--- /dev/null
+++
b/transport/local-transport/src/main/java/org/apache/airavata/mft/transport/local/LocalOutgoingStreamingConnector.java
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.airavata.mft.transport.local;
+
+import org.apache.airavata.mft.core.api.ConnectorConfig;
+import org.apache.airavata.mft.core.api.OutgoingStreamingConnector;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.FileOutputStream;
+import java.io.OutputStream;
+
+public class LocalOutgoingStreamingConnector implements
OutgoingStreamingConnector {
+
+private String resourcePath;
+
+private static final Logger logger =
LoggerFactory.getLogger(LocalOutgoingStreamingConnector.class);
+
+@Override
+public void init(ConnectorConfig connectorConfig) throws Exception {
+this.resourcePath = connectorConfig.getResourcePath();
+}
+
+@Override
+public void complete() throws Exception {
+logger.info("File {} successfully written", this.resourcePath);
+}
+
+@Override
+public void failed() throws Exception {
+logger.error("Failed while writing file {}", this.resourcePath);
+}
+
+@Override
+public OutputStream fetchOutputStream() throws Exception {
+return new FileOutputStream(this.resourcePath);
+}
+}
[airavata] branch airavata-v2-refactoring updated: Adding data movement task with MFT integration
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch airavata-v2-refactoring
in repository https://gitbox.apache.org/repos/asf/airavata.git
The following commit(s) were added to refs/heads/airavata-v2-refactoring by
this push:
new 1069f3265c Adding data movement task with MFT integration
1069f3265c is described below
commit 1069f3265cb4dadae155014bb4670b559b6bc1ca
Author: DImuthuUpe
AuthorDate: Wed Jun 28 10:38:41 2023 -0400
Adding data movement task with MFT integration
---
modules/airavata-apis/airavata-apis-server/pom.xml | 9 +
.../apis/scheduling/ExperimentLauncher.java| 36 +++-
.../apis/workflow/task/data/DataMovementTask.java | 209 -
.../workflow/task/ec2/CreateEC2InstanceTask.java | 14 ++
.../src/main/resources/application.properties | 9 +-
pom.xml| 2 +-
6 files changed, 266 insertions(+), 13 deletions(-)
diff --git a/modules/airavata-apis/airavata-apis-server/pom.xml
b/modules/airavata-apis/airavata-apis-server/pom.xml
index 78e7f3bdb4..03802bdbf3 100644
--- a/modules/airavata-apis/airavata-apis-server/pom.xml
+++ b/modules/airavata-apis/airavata-apis-server/pom.xml
@@ -31,8 +31,17 @@
com.google.protobuf
protobuf-java
+
+org.bouncycastle
+bcprov-jdk15on
+
+
+org.bouncycastle
+bcprov-jdk15on
+${bouncy.castle.version}
+
org.apache.airavata
mft-agent-service
diff --git
a/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/scheduling/ExperimentLauncher.java
b/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/scheduling/ExperimentLauncher.java
index e0c6472a15..f7e2cdc3d7 100644
---
a/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/scheduling/ExperimentLauncher.java
+++
b/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/scheduling/ExperimentLauncher.java
@@ -74,11 +74,6 @@ public class ExperimentLauncher {
Map taskMap = new HashMap<>();
-DataMovementTask dataMovementTask = new DataMovementTask();
-dataMovementTask.setTaskId(UUID.randomUUID().toString());
-
-taskMap.put(dataMovementTask.getTaskId(), dataMovementTask);
-
EC2Backend ec2Backend = EC2Backend.newBuilder()
.setAwsCredentialId(s3Secret.getSecretId())
.setLoginUserName("ubuntu")
@@ -92,9 +87,31 @@ public class ExperimentLauncher {
ec2InstanceTask.setSecretServiceHost("localhost");
ec2InstanceTask.setSecretServicePort(7002);
ec2InstanceTask.setUserToken("token");
-
taskMap.put(ec2InstanceTask.getTaskId(), ec2InstanceTask);
+DataMovementTask dataMovementTask = new DataMovementTask();
+dataMovementTask.setTaskId(UUID.randomUUID().toString());
+dataMovementTask.setSecretServiceHost("localhost");
+dataMovementTask.setSecretServicePort(7002);
+dataMovementTask.setTransferServiceHost("localhost");
+dataMovementTask.setTransferServicePort(7002);
+dataMovementTask.setResourceServiceHost("localhost");
+dataMovementTask.setResourceServicePort(7002);
+dataMovementTask.setUserToken("token");
+
dataMovementTask.setSourceStorageId("504643b6-f813-4aa1-8e66-2533cb4f837c");
+dataMovementTask.setSourceCredentialId("");
+
dataMovementTask.setSourcePath("/Users/dwannipu/Downloads/IMG-9309.jpg");
+dataMovementTask.setDestinationPath("/tmp/IMG-9309.jpg");
+dataMovementTask.setDestinationStorageId("");
+dataMovementTask.setDestinationCredentialId("");
+
dataMovementTask.overrideParameterFromWorkflowContext("destinationStorageId",
// Loading context parameter from previous Task
+CreateEC2InstanceTask.EC2_INSTANCE_STORAGE_ID);
+
dataMovementTask.overrideParameterFromWorkflowContext("destinationCredentialId",
+CreateEC2InstanceTask.EC2_INSTANCE_SECRET_ID);
+
+
+taskMap.put(dataMovementTask.getTaskId(), dataMovementTask);
+
DestroyEC2InstanceTask destroyEC2InstanceTask = new
DestroyEC2InstanceTask();
destroyEC2InstanceTask.setTaskId(UUID.randomUUID().toString());
destroyEC2InstanceTask.setEc2Backend(ec2Backend);
@@ -104,12 +121,11 @@ public class ExperimentLauncher {
destroyEC2InstanceTask.setInstanceId(""); // Override by workflow
destroyEC2InstanceTask.overrideParameterFromWorkflowContext("instanceId",
Crea
[airavata] branch airavata-v2-refactoring updated: Bundling MFT with main service
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch airavata-v2-refactoring
in repository https://gitbox.apache.org/repos/asf/airavata.git
The following commit(s) were added to refs/heads/airavata-v2-refactoring by
this push:
new 5fe4af33d2 Bundling MFT with main service
5fe4af33d2 is described below
commit 5fe4af33d2619aa5f9ff8dad714db2eea4165905
Author: DImuthuUpe
AuthorDate: Wed Jun 28 00:55:55 2023 -0400
Bundling MFT with main service
---
modules/airavata-apis/airavata-apis-server/pom.xml | 50 ++
.../java/org/apache/airavata/apis/APIRunner.java | 9 ++--
.../{api.properties => application.properties} | 6 ++-
3 files changed, 60 insertions(+), 5 deletions(-)
diff --git a/modules/airavata-apis/airavata-apis-server/pom.xml
b/modules/airavata-apis/airavata-apis-server/pom.xml
index 5014124e93..78e7f3bdb4 100644
--- a/modules/airavata-apis/airavata-apis-server/pom.xml
+++ b/modules/airavata-apis/airavata-apis-server/pom.xml
@@ -22,6 +22,39 @@
+
+org.apache.airavata
+mft-controller
+0.01-SNAPSHOT
+
+
+com.google.protobuf
+protobuf-java
+
+
+
+
+org.apache.airavata
+mft-agent-service
+0.01-SNAPSHOT
+
+
+com.google.protobuf
+protobuf-java
+
+
+
+
+org.apache.airavata
+mft-api-service
+0.01-SNAPSHOT
+
+
+com.google.protobuf
+protobuf-java
+
+
+
org.apache.airavata
mft-secret-service-client
@@ -55,6 +88,18 @@
+
+org.apache.airavata
+mft-api-client
+0.01-SNAPSHOT
+
+
+
+com.google.protobuf
+protobuf-java
+
+
+
org.apache.airavata
airavata-apis-stub
@@ -156,6 +201,11 @@
aws-java-sdk
${aws.sdk.version}
+
+org.awaitility
+awaitility
+4.2.0
+
1.12.372
diff --git
a/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/APIRunner.java
b/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/APIRunner.java
index c3f9d5a9cb..5018571b9a 100644
---
a/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/APIRunner.java
+++
b/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/APIRunner.java
@@ -4,17 +4,18 @@ import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.autoconfigure.domain.EntityScan;
import org.springframework.context.annotation.ComponentScan;
-import org.springframework.context.annotation.PropertySource;
import org.springframework.data.jpa.repository.config.EnableJpaAuditing;
@SpringBootApplication
-@ComponentScan(basePackages = {"org.apache.airavata",
+@ComponentScan(basePackages = {"org.apache.airavata.apis",
"org.apache.airavata.mft.secret.server.handler",
-"org.apache.airavata.mft.resource.server.handler"
+"org.apache.airavata.mft.resource.server.handler",
+"org.apache.airavata.mft.api.handler",
+"org.apache.airavata.mft.controller",
+"org.apache.airavata.mft.agent"
})
@EntityScan(basePackages = {"org.apache.airavata"})
@EnableJpaAuditing
-@PropertySource(value = "classpath:api.properties")
public class APIRunner {
public static void main(String[] args) {
diff --git
a/modules/airavata-apis/airavata-apis-server/src/main/resources/api.properties
b/modules/airavata-apis/airavata-apis-server/src/main/resources/application.properties
similarity index 87%
rename from
modules/airavata-apis/airavata-apis-server/src/main/resources/api.properties
rename to
modules/airavata-apis/airavata-apis-server/src/main/resources/application.properties
index a9fd83fea0..e186b72272 100644
---
a/modules/airavata-apis/airavata-apis-server/src/main/resources/api.properties
+++
b/modules/airavata-apis/airavata-apis-server/src/main/resources/application.properties
@@ -13,4 +13,8 @@
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-#
\ No newline at end of file
+#
+
+spring.main.allo
[airavata] branch airavata-v2-refactoring updated: Supporting inter-task communication. Passing task params through workflow context
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch airavata-v2-refactoring
in repository https://gitbox.apache.org/repos/asf/airavata.git
The following commit(s) were added to refs/heads/airavata-v2-refactoring by
this push:
new bdfd0e3b68 Supporting inter-task communication. Passing task params
through workflow context
bdfd0e3b68 is described below
commit bdfd0e3b686a7cd6a1b13cddd7db0889ef622ee4
Author: DImuthuUpe
AuthorDate: Tue Jun 27 08:15:45 2023 -0400
Supporting inter-task communication. Passing task params through workflow
context
---
modules/airavata-apis/airavata-apis-server/pom.xml | 28 +++
.../apis/db/entity/backend/EC2BackendEntity.java | 11 +
.../apis/scheduling/ExperimentLauncher.java| 48 -
.../airavata/apis/workflow/WorkflowExecutor.java | 3 +-
.../apis/workflow/task/common/BaseTask.java| 32 ++-
.../apis/workflow/task/common/TaskUtil.java| 79 ---
.../workflow/task/ec2/CreateEC2InstanceTask.java | 233 +
.../workflow/task/ec2/DestroyEC2InstanceTask.java | 116 +-
.../src/main/proto/execution/experiment_stub.proto | 5 +-
9 files changed, 520 insertions(+), 35 deletions(-)
diff --git a/modules/airavata-apis/airavata-apis-server/pom.xml
b/modules/airavata-apis/airavata-apis-server/pom.xml
index eff620301e..5014124e93 100644
--- a/modules/airavata-apis/airavata-apis-server/pom.xml
+++ b/modules/airavata-apis/airavata-apis-server/pom.xml
@@ -22,6 +22,28 @@
+
+org.apache.airavata
+mft-secret-service-client
+0.01-SNAPSHOT
+
+
+com.google.protobuf
+protobuf-java
+
+
+
+
+org.apache.airavata
+mft-resource-service-client
+0.01-SNAPSHOT
+
+
+com.google.protobuf
+protobuf-java
+
+
+
org.apache.airavata
mft-resource-service-server
@@ -129,8 +151,14 @@
+
+com.amazonaws
+aws-java-sdk
+${aws.sdk.version}
+
+1.12.372
18
18
UTF-8
diff --git
a/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/db/entity/backend/EC2BackendEntity.java
b/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/db/entity/backend/EC2BackendEntity.java
index cfd8d8b4d2..87a7ce5cc0 100644
---
a/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/db/entity/backend/EC2BackendEntity.java
+++
b/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/db/entity/backend/EC2BackendEntity.java
@@ -17,6 +17,9 @@ public class EC2BackendEntity extends ComputeBackendEntity {
@Column
String awsCredentialId;
+@Column
+String loginUserName;
+
public String getFlavor() {
return flavor;
}
@@ -48,4 +51,12 @@ public class EC2BackendEntity extends ComputeBackendEntity {
public void setImageId(String imageId) {
this.imageId = imageId;
}
+
+public String getLoginUserName() {
+return loginUserName;
+}
+
+public void setLoginUserName(String loginUserName) {
+this.loginUserName = loginUserName;
+}
}
diff --git
a/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/scheduling/ExperimentLauncher.java
b/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/scheduling/ExperimentLauncher.java
index 1d77d6ab56..e0c6472a15 100644
---
a/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/scheduling/ExperimentLauncher.java
+++
b/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/scheduling/ExperimentLauncher.java
@@ -3,6 +3,7 @@ package org.apache.airavata.apis.scheduling;
import org.apache.airavata.api.execution.ExperimentLaunchRequest;
import org.apache.airavata.api.execution.stubs.EC2Backend;
import org.apache.airavata.api.execution.stubs.Experiment;
+import org.apache.airavata.api.execution.stubs.RunConfiguration;
import org.apache.airavata.apis.service.ExecutionService;
import org.apache.airavata.apis.workflow.task.common.BaseTask;
import org.apache.airavata.apis.workflow.task.common.OutPort;
@@ -11,6 +12,11 @@ import
org.apache.airavata.apis.workflow.task.common.annotation.TaskDef;
import org.apache.airavata.apis.workflow.task.common.annotation.TaskOutPort;
import org.apache.airavata.apis.workflow.task.data.DataMovementTask;
import org.apache.airavata.apis.workflow.task.ec2.CreateEC2InstanceTask;
+import org.apache.airavata.apis.workflow.task.ec2
[airavata-mft] branch master updated: grpcio grpcio tools issue 104
This is an automated email from the ASF dual-hosted git repository. dimuthuupe pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/airavata-mft.git The following commit(s) were added to refs/heads/master by this push: new e530d65 grpcio grpcio tools issue 104 e530d65 is described below commit e530d65a7590b267a5b0ccb8e344efdf3e2cd50f Author: jfcsgthmbvnztjkcfcvjqjr AuthorDate: Sat Jun 24 20:17:07 2023 -0400 grpcio grpcio tools issue 104 --- python-sdk/pyproject.toml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/python-sdk/pyproject.toml b/python-sdk/pyproject.toml index 9970627..a169d51 100644 --- a/python-sdk/pyproject.toml +++ b/python-sdk/pyproject.toml @@ -17,5 +17,5 @@ # under the License. [build-system] -requires = ["setuptools>=42", "grpcio==1.46.3", "grpcio-tools==1.46.3"] +requires = ["setuptools>=42"] build-backend = "setuptools.build_meta" \ No newline at end of file
[airavata-mft] 02/07: Updated code to check for java run time
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
commit ee1c556fd3430844b19ba37ad027e9f6db9d6dcc
Author: jfcsgthmbvnztjkcfcvjqjr
AuthorDate: Fri Jun 23 22:25:00 2023 -0400
Updated code to check for java run time
---
python-cli/mft_cli/airavata_mft_cli/bootstrap.py | 286 +--
python-cli/mft_cli/venv/pyvenv.cfg | 3 +
2 files changed, 165 insertions(+), 124 deletions(-)
diff --git a/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
b/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
index da704ba..03777f8 100644
--- a/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
+++ b/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
@@ -20,6 +20,8 @@ import typer
import requests
import os
import zipfile
+from subprocess import check_output
+from subprocess import STDOUT
from subprocess import call
from subprocess import Popen
from pathlib import Path
@@ -27,152 +29,188 @@ from sys import platform
import shutil
import time
-def download_and_unarchive(url, download_path, extract_dir =
os.path.join(os.path.expanduser('~'), ".mft/")):
- response = requests.get(url, stream=True)
- file_size = int(response.headers['Content-Length'])
- with typer.progressbar(length=file_size) as progress:
-with open(download_path, "wb") as handle:
- for data in response.iter_content(chunk_size=8192 * 2):
-progress.update(len(data))
-handle.write(data)
+def download_and_unarchive(url, download_path,
extract_dir=os.path.join(os.path.expanduser('~'), ".mft/")):
+response = requests.get(url, stream=True)
+file_size = int(response.headers['Content-Length'])
+with typer.progressbar(length=file_size) as progress:
+with open(download_path, "wb") as handle:
+for data in response.iter_content(chunk_size=8192 * 2):
+progress.update(len(data))
+handle.write(data)
- print("Un archiving ")
- with zipfile.ZipFile(download_path,"r") as zip_ref:
-zip_ref.extractall(extract_dir)
+print("Un archiving ")
+with zipfile.ZipFile(download_path, "r") as zip_ref:
+zip_ref.extractall(extract_dir)
+
+os.remove(download_path)
- os.remove(download_path)
def restart_service(bin_path, daemon_script_name):
- current_dir = os.getcwd()
- try:
-os.chdir(bin_path)
-os.chmod(daemon_script_name, 0o744)
-rc = call(["./" + daemon_script_name, "stop"])
-rc = call(["./" + daemon_script_name, "start"])
- finally:
-os.chdir(current_dir)
+current_dir = os.getcwd()
+try:
+os.chdir(bin_path)
+os.chmod(daemon_script_name, 0o744)
+rc = call(["./" + daemon_script_name, "stop"])
+rc = call(["./" + daemon_script_name, "start"])
+finally:
+os.chdir(current_dir)
+
def stop_service(bin_path, daemon_script_name):
- current_dir = os.getcwd()
- try:
-os.chdir(bin_path)
-os.chmod(daemon_script_name, 0o744)
-rc = call(["./" + daemon_script_name, "stop"])
- finally:
-os.chdir(current_dir)
-
-
-"""
- Issue 96: https://github.com/apache/airavata-mft/issues/96
-"""
+current_dir = os.getcwd()
+try:
+os.chdir(bin_path)
+os.chmod(daemon_script_name, 0o744)
+rc = call(["./" + daemon_script_name, "stop"])
+finally:
+os.chdir(current_dir)
+
+
+def check_java_version_method(required_version):
+"""
+Issue 96: https://github.com/apache/airavata-mft/issues/96
+
+References:
+-
+
https://stackoverflow.com/questions/31807882/get-java-version-number-from-python
+
https://stackoverflow.com/questions/1332598/how-to-determine-whether-java-is-installed-on-a-system-through-python
+"""
+if shutil.which("java"):
+res = check_output(['java', '-version'], stderr=STDOUT).decode('utf-8')
+system_version = ''
+count = 0
+for c in res:
+if c == '"' or count == 1:
+if (c == '.' or c == '"') and count == 1:
+break
+if count == 0:
+count += 1
+continue
+system_version += c
+
+system_version = int(system_version)
+if system_version < required_version:
+print("Airavata MFT requires Java version 11 or higher")
+print("If you have more than one version of java please set java
version 11 or higher to the path")
+raise typer.Exit()
+else:
+print("Java is either not installed or path hasn't
[airavata-mft] 04/07: format fixes
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
commit 08b66828a236dd065b9f3014f0692132999c2c9e
Author: jfcsgthmbvnztjkcfcvjqjr
AuthorDate: Fri Jun 23 22:57:30 2023 -0400
format fixes
---
python-cli/mft_cli/airavata_mft_cli/bootstrap.py | 26 +++-
python-cli/mft_cli/venv/pyvenv.cfg | 3 ---
2 files changed, 12 insertions(+), 17 deletions(-)
diff --git a/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
b/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
index b9884bd..6bb96ac 100644
--- a/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
+++ b/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
@@ -30,7 +30,7 @@ import shutil
import time
-def download_and_unarchive(url, download_path,
extract_dir=os.path.join(os.path.expanduser('~'), ".mft/")):
+def download_and_unarchive(url, download_path, extract_dir =
os.path.join(os.path.expanduser('~'), ".mft/")):
response = requests.get(url, stream=True)
file_size = int(response.headers['Content-Length'])
with typer.progressbar(length=file_size) as progress:
@@ -40,14 +40,14 @@ def download_and_unarchive(url, download_path,
extract_dir=os.path.join(os.path.
handle.write(data)
print("Un archiving ")
- with zipfile.ZipFile(download_path, "r") as zip_ref:
+ with zipfile.ZipFile(download_path,"r") as zip_ref:
zip_ref.extractall(extract_dir)
os.remove(download_path)
def restart_service(bin_path, daemon_script_name):
- current_dir = os.getcwd()
+ current_dir = os.getcwd()
try:
os.chdir(bin_path)
os.chmod(daemon_script_name, 0o744)
@@ -56,9 +56,8 @@ def restart_service(bin_path, daemon_script_name):
finally:
os.chdir(current_dir)
-
def stop_service(bin_path, daemon_script_name):
- current_dir = os.getcwd()
+ current_dir = os.getcwd()
try:
os.chdir(bin_path)
os.chmod(daemon_script_name, 0o744)
@@ -126,7 +125,7 @@ def start_mft():
zip_path = os.path.join(os.path.expanduser('~'), ".mft/consul.zip")
download_and_unarchive(consul_url, zip_path,
os.path.join(os.path.expanduser('~'), ".mft/"))
- current_dir = os.getcwd()
+ current_dir = os.getcwd()
try:
os.chdir(os.path.join(os.path.expanduser('~'), ".mft"))
os.chmod("consul", 0o744)
@@ -136,9 +135,9 @@ def start_mft():
call(["kill", "-9", pid])
consul_process = Popen(['nohup', './consul', "agent", "-dev"],
- stdout=open('consul.log', 'w'),
- stderr=open('consul.err.log', 'a'),
- preexec_fn=os.setpgrp)
+ stdout=open('consul.log', 'w'),
+ stderr=open('consul.err.log', 'a'),
+ preexec_fn=os.setpgrp)
print("Consul process id: " + str(consul_process.pid))
with open("consul.pid", "w") as consul_pid:
@@ -163,7 +162,7 @@ def stop_mft():
path = os.path.join(os.path.expanduser('~'), ".mft/consul")
if os.path.exists(path):
-current_dir = os.getcwd()
+current_dir = os.getcwd()
try:
os.chdir(os.path.join(os.path.expanduser('~'), ".mft"))
os.chmod("consul", 0o744)
@@ -180,7 +179,6 @@ def stop_mft():
print("MFT Stopped")
-
def update_mft():
stop_mft()
@@ -194,17 +192,16 @@ def update_mft():
os.remove(database)
start_mft()
-
def print_log():
log_file_path = os.path.join(os.path.expanduser('~'), ".mft",
"Standalone-Service-0.01", "logs", "airavata.log")
- log_file = open(log_file_path, "r")
+ log_file = open(log_file_path, "r")
lines = follow_file(log_file)
for line in lines:
print(line)
def follow_file(file):
- # file.seek(0, os.SEEK_END)
+ #file.seek(0, os.SEEK_END)
while True:
line = file.readline()
@@ -213,3 +210,4 @@ def follow_file(file):
continue
yield line
+
diff --git a/python-cli/mft_cli/venv/pyvenv.cfg
b/python-cli/mft_cli/venv/pyvenv.cfg
deleted file mode 100644
index 740014e..000
--- a/python-cli/mft_cli/venv/pyvenv.cfg
+++ /dev/null
@@ -1,3 +0,0 @@
-home = /Library/Frameworks/Python.framework/Versions/3.10/bin
-include-system-site-packages = false
-version = 3.10.11
[airavata-mft] 05/07: format fixes
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
commit a205e4ac9f5892b776759023d99433e86cda33de
Author: jfcsgthmbvnztjkcfcvjqjr
AuthorDate: Fri Jun 23 23:01:07 2023 -0400
format fixes
---
python-cli/mft_cli/airavata_mft_cli/bootstrap.py | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
b/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
index 6bb96ac..d94b09d 100644
--- a/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
+++ b/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
@@ -29,7 +29,6 @@ from sys import platform
import shutil
import time
-
def download_and_unarchive(url, download_path, extract_dir =
os.path.join(os.path.expanduser('~'), ".mft/")):
response = requests.get(url, stream=True)
file_size = int(response.headers['Content-Length'])
@@ -194,12 +193,11 @@ def update_mft():
def print_log():
log_file_path = os.path.join(os.path.expanduser('~'), ".mft",
"Standalone-Service-0.01", "logs", "airavata.log")
- log_file = open(log_file_path, "r")
+ log_file = open(log_file_path,"r")
lines = follow_file(log_file)
for line in lines:
print(line)
-
def follow_file(file):
#file.seek(0, os.SEEK_END)
@@ -211,3 +209,4 @@ def follow_file(file):
yield line
+
[airavata-mft] 07/07: PR changes
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
commit 78b7aa09da64c11e34db28c512d3da123b3ea2bd
Author: jfcsgthmbvnztjkcfcvjqjr
AuthorDate: Sat Jun 24 07:36:26 2023 -0400
PR changes
---
python-cli/mft_cli/airavata_mft_cli/bootstrap.py | 27
1 file changed, 18 insertions(+), 9 deletions(-)
diff --git a/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
b/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
index 0c0fbc8..a2d9085 100644
--- a/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
+++ b/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
@@ -65,7 +65,7 @@ def stop_service(bin_path, daemon_script_name):
os.chdir(current_dir)
-def check_java_version_method(required_version):
+def validate_java_availability(required_version):
"""
Issue 96: https://github.com/apache/airavata-mft/issues/96
@@ -73,10 +73,19 @@ def check_java_version_method(required_version):
-
https://stackoverflow.com/questions/31807882/get-java-version-number-from-python
https://stackoverflow.com/questions/1332598/how-to-determine-whether-java-is-installed-on-a-system-through-python
+
https://stackoverflow.com/questions/74206258/how-are-oracle-jdk-versions-numbered
+
https://docs.python.org/3.8/library/subprocess.html#subprocess.check_output
+
https://stackoverflow.com/questions/2411288/java-versioning-and-terminology-1-6-vs-6-0-openjdk-vs-sun
"""
if shutil.which("java"):
res = check_output(['java', '-version'], stderr=STDOUT).decode('utf-8')
-system_version = ''
+"""
+ res will have the value similar to the following
+ openjdk version "17.0.5" 2022-10-18 LTS
+ OpenJDK Runtime Environment Corretto-17.0.5.8.1 (build 17.0.5+8-LTS)
+ OpenJDK 64-Bit Server VM Corretto-17.0.5.8.1 (build 17.0.5+8-LTS, mixed
mode, sharing)
+"""
+java_version = ''
count = 0
for c in res:
if c == '"' or count == 1:
@@ -85,12 +94,12 @@ def check_java_version_method(required_version):
if count == 0:
count += 1
continue
-system_version += c
+java_version += c
-system_version = int(system_version)
-if system_version < required_version:
- print("Airavata MFT requires Java version 11 or higher")
- print("If you have more than one version of java please set java version
11 or higher to the path")
+java_version = int(java_version)
+if java_version < required_version:
+ print("Airavata MFT requires Java version " + required_version + " or
higher")
+ print("If you have more than one version of java please set java version
"+ required_version +" or higher to the path")
raise typer.Exit()
else:
print("Java is either not installed or path hasn't been set properly")
@@ -103,10 +112,10 @@ def start_mft():
required_java_version = 11
if platform == "linux" or platform == "linux2":
consul_url =
"https://releases.hashicorp.com/consul/1.7.1/consul_1.7.1_linux_amd64.zip;
-check_java_version_method(required_java_version)
+validate_java_availability(required_java_version)
elif platform == "darwin":
consul_url =
"https://releases.hashicorp.com/consul/1.7.1/consul_1.7.1_darwin_amd64.zip;
-check_java_version_method(required_java_version)
+validate_java_availability(required_java_version)
elif platform == "win32":
print("Windows support is not available yet")
raise typer.Exit()
[airavata-mft] 03/07: format fixes
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
commit 85e2e7294491e76785405ef403c8c4b17d7a00d4
Author: jfcsgthmbvnztjkcfcvjqjr
AuthorDate: Fri Jun 23 22:48:42 2023 -0400
format fixes
---
python-cli/mft_cli/airavata_mft_cli/bootstrap.py | 297 +++
1 file changed, 148 insertions(+), 149 deletions(-)
diff --git a/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
b/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
index 03777f8..b9884bd 100644
--- a/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
+++ b/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
@@ -31,186 +31,185 @@ import time
def download_and_unarchive(url, download_path,
extract_dir=os.path.join(os.path.expanduser('~'), ".mft/")):
-response = requests.get(url, stream=True)
-file_size = int(response.headers['Content-Length'])
-with typer.progressbar(length=file_size) as progress:
-with open(download_path, "wb") as handle:
-for data in response.iter_content(chunk_size=8192 * 2):
-progress.update(len(data))
-handle.write(data)
+ response = requests.get(url, stream=True)
+ file_size = int(response.headers['Content-Length'])
+ with typer.progressbar(length=file_size) as progress:
+with open(download_path, "wb") as handle:
+ for data in response.iter_content(chunk_size=8192 * 2):
+progress.update(len(data))
+handle.write(data)
-print("Un archiving ")
-with zipfile.ZipFile(download_path, "r") as zip_ref:
-zip_ref.extractall(extract_dir)
+ print("Un archiving ")
+ with zipfile.ZipFile(download_path, "r") as zip_ref:
+zip_ref.extractall(extract_dir)
-os.remove(download_path)
+ os.remove(download_path)
def restart_service(bin_path, daemon_script_name):
-current_dir = os.getcwd()
-try:
-os.chdir(bin_path)
-os.chmod(daemon_script_name, 0o744)
-rc = call(["./" + daemon_script_name, "stop"])
-rc = call(["./" + daemon_script_name, "start"])
-finally:
-os.chdir(current_dir)
+ current_dir = os.getcwd()
+ try:
+os.chdir(bin_path)
+os.chmod(daemon_script_name, 0o744)
+rc = call(["./" + daemon_script_name, "stop"])
+rc = call(["./" + daemon_script_name, "start"])
+ finally:
+os.chdir(current_dir)
def stop_service(bin_path, daemon_script_name):
-current_dir = os.getcwd()
-try:
-os.chdir(bin_path)
-os.chmod(daemon_script_name, 0o744)
-rc = call(["./" + daemon_script_name, "stop"])
-finally:
-os.chdir(current_dir)
+ current_dir = os.getcwd()
+ try:
+os.chdir(bin_path)
+os.chmod(daemon_script_name, 0o744)
+rc = call(["./" + daemon_script_name, "stop"])
+ finally:
+os.chdir(current_dir)
def check_java_version_method(required_version):
-"""
-Issue 96: https://github.com/apache/airavata-mft/issues/96
-
-References:
--
-
https://stackoverflow.com/questions/31807882/get-java-version-number-from-python
-
https://stackoverflow.com/questions/1332598/how-to-determine-whether-java-is-installed-on-a-system-through-python
-"""
-if shutil.which("java"):
-res = check_output(['java', '-version'], stderr=STDOUT).decode('utf-8')
-system_version = ''
-count = 0
-for c in res:
-if c == '"' or count == 1:
-if (c == '.' or c == '"') and count == 1:
-break
-if count == 0:
-count += 1
-continue
-system_version += c
-
-system_version = int(system_version)
-if system_version < required_version:
-print("Airavata MFT requires Java version 11 or higher")
-print("If you have more than one version of java please set java
version 11 or higher to the path")
-raise typer.Exit()
-else:
-print("Java is either not installed or path hasn't been set properly")
-raise typer.Exit()
+ """
+ Issue 96: https://github.com/apache/airavata-mft/issues/96
+
+ References:
+ -
+
https://stackoverflow.com/questions/31807882/get-java-version-number-from-python
+
https://stackoverflow.com/questions/1332598/how-to-determine-whether-java-is-installed-on-a-system-through-python
+ """
+ if shutil.which("java"):
+res = check_output(['java', '-version'], stderr=STDOUT).decode('utf-8')
+system_version = ''
+count = 0
+for c in res:
+ if c
[airavata-mft] 06/07: format fixes
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
commit e2edeaaa089907dae1c3343e9c706a09cd611264
Author: jfcsgthmbvnztjkcfcvjqjr
AuthorDate: Fri Jun 23 23:03:19 2023 -0400
format fixes
---
python-cli/mft_cli/airavata_mft_cli/bootstrap.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
b/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
index d94b09d..0c0fbc8 100644
--- a/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
+++ b/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
@@ -30,6 +30,7 @@ import shutil
import time
def download_and_unarchive(url, download_path, extract_dir =
os.path.join(os.path.expanduser('~'), ".mft/")):
+
response = requests.get(url, stream=True)
file_size = int(response.headers['Content-Length'])
with typer.progressbar(length=file_size) as progress:
@@ -44,7 +45,6 @@ def download_and_unarchive(url, download_path, extract_dir =
os.path.join(os.pat
os.remove(download_path)
-
def restart_service(bin_path, daemon_script_name):
current_dir = os.getcwd()
try:
[airavata-mft] 01/07: added a comment to test commit and push in Intellij
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
commit 6fa77cfe662d01343568e3d4beb82ef50f76967a
Author: jfcsgthmbvnztjkcfcvjqjr
AuthorDate: Wed Jun 21 13:54:25 2023 -0400
added a comment to test commit and push in Intellij
---
python-cli/mft_cli/airavata_mft_cli/bootstrap.py | 4
1 file changed, 4 insertions(+)
diff --git a/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
b/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
index 9e78be6..da704ba 100644
--- a/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
+++ b/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
@@ -62,6 +62,10 @@ def stop_service(bin_path, daemon_script_name):
finally:
os.chdir(current_dir)
+
+"""
+ Issue 96: https://github.com/apache/airavata-mft/issues/96
+"""
def start_mft():
print("Setting up MFT Services")
[airavata-mft] branch master updated (2e48f51 -> 78b7aa0)
This is an automated email from the ASF dual-hosted git repository. dimuthuupe pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airavata-mft.git from 2e48f51 Build error on mac M1 Fixes #53 new 6fa77cf added a comment to test commit and push in Intellij new ee1c556 Updated code to check for java run time new 85e2e72 format fixes new 08b6682 format fixes new a205e4a format fixes new e2edeaa format fixes new 78b7aa0 PR changes The 7 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: python-cli/mft_cli/airavata_mft_cli/bootstrap.py | 49 +++- 1 file changed, 48 insertions(+), 1 deletion(-)
[airavata] branch airavata-v2-refactoring updated: Supporting serialization of protobuf objects in task parameters
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch airavata-v2-refactoring
in repository https://gitbox.apache.org/repos/asf/airavata.git
The following commit(s) were added to refs/heads/airavata-v2-refactoring by
this push:
new 682948e8c1 Supporting serialization of protobuf objects in task
parameters
682948e8c1 is described below
commit 682948e8c14ab54af33d61a7b561138f38128cf2
Author: DImuthuUpe
AuthorDate: Wed Jun 21 10:39:30 2023 -0400
Supporting serialization of protobuf objects in task parameters
---
modules/airavata-apis/airavata-apis-server/pom.xml | 24 ++
.../apis/scheduling/ExperimentLauncher.java| 7 +++
.../apis/workflow/task/common/TaskUtil.java| 12 +--
.../workflow/task/ec2/CreateEC2InstanceTask.java | 15 +-
pom.xml| 2 +-
5 files changed, 56 insertions(+), 4 deletions(-)
diff --git a/modules/airavata-apis/airavata-apis-server/pom.xml
b/modules/airavata-apis/airavata-apis-server/pom.xml
index 8660d06d2d..eff620301e 100644
--- a/modules/airavata-apis/airavata-apis-server/pom.xml
+++ b/modules/airavata-apis/airavata-apis-server/pom.xml
@@ -15,11 +15,23 @@
org.apache.airavata
mft-secret-service-server
0.01-SNAPSHOT
+
+
+com.google.protobuf
+protobuf-java
+
+
org.apache.airavata
mft-resource-service-server
0.01-SNAPSHOT
+
+
+com.google.protobuf
+protobuf-java
+
+
org.apache.airavata
@@ -103,6 +115,18 @@
org.slf4j
slf4j-log4j12
+
+javax.jms
+jms
+
+
+ com.sun.jdmk
+jmxtools
+
+
+com.sun.jmx
+jmxri
+
diff --git
a/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/scheduling/ExperimentLauncher.java
b/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/scheduling/ExperimentLauncher.java
index b1fd6d5846..1d77d6ab56 100644
---
a/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/scheduling/ExperimentLauncher.java
+++
b/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/scheduling/ExperimentLauncher.java
@@ -1,6 +1,7 @@
package org.apache.airavata.apis.scheduling;
import org.apache.airavata.api.execution.ExperimentLaunchRequest;
+import org.apache.airavata.api.execution.stubs.EC2Backend;
import org.apache.airavata.api.execution.stubs.Experiment;
import org.apache.airavata.apis.service.ExecutionService;
import org.apache.airavata.apis.workflow.task.common.BaseTask;
@@ -54,8 +55,14 @@ public class ExperimentLauncher {
taskMap.put(dataMovementTask.getTaskId(), dataMovementTask);
+EC2Backend ec2Backend = EC2Backend.newBuilder()
+.setAwsCredentialId("SomeCred")
+.setFlavor("m2")
+.setRegion("us-west").build();
+
CreateEC2InstanceTask ec2InstanceTask = new CreateEC2InstanceTask();
ec2InstanceTask.setTaskId(UUID.randomUUID().toString());
+ec2InstanceTask.setEc2Backend(ec2Backend);
taskMap.put(ec2InstanceTask.getTaskId(), ec2InstanceTask);
dataMovementTask.addOutPort(new
OutPort().setNextTaskId(ec2InstanceTask.getTaskId()));
diff --git
a/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/workflow/task/common/TaskUtil.java
b/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/workflow/task/common/TaskUtil.java
index 7aa23b7c5e..654c0b6156 100644
---
a/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/workflow/task/common/TaskUtil.java
+++
b/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/workflow/task/common/TaskUtil.java
@@ -1,5 +1,7 @@
package org.apache.airavata.apis.workflow.task.common;
+import com.google.protobuf.AbstractMessage;
+import com.google.protobuf.GeneratedMessageV3;
import org.apache.airavata.apis.workflow.task.common.annotation.TaskParam;
import org.apache.commons.beanutils.PropertyUtils;
import org.slf4j.Logger;
@@ -42,7 +44,11 @@ public class TaskUtil {
Class[] methodParamType =
writeMethod.getParameterTypes();
Class writeParameterType = methodParamType[0];
-if (writeParameterType
[airavata] branch airavata-v2-refactoring updated: Initial workflow framework
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch airavata-v2-refactoring
in repository https://gitbox.apache.org/repos/asf/airavata.git
The following commit(s) were added to refs/heads/airavata-v2-refactoring by
this push:
new fb3c2053ff Initial workflow framework
fb3c2053ff is described below
commit fb3c2053ffb1d666ca7ac0905b0e604eca8ae85e
Author: DImuthuUpe
AuthorDate: Wed Jun 21 04:39:33 2023 -0400
Initial workflow framework
---
modules/airavata-apis/airavata-apis-server/pom.xml | 11 +
.../apache/airavata/apis/config/ConfigBeans.java | 7 +
.../apis/db/entity/backend/EC2BackendEntity.java | 10 +
.../apis/scheduling/ExperimentLauncher.java| 189 +
.../airavata/apis/service/ExecutionService.java| 4 +
.../apis/service/impl/ExecutionServiceImpl.java| 12 ++
.../airavata/apis/workflow/WorkflowExecutor.java | 225 +
.../apis/workflow/task/common/BaseTask.java| 109 ++
.../apis/workflow/task/common/OutPort.java | 14 ++
.../apis/workflow/task/common/TaskParamType.java | 6 +
.../apis/workflow/task/common/TaskUtil.java| 91 +
.../workflow/task/common/annotation/TaskDef.java | 12 ++
.../task/common/annotation/TaskOutPort.java| 12 ++
.../workflow/task/common/annotation/TaskParam.java | 14 ++
.../apis/workflow/task/data/DataMovementTask.java | 25 +++
.../task/docker/DestroyDockerContainerTask.java| 4 +
.../task/docker/StartDockerContainerTask.java | 4 +
.../workflow/task/ec2/CreateEC2InstanceTask.java | 25 +++
.../workflow/task/ec2/DestroyEC2InstanceTask.java | 4 +
.../workflow/task/runners/RunLocalCommandTask.java | 4 +
.../task/runners/RunServerCommandTask.java | 4 +
.../src/main/resources/api.properties | 16 ++
.../src/main/resources/log4j2.xml | 2 +-
.../src/main/resources/workflow.properties | 20 ++
.../src/main/proto/execution/experiment_stub.proto | 1 +
pom.xml| 116 ---
26 files changed, 824 insertions(+), 117 deletions(-)
diff --git a/modules/airavata-apis/airavata-apis-server/pom.xml
b/modules/airavata-apis/airavata-apis-server/pom.xml
index d4642ec899..8660d06d2d 100644
--- a/modules/airavata-apis/airavata-apis-server/pom.xml
+++ b/modules/airavata-apis/airavata-apis-server/pom.xml
@@ -94,6 +94,17 @@
dozer-proto3
6.5.2
+
+org.apache.helix
+helix-core
+0.9.9
+
+
+org.slf4j
+slf4j-log4j12
+
+
+
18
diff --git
a/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/config/ConfigBeans.java
b/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/config/ConfigBeans.java
index c421330640..31f0c436b3 100644
---
a/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/config/ConfigBeans.java
+++
b/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/config/ConfigBeans.java
@@ -1,5 +1,7 @@
package org.apache.airavata.apis.config;
+import org.apache.airavata.api.execution.ExperimentService;
+import org.apache.airavata.apis.scheduling.ExperimentLauncher;
import org.apache.airavata.apis.scheduling.MetaScheduler;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@@ -10,4 +12,9 @@ public class ConfigBeans {
public MetaScheduler metaScheduler() {
return new MetaScheduler();
}
+
+@Bean
+ExperimentLauncher experimentLauncher() {
+return new ExperimentLauncher();
+}
}
diff --git
a/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/db/entity/backend/EC2BackendEntity.java
b/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/db/entity/backend/EC2BackendEntity.java
index 5fb599e8f4..cfd8d8b4d2 100644
---
a/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/db/entity/backend/EC2BackendEntity.java
+++
b/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/db/entity/backend/EC2BackendEntity.java
@@ -11,6 +11,9 @@ public class EC2BackendEntity extends ComputeBackendEntity {
@Column
String region;
+@Column
+String imageId;
+
@Column
String awsCredentialId;
@@ -38,4 +41,11 @@ public class EC2BackendEntity extends ComputeBackendEntity {
this.awsCredentialId = awsCredentialId;
}
+public String getImageId() {
+return imageId;
+}
+
+public void setImageId(String imageId) {
+this.imageId = imageId;
+}
}
diff --git
a/modules/airavata-apis/airavata-apis-server/src/main
[airavata] branch v2 created (now 0b1880f8b7)
This is an automated email from the ASF dual-hosted git repository. dimuthuupe pushed a change to branch v2 in repository https://gitbox.apache.org/repos/asf/airavata.git at 0b1880f8b7 Merge pull request #428 from isururanawaka/metaschedular No new revisions were added by this update.
[airavata-mft] branch master updated (3534959 -> 2e48f51)
This is an automated email from the ASF dual-hosted git repository. dimuthuupe pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airavata-mft.git from 3534959 #94 Utility exception handler add 2e48f51 Build error on mac M1 Fixes #53 No new revisions were added by this update. Summary of changes: agent/stub/pom.xml | 6 +++--- api/stub/pom.xml | 6 +++--- common/mft-common-proto/pom.xml| 6 +++--- pom.xml| 3 ++- services/resource-service/stub/pom.xml | 6 +++--- services/secret-service/stub/pom.xml | 6 +++--- 6 files changed, 17 insertions(+), 16 deletions(-)
[airavata] branch airavata-v2-refactoring updated: Framework to connect launch request with meta scheduler
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch airavata-v2-refactoring
in repository https://gitbox.apache.org/repos/asf/airavata.git
The following commit(s) were added to refs/heads/airavata-v2-refactoring by
this push:
new 6f506542e2 Framework to connect launch request with meta scheduler
6f506542e2 is described below
commit 6f506542e2bc68757ca385e1119e45a17fb8e450
Author: DImuthuUpe
AuthorDate: Mon May 15 11:40:36 2023 -0400
Framework to connect launch request with meta scheduler
---
.../apache/airavata/apis/config/ConfigBeans.java | 13 ++
.../airavata/apis/db/entity/ExperimentEntity.java | 8
.../apis/db/entity/RunConfigurationEntity.java | 48 ++
.../airavata/apis/handlers/ExecutionHandler.java | 34 ++-
.../airavata/apis/scheduling/MetaScheduler.java| 19 +
5 files changed, 120 insertions(+), 2 deletions(-)
diff --git
a/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/config/ConfigBeans.java
b/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/config/ConfigBeans.java
new file mode 100644
index 00..c421330640
--- /dev/null
+++
b/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/config/ConfigBeans.java
@@ -0,0 +1,13 @@
+package org.apache.airavata.apis.config;
+
+import org.apache.airavata.apis.scheduling.MetaScheduler;
+import org.springframework.context.annotation.Bean;
+import org.springframework.context.annotation.Configuration;
+
+@Configuration
+public class ConfigBeans {
+@Bean
+public MetaScheduler metaScheduler() {
+return new MetaScheduler();
+}
+}
diff --git
a/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/db/entity/ExperimentEntity.java
b/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/db/entity/ExperimentEntity.java
index db402fb4de..d1073c0017 100644
---
a/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/db/entity/ExperimentEntity.java
+++
b/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/db/entity/ExperimentEntity.java
@@ -80,4 +80,12 @@ public class ExperimentEntity {
public void setDescription(String description) {
this.description = description;
}
+
+public List getRunConfigs() {
+return runConfigs;
+}
+
+public void setRunConfigs(List runConfigs) {
+this.runConfigs = runConfigs;
+}
}
diff --git
a/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/db/entity/RunConfigurationEntity.java
b/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/db/entity/RunConfigurationEntity.java
index 58deef412d..b4546b8066 100644
---
a/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/db/entity/RunConfigurationEntity.java
+++
b/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/db/entity/RunConfigurationEntity.java
@@ -24,4 +24,52 @@ public class RunConfigurationEntity {
LocalBackendEntity local;
ApplicationRunInfoEntity appRunInfo;
List dataMovementConfigs;
+
+public String getRunConfigId() {
+return runConfigId;
+}
+
+public void setRunConfigId(String runConfigId) {
+this.runConfigId = runConfigId;
+}
+
+public ServerBackendEntity getServer() {
+return server;
+}
+
+public void setServer(ServerBackendEntity server) {
+this.server = server;
+}
+
+public EC2BackendEntity getEc2() {
+return ec2;
+}
+
+public void setEc2(EC2BackendEntity ec2) {
+this.ec2 = ec2;
+}
+
+public LocalBackendEntity getLocal() {
+return local;
+}
+
+public void setLocal(LocalBackendEntity local) {
+this.local = local;
+}
+
+public ApplicationRunInfoEntity getAppRunInfo() {
+return appRunInfo;
+}
+
+public void setAppRunInfo(ApplicationRunInfoEntity appRunInfo) {
+this.appRunInfo = appRunInfo;
+}
+
+public List getDataMovementConfigs() {
+return dataMovementConfigs;
+}
+
+public void setDataMovementConfigs(List
dataMovementConfigs) {
+this.dataMovementConfigs = dataMovementConfigs;
+}
}
diff --git
a/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/handlers/ExecutionHandler.java
b/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/handlers/ExecutionHandler.java
index c267ca6d6a..eb4415d721 100644
---
a/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/handlers/ExecutionHandler.java
+++
b/modules/airavata-apis/airavata-apis-server/src/main/java/org/apache/airavata/apis/handlers/ExecutionHandler.java
@@ -1,18 +1,48 @@
package
[airavata] branch airavata-v2-refactoring updated: Cli example update for new API
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch airavata-v2-refactoring
in repository https://gitbox.apache.org/repos/asf/airavata.git
The following commit(s) were added to refs/heads/airavata-v2-refactoring by
this push:
new a709defc6c Cli example update for new API
a709defc6c is described below
commit a709defc6c1740e6a8b36dd95433b5d44853a0a0
Author: DImuthuUpe
AuthorDate: Mon May 15 11:14:49 2023 -0400
Cli example update for new API
---
.../org/apache/airavata/apis/client/AiravataAPIClient.java | 10 ++
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git
a/modules/airavata-apis/airavata-apis-client/src/main/java/org/apache/airavata/apis/client/AiravataAPIClient.java
b/modules/airavata-apis/airavata-apis-client/src/main/java/org/apache/airavata/apis/client/AiravataAPIClient.java
index 81d7f21f50..adf30f9f42 100644
---
a/modules/airavata-apis/airavata-apis-client/src/main/java/org/apache/airavata/apis/client/AiravataAPIClient.java
+++
b/modules/airavata-apis/airavata-apis-client/src/main/java/org/apache/airavata/apis/client/AiravataAPIClient.java
@@ -31,7 +31,6 @@ public class AiravataAPIClient {
SSHInterface.newBuilder().setHostName("localhost").build())
.setDataInterface(SCPInterface.newBuilder().build()).setHostName("localhost").setPort(22).build();
-RunConfiguration runConfig =
RunConfiguration.newBuilder().setAppRunInfo(appRunInfo).setServer(serverBackend).build();
DataMovementConfiguration dataMovement =
DataMovementConfiguration.newBuilder()
.addInMovements(
@@ -43,12 +42,15 @@ public class AiravataAPIClient {
.setStorageCredentialId("Crede").build())
.build()).build();
+RunConfiguration runConfig = RunConfiguration.newBuilder()
+.setAppRunInfo(appRunInfo)
+.setServer(serverBackend)
+.addDataMovementConfigs(dataMovement).build();
+
Experiment experiment = Experiment.newBuilder()
.setCreationTime(System.currentTimeMillis())
.setDescription("Sample Exp")
-.addRunConfig(runConfig)
-.addDataMovementConfig(dataMovement).build();
-
+.addRunConfigs(runConfig).build();
ExperimentRegisterRequest experimentRegisterRequest =
ExperimentRegisterRequest.newBuilder()
.setExperiment(experiment).build();
[airavata] branch airavata-v2-refactoring created (now 5f9a4be1a4)
This is an automated email from the ASF dual-hosted git repository. dimuthuupe pushed a change to branch airavata-v2-refactoring in repository https://gitbox.apache.org/repos/asf/airavata.git at 5f9a4be1a4 WIP New API and Code restructuring This branch includes the following new commits: new 5f9a4be1a4 WIP New API and Code restructuring The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[airavata-mft] 01/02: #94 Rich client messages
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
commit 22cedf96bfd32c49ea508124101ea228ea238560
Author: impiyush83
AuthorDate: Sat Apr 8 05:57:24 2023 -0400
#94 Rich client messages
---
python-cli/mft_cli/airavata_mft_cli/base.py| 14 +++-
.../mft_cli/airavata_mft_cli/storage/__init__.py | 79 --
2 files changed, 56 insertions(+), 37 deletions(-)
diff --git a/python-cli/mft_cli/airavata_mft_cli/base.py
b/python-cli/mft_cli/airavata_mft_cli/base.py
index 4e9a6a8..92d28bd 100644
--- a/python-cli/mft_cli/airavata_mft_cli/base.py
+++ b/python-cli/mft_cli/airavata_mft_cli/base.py
@@ -19,16 +19,26 @@
import typer
import airavata_mft_cli.operations as operations
import airavata_mft_cli.bootstrap as bootstrap
+import grpc
+from rich import print
app = typer.Typer()
@app.command("ls")
def list(storage_path):
- operations.list(storage_path)
+ try:
+operations.list(storage_path)
+ except grpc.RpcError as rpc_error:
+if rpc_error.code() == grpc.StatusCode.UNAVAILABLE:
+ print(f'Could not list resources for your storage path {storage_path}
due to MFT server unavailable')
@app.command("cp")
def copy(source, destination):
- operations.copy(source, destination)
+ try:
+operations.copy(source, destination)
+ except grpc.RpcError as rpc_error:
+if rpc_error.code() == grpc.StatusCode.UNAVAILABLE:
+ print(f'Could not copy resources from source = {source} to destination =
{destination} due to MFT server unavailable')
@app.command("init")
def init_mft():
diff --git a/python-cli/mft_cli/airavata_mft_cli/storage/__init__.py
b/python-cli/mft_cli/airavata_mft_cli/storage/__init__.py
index d4f00e8..cb6b8b6 100644
--- a/python-cli/mft_cli/airavata_mft_cli/storage/__init__.py
+++ b/python-cli/mft_cli/airavata_mft_cli/storage/__init__.py
@@ -28,7 +28,9 @@ from airavata_mft_sdk import mft_client
from airavata_mft_sdk.common import StorageCommon_pb2
from rich.console import Console
from rich.table import Table
+from rich import print
import sys
+import grpc
sys.path.append('../airavata_mft_cli')
from airavata_mft_cli import config as configcli
@@ -36,46 +38,53 @@ app = typer.Typer()
@app.command("add")
def add_storage():
-title = "Select storage type: "
-options = ["S3", "Google Cloud Storage (GCS)", "Azure Storage", "Openstack
SWIFT", "SCP", "FTP", "Box", "DropBox", "OData", "Agent" ]
-option, index = pick(options, title, indicator="=>")
-if option == "S3":
-s3.handle_add_storage()
-elif option == "Azure Storage":
-azure.handle_add_storage()
-elif option == "Google Cloud Storage (GCS)":
-gcs.handle_add_storage()
-elif option == "Agent":
-local.handle_add_storage()
-elif option == "Openstack SWIFT":
-swift.handle_add_storage()
-elif option == "SCP":
-scp.handle_add_storage()
-
+try:
+title = "Select storage type: "
+options = ["S3", "Google Cloud Storage (GCS)", "Azure Storage",
"Openstack SWIFT", "SCP", "FTP", "Box", "DropBox", "OData", "Agent" ]
+option, index = pick(options, title, indicator="=>")
+if option == "S3":
+s3.handle_add_storage()
+elif option == "Azure Storage":
+azure.handle_add_storage()
+elif option == "Google Cloud Storage (GCS)":
+gcs.handle_add_storage()
+elif option == "Agent":
+local.handle_add_storage()
+elif option == "Openstack SWIFT":
+swift.handle_add_storage()
+elif option == "SCP":
+scp.handle_add_storage()
+except grpc.RpcError as rpc_error:
+if rpc_error.code() == grpc.StatusCode.UNAVAILABLE:
+print(f'Could not add storage in {option} due to MFT server grpc
unavailable error')
@app.command("list")
def list_storage():
-client = mft_client.MFTClient(transfer_api_port =
configcli.transfer_api_port,
- transfer_api_secured =
configcli.transfer_api_secured,
- resource_service_host =
configcli.resource_service_host,
- resource_service_port =
configcli.resource_service_port,
- resource_service_secured =
configcli.resource_service_secured,
- secret_service_host =
configcli.secret_service_host,
- secre
[airavata-mft] 02/02: #94 Utility exception handler
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
commit 3534959886e61b4c71518f0feac1630ed9b8621c
Author: impiyush83
AuthorDate: Sat Apr 15 07:02:58 2023 -0400
#94 Utility exception handler
---
python-cli/mft_cli/airavata_mft_cli/base.py| 13 +--
.../mft_cli/airavata_mft_cli/storage/__init__.py | 12 +-
python-cli/mft_cli/airavata_mft_cli/util.py| 26 ++
3 files changed, 36 insertions(+), 15 deletions(-)
diff --git a/python-cli/mft_cli/airavata_mft_cli/base.py
b/python-cli/mft_cli/airavata_mft_cli/base.py
index 92d28bd..1d1e2d3 100644
--- a/python-cli/mft_cli/airavata_mft_cli/base.py
+++ b/python-cli/mft_cli/airavata_mft_cli/base.py
@@ -19,8 +19,7 @@
import typer
import airavata_mft_cli.operations as operations
import airavata_mft_cli.bootstrap as bootstrap
-import grpc
-from rich import print
+from airavata_mft_cli.util import exception_handler
app = typer.Typer()
@@ -28,17 +27,15 @@ app = typer.Typer()
def list(storage_path):
try:
operations.list(storage_path)
- except grpc.RpcError as rpc_error:
-if rpc_error.code() == grpc.StatusCode.UNAVAILABLE:
- print(f'Could not list resources for your storage path {storage_path}
due to MFT server unavailable')
+ except Exception as e:
+exception_handler(e)
@app.command("cp")
def copy(source, destination):
try:
operations.copy(source, destination)
- except grpc.RpcError as rpc_error:
-if rpc_error.code() == grpc.StatusCode.UNAVAILABLE:
- print(f'Could not copy resources from source = {source} to destination =
{destination} due to MFT server unavailable')
+ except Exception as e:
+exception_handler(e)
@app.command("init")
def init_mft():
diff --git a/python-cli/mft_cli/airavata_mft_cli/storage/__init__.py
b/python-cli/mft_cli/airavata_mft_cli/storage/__init__.py
index cb6b8b6..d461682 100644
--- a/python-cli/mft_cli/airavata_mft_cli/storage/__init__.py
+++ b/python-cli/mft_cli/airavata_mft_cli/storage/__init__.py
@@ -30,9 +30,9 @@ from rich.console import Console
from rich.table import Table
from rich import print
import sys
-import grpc
sys.path.append('../airavata_mft_cli')
from airavata_mft_cli import config as configcli
+from airavata_mft_cli.util import exception_handler
app = typer.Typer()
@@ -54,9 +54,8 @@ def add_storage():
swift.handle_add_storage()
elif option == "SCP":
scp.handle_add_storage()
-except grpc.RpcError as rpc_error:
-if rpc_error.code() == grpc.StatusCode.UNAVAILABLE:
-print(f'Could not add storage in {option} due to MFT server grpc
unavailable error')
+except Exception as e:
+exception_handler(e)
@app.command("list")
def list_storage():
@@ -85,6 +84,5 @@ def list_storage():
storage.storageId)
console.print(table)
-except grpc.RpcError as rpc_error:
-if rpc_error.code() == grpc.StatusCode.UNAVAILABLE:
-print('Could not fetch storage list due to MFT server grpc
unavailable error')
+except Exception as e:
+exception_handler(e)
\ No newline at end of file
diff --git a/python-cli/mft_cli/airavata_mft_cli/util.py
b/python-cli/mft_cli/airavata_mft_cli/util.py
new file mode 100644
index 000..186a6b5
--- /dev/null
+++ b/python-cli/mft_cli/airavata_mft_cli/util.py
@@ -0,0 +1,26 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+from rich import print
+import grpc
+
+def exception_handler(e):
+if isinstance(e, grpc.RpcError):
+if e.code() == grpc.StatusCode.UNAVAILABLE:
+print(f"MFT server is unavailable")
[airavata-mft] branch master updated (b3e069e -> 3534959)
This is an automated email from the ASF dual-hosted git repository. dimuthuupe pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airavata-mft.git from b3e069e Modifications for ConnectorConfig new 22cedf9 #94 Rich client messages new 3534959 #94 Utility exception handler The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: python-cli/mft_cli/airavata_mft_cli/base.py| 11 +++- .../mft_cli/airavata_mft_cli/storage/__init__.py | 77 -- .../mft_cli/airavata_mft_cli/util.py | 11 +++- 3 files changed, 59 insertions(+), 40 deletions(-) copy python-sdk/pyproject.toml => python-cli/mft_cli/airavata_mft_cli/util.py (79%)
[airavata-mft] branch master updated (cedd283 -> b3e069e)
This is an automated email from the ASF dual-hosted git repository. dimuthuupe pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airavata-mft.git from cedd283 Update issue templates add f59d178 Transport Config Addon add f4ce251 Transport Config Addon add df82298 Transport Config Util add 5dfd7d9 Modifications for TransportProperties add b3e069e Modifications for ConnectorConfig No new revisions were added by this update. Summary of changes: .../airavata/mft/agent/TransferOrchestrator.java | 5 ++ .../apache/airavata/mft/agent/TransportConfig.java | 31 +-- .../main/resources/agent-application.properties| 8 ++- .../airavata/mft/core/api/ConnectorConfig.java | 62 ++ standalone-service/pom.xml | 5 -- transport/local-transport/pom.xml | 5 ++ .../local/LocalIncomingChunkedConnector.java | 59 ++-- 7 files changed, 127 insertions(+), 48 deletions(-) copy controller/src/main/java/org/apache/airavata/mft/controller/ControllerException.java => agent/service/src/main/java/org/apache/airavata/mft/agent/TransportConfig.java (57%)
[airavata-mft] branch master updated: Update issue templates
This is an automated email from the ASF dual-hosted git repository. dimuthuupe pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/airavata-mft.git The following commit(s) were added to refs/heads/master by this push: new cedd283 Update issue templates cedd283 is described below commit cedd283d6c96e4bb9beeb8f0016005cb911f976e Author: Dimuthu Wannipurage AuthorDate: Thu Apr 6 13:42:06 2023 -0400 Update issue templates --- .github/ISSUE_TEMPLATE/bug_report.md | 63 ++ .github/ISSUE_TEMPLATE/feature_request.md | 47 +- .github/ISSUE_TEMPLATE/usability-issues.md | 10 + 3 files changed, 32 insertions(+), 88 deletions(-) diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md index ee367b6..e67df16 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.md +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -1,60 +1,23 @@ - - --- -name: Report a bug +name: Bug report about: Create a report to help us improve -labels: bug +title: '' +labels: '' assignees: '' --- - - -[NOTE]: # ( ^^ Provide a general summary of the issue in the title above. ^^ ) - -## Description - -[NOTE]: # ( Describe the problem you're encountering. ) - -## Steps to Reproduce - -[NOTE]: # ( Include details description or commands to reproduce. ) - -## Expected Behaviour - -[NOTE]: # ( Tell us what you did and what you expected to happen and what you instead saw. ) - -## Your Environment +**Describe the bug** +A clear and concise description of what the bug is. -[TIP]: # ( Include as many relevant details about your environment as possible. ) -[TIP]: # ( Mention Custos Branch or release version, runtime and compiler version ) +**To Reproduce** +Steps to reproduce the behavior: -* Custos branch or release version used: -* Operating system and version: +**Expected behavior** +A clear and concise description of what you expected to happen. -## Additional Context +**Screenshots** +If applicable, add screenshots to help explain your problem. -[TIP]: # ( full error message, exception listing, stack trace, logs or other information which can assist in diagnosing the bug. ) \ No newline at end of file +**Additional context** +Add any other context about the problem here. diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md index edc1366..bbcbbe7 100644 --- a/.github/ISSUE_TEMPLATE/feature_request.md +++ b/.github/ISSUE_TEMPLATE/feature_request.md @@ -1,49 +1,20 @@ - - --- -name: Request a feature or enhancement +name: Feature request about: Suggest an idea for this project -labels: enhancement +title: '' +labels: '' assignees: '' --- - - -[NOTE]: # ( ^^ Provide a general summary of the request in the title above. ^^ ) - -## Is your feature request related to a problem? Please describe. -A clear and concise description of what the problem is. +**Is your feature request related to a problem? Please describe.** +A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -## Describe the solution you'd like +**Describe the solution you'd like** A clear and concise description of what you want to happen. -## Describe alternatives you've considered +**Describe alternatives you've considered** A clear and concise description of any alternative solutions or features you've considered. -## Additional context -[TIP]: # ( Why does this feature matter to you? What unique use cases do you have? ) -Add any other context about the feature request here. +**Additional context** +Add any other context or screenshots about the feature request here. diff --git a/.github/ISSUE_TEMPLATE/usability-issues.md b/.github/ISSUE_TEMPLATE/usability-issues.md new file mode 100644 index 000..16aabc6 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/usability-issues.md @@ -0,0 +1,10 @@ +--- +name: Usability issues +about: Issues related to installation, build and usability of the project +title: '' +labels: '' +assignees: '' + +--- + +
[airavata-mft] branch master updated: Update cli medata
This is an automated email from the ASF dual-hosted git repository. dimuthuupe pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/airavata-mft.git The following commit(s) were added to refs/heads/master by this push: new e28bd08 Update cli medata e28bd08 is described below commit e28bd083efc4f882ca8013274da5ee6dd5716e76 Author: DImuthuUpe AuthorDate: Thu Apr 6 13:22:07 2023 -0400 Update cli medata --- python-cli/README.md | 11 +- python-cli/mft_cli/README.md | 77 ++- python-cli/mft_cli/pyproject.toml | 14 +-- 3 files changed, 89 insertions(+), 13 deletions(-) diff --git a/python-cli/README.md b/python-cli/README.md index 3a2639c..539309d 100644 --- a/python-cli/README.md +++ b/python-cli/README.md @@ -17,7 +17,7 @@ specific language governing permissions and limitations under the License. --> -### MFT-Cli Build Instructions +### MFT-Cli Local Build Instructions Install Poetry ``` @@ -29,17 +29,10 @@ pip install pick Load Poetry shell ``` -cd airavata_mft_cli +cd mft_cli poetry shell ``` -Install dependencies -``` -pip install grpcio==1.46.3 -pip install grpcio-tools==1.46.3 -pip install airavata_mft_sdk==0.0.1-alpha26 -``` - Build the binary ``` poetry install diff --git a/python-cli/mft_cli/README.md b/python-cli/mft_cli/README.md index af79dd7..d35cf56 100644 --- a/python-cli/mft_cli/README.md +++ b/python-cli/mft_cli/README.md @@ -17,4 +17,79 @@ specific language governing permissions and limitations under the License. --> -# Airavata MFT Command Line Client \ No newline at end of file +# Airavata Managed File Transfers (MFT) + +Apache Airavata MFT is a high-performance, multi-protocol data transfer engine to orchestrate data movement and operations across most cloud and On-premises storages. MFT aims to abstract the complexity of heterogenous storages by providing a unified and simple interface for users to seamlessly access and move data across any storage endpoint. To accomplish this goal, MFT provides simple but highly-performing tools to access most cloud and on-premise storages as seamlessly as they access [...] + +Apache Airavata MFT bundles easily deployable agents that auto determine optimum network path with additional multi-channel, parallel data paths to optimize the transfer performance to gain the maximum throughput between storage endpoints. MFT utilizes parallel Agents to transfer data between endpoints to gain the advantage of multiple network links. + +# Try Airavata MFT +MFT requires Java 11+ and python3.10+ to install Airavata MFT in your environment. MFT currently supports Linux and MacOS operating systems. Contributions to support Windows are welcome!!. + +### Download and Install + +Following commands will download Airavata MFT into your machine and start the MFT service. +``` +pip3 install airavata-mft-cli +mft init +``` + +> If the installer failed for M1 and M2 Macs complaining about grpcio installation. Follow the solution mentioned in [here](https://github.com/apache/airavata-mft/issues/71). You might have to uninstall already installed grpcio and grpcio-tools distributions first. +> For other common installation issues, please refer to the [troubleshooting section](https://github.com/apache/airavata-mft#common-issues). + +To stop MFT after using + +``` +mft stop +``` + + +### Registering Storages + +First you need to register your storage endpoints into MFT in order to access them. Registering storage is an interactive process and you can easily register those without prior knowledge + +``` +mft storage add +``` + +This will ask the type of storage you need and credentials to access those storages. To list already added storages, you can run + +``` +mft storage list +``` +### Accessing Data in Storages + +In Airavata MFT, we provide a unified interface to access the data in any storage. Users can access data in storages just as they access data in their computers. MFT converts user queries into storage specific data representations (POSIX, Block, Objects, ..) internally + +``` +mft ls +mft ls / +``` + +### Moving Data between Storages + +Copying data between storages are simple as copying data between directories of local machine for users. MFT takes care of network path optimizations, parallel data path selections and selections or creations of suitable transfer agents. + + ``` + mft cp / / + ``` +MFT is capable of auto detecting directory copying and file copying based on the path given. + +### Troubleshooting and Issue Reporting + +This is our very first attempt release Airavata MFT for community usage and there might be lots of corner cases that we have not noticed. All the logs of MFT service are available in ```~/.mft/Standalone-Service-0.01/logs/airavata.log```. If you see any error while using MFT, please report that in our Github issue page and we will respond as soon as possibl
[airavata-mft] branch master updated: Update and log commands for mft-cli
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
The following commit(s) were added to refs/heads/master by this push:
new f4e2bd9 Update and log commands for mft-cli
f4e2bd9 is described below
commit f4e2bd953b343cce2f2e1a7c5b4af2eb3e501851
Author: DImuthuUpe
AuthorDate: Tue Mar 21 09:25:18 2023 -0400
Update and log commands for mft-cli
---
python-cli/mft_cli/airavata_mft_cli/base.py| 10 ++-
python-cli/mft_cli/airavata_mft_cli/bootstrap.py | 33 +-
python-cli/mft_cli/pyproject.toml | 2 +-
.../assembly/standalone-service-bin-assembly.xml | 7 +
.../src/main/resources/application.properties | 3 +-
.../distribution/conf/application.properties | 5 ++--
6 files changed, 54 insertions(+), 6 deletions(-)
diff --git a/python-cli/mft_cli/airavata_mft_cli/base.py
b/python-cli/mft_cli/airavata_mft_cli/base.py
index 9cdf4d5..4e9a6a8 100644
--- a/python-cli/mft_cli/airavata_mft_cli/base.py
+++ b/python-cli/mft_cli/airavata_mft_cli/base.py
@@ -36,4 +36,12 @@ def init_mft():
@app.command("stop")
def init_mft():
- bootstrap.stop_mft()
\ No newline at end of file
+ bootstrap.stop_mft()
+
+@app.command("update")
+def init_mft():
+ bootstrap.update_mft()
+
+@app.command("log")
+def init_mft():
+ bootstrap.print_log()
\ No newline at end of file
diff --git a/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
b/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
index e6e9b9e..9e78be6 100644
--- a/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
+++ b/python-cli/mft_cli/airavata_mft_cli/bootstrap.py
@@ -24,6 +24,8 @@ from subprocess import call
from subprocess import Popen
from pathlib import Path
from sys import platform
+import shutil
+import time
def download_and_unarchive(url, download_path, extract_dir =
os.path.join(os.path.expanduser('~'), ".mft/")):
@@ -117,7 +119,7 @@ def start_mft():
def stop_mft():
- print("Stopping up MFT Services")
+ print("Stopping MFT Services")
path = os.path.join(os.path.expanduser('~'), ".mft/consul")
if os.path.exists(path):
@@ -138,6 +140,35 @@ def stop_mft():
print("MFT Stopped")
+def update_mft():
+ stop_mft()
+ mft_dir = os.path.join(os.path.expanduser('~'), ".mft")
+ if os.path.exists(mft_dir):
+print("Removing .mft directory")
+shutil.rmtree(mft_dir)
+
+ database = os.path.join(os.path.expanduser('~'), "mft_db.mv.db")
+ if os.path.exists(database):
+os.remove(database)
+ start_mft()
+
+def print_log():
+ log_file_path = os.path.join(os.path.expanduser('~'), ".mft",
"Standalone-Service-0.01", "logs", "airavata.log")
+ log_file = open(log_file_path,"r")
+ lines = follow_file(log_file)
+ for line in lines:
+print(line)
+
+def follow_file(file):
+ #file.seek(0, os.SEEK_END)
+
+ while True:
+line = file.readline()
+if not line:
+ time.sleep(0.1)
+ continue
+
+yield line
diff --git a/python-cli/mft_cli/pyproject.toml
b/python-cli/mft_cli/pyproject.toml
index 06d4809..a6d8acd 100644
--- a/python-cli/mft_cli/pyproject.toml
+++ b/python-cli/mft_cli/pyproject.toml
@@ -18,7 +18,7 @@
[tool.poetry]
name = "airavata-mft-cli"
-version = "0.1.8"
+version = "0.1.9"
description = "Command Line Client for Airavata MFT data transfer software"
authors = ["Apache Airavata "]
readme = "README.md"
diff --git
a/standalone-service/src/main/assembly/standalone-service-bin-assembly.xml
b/standalone-service/src/main/assembly/standalone-service-bin-assembly.xml
index ee64c7a..731ef4f 100644
--- a/standalone-service/src/main/assembly/standalone-service-bin-assembly.xml
+++ b/standalone-service/src/main/assembly/standalone-service-bin-assembly.xml
@@ -52,6 +52,13 @@
*.sh
+
+../plugins
+plugins
+
+*.zip
+
+
src/main/resources/distribution/conf
conf
diff --git a/standalone-service/src/main/resources/application.properties
b/standalone-service/src/main/resources/application.properties
index 629d6f3..8d208ad 100644
--- a/standalone-service/src/main/resources/application.properties
+++ b/standalone-service/src/main/resources/application.properties
@@ -25,4 +25,5 @@ resource.service.host=localhost
resource.service.port=7003
secret.service.host=localhost
secret.service.port=7003
-agent.id=local-agent
\ No newline at end of file
+agent.id=local-agent
+agent.transport.directory=plugins
\ No newline at end of file
diff --git
a/standalone-service/src/main/resources/distribution/conf/application.properties
b/standalon
[airavata-mft] branch master updated (a61fa4a -> 3c89f17)
This is an automated email from the ASF dual-hosted git repository. dimuthuupe pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airavata-mft.git from a61fa4a Upgrading Jcraft to an actively maintained fork add f244b81 Minor logging improvement add 3c89f17 Overriding the http client for streaming s3 connector No new revisions were added by this update. Summary of changes: .../apache/airavata/mft/agent/transport/TransportClassLoader.java | 6 +- controller/pom.xml | 2 +- pom.xml | 4 ++-- standalone-service/src/main/resources/application.properties| 5 - transport/s3-transport/pom.xml | 5 + .../airavata/mft/transport/s3/S3OutgoingStreamingConnector.java | 2 ++ 6 files changed, 19 insertions(+), 5 deletions(-)
[airavata-mft] branch master updated (2a54633 -> a61fa4a)
This is an automated email from the ASF dual-hosted git repository. dimuthuupe pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airavata-mft.git from 2a54633 Implement dedicated class loaders for transports add a61fa4a Upgrading Jcraft to an actively maintained fork No new revisions were added by this update. Summary of changes: .../mft_cli/airavata_mft_cli/storage/__init__.py | 3 + python-cli/mft_cli/airavata_mft_cli/storage/scp.py | 75 ++ python-cli/mft_cli/pyproject.toml | 2 +- python-sdk/setup.cfg | 2 +- .../src/airavata_mft_sdk/scp/SCPStorage_pb2.py | 20 +++--- .../stub/src/main/proto/scp/SCPStorage.proto | 1 - .../server/backend/sql/entity/SCPSecretEntity.java | 12 ++-- transport/scp-transport/pom.xml| 6 +- .../mft/transport/scp/SCPMetadataCollector.java| 18 +- 9 files changed, 115 insertions(+), 24 deletions(-) create mode 100644 python-cli/mft_cli/airavata_mft_cli/storage/scp.py
[airavata-mft] branch master updated (badd42c -> 2a54633)
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
from badd42c consul update to accept all
add 2a54633 Implement dedicated class loaders for transports
No new revisions were added by this update.
Summary of changes:
.gitignore | 3 +-
agent/service/pom.xml | 73 --
.../org/apache/airavata/mft/agent/AppConfig.java | 15 +-
.../org/apache/airavata/mft/agent/MFTAgent.java| 20 ---
.../airavata/mft/agent/TransferOrchestrator.java | 15 +-
.../airavata/mft/agent/TransportMediator.java | 13 +-
.../mft/agent/http/AgentHttpDownloadData.java | 106 --
.../apache/airavata/mft/agent/http/HttpServer.java | 78 --
.../airavata/mft/agent/http/HttpServerHandler.java | 160 -
.../mft/agent/http/HttpServerInitializer.java | 52 ---
.../mft/agent/http/HttpTransferRequestsStore.java | 90
.../mft/agent/ingress/ConsulIngressHandler.java| 7 +-
.../apache/airavata/mft/agent/rpc/RPCParser.java | 18 ++-
.../mft/agent/transport}/ConnectorResolver.java| 22 +--
.../transport}/MetadataCollectorResolver.java | 8 +-
.../mft/agent/transport/TransportClassLoader.java | 119 +++
.../agent/transport/TransportClassLoaderCache.java | 45 ++
.../main/resources/agent-application.properties| 3 +-
.../apache/airavata/mft/core/ResourceTypes.java| 23 ---
pom.xml| 2 +
python-cli/mft_cli/airavata_mft_cli/config.py | 4 +-
standalone-service/pom.xml | 1 -
.../server/StandaloneServiceApplication.java | 6 -
.../src/main/resources/application.properties | 7 +-
transport/azure-transport/pom.xml | 46 ++
.../src/main/assembly/transport-assembly.xml | 39 +
transport/box-transport/pom.xml| 45 ++
.../src/main/assembly/transport-assembly.xml | 40 +-
transport/dropbox-transport/pom.xml| 45 ++
.../src/main/assembly/transport-assembly.xml | 40 +-
transport/ftp-transport/pom.xml| 45 ++
.../src/main/assembly/transport-assembly.xml | 40 +-
transport/gcp-transport/pom.xml| 46 ++
.../src/main/assembly/transport-assembly.xml | 40 +-
transport/local-transport/pom.xml | 46 +-
.../src/main/assembly/transport-assembly.xml | 40 +-
transport/odata-transport/pom.xml | 46 ++
.../src/main/assembly/transport-assembly.xml | 40 +-
transport/s3-transport/pom.xml | 46 ++
.../src/main/assembly/transport-assembly.xml | 40 +-
transport/scp-transport/pom.xml| 45 ++
.../src/main/assembly/transport-assembly.xml | 40 +-
transport/swift-transport/pom.xml | 46 ++
.../src/main/assembly/transport-assembly.xml | 40 +-
44 files changed, 706 insertions(+), 1039 deletions(-)
delete mode 100644
agent/service/src/main/java/org/apache/airavata/mft/agent/http/AgentHttpDownloadData.java
delete mode 100644
agent/service/src/main/java/org/apache/airavata/mft/agent/http/HttpServer.java
delete mode 100644
agent/service/src/main/java/org/apache/airavata/mft/agent/http/HttpServerHandler.java
delete mode 100644
agent/service/src/main/java/org/apache/airavata/mft/agent/http/HttpServerInitializer.java
delete mode 100644
agent/service/src/main/java/org/apache/airavata/mft/agent/http/HttpTransferRequestsStore.java
rename {core/src/main/java/org/apache/airavata/mft/core =>
agent/service/src/main/java/org/apache/airavata/mft/agent/transport}/ConnectorResolver.java
(80%)
rename {core/src/main/java/org/apache/airavata/mft/core =>
agent/service/src/main/java/org/apache/airavata/mft/agent/transport}/MetadataCollectorResolver.java
(89%)
create mode 100644
agent/service/src/main/java/org/apache/airavata/mft/agent/transport/TransportClassLoader.java
create mode 100644
agent/service/src/main/java/org/apache/airavata/mft/agent/transport/TransportClassLoaderCache.java
delete mode 100644
core/src/main/java/org/apache/airavata/mft/core/ResourceTypes.java
copy agent/service/src/main/assembly/agent-bin-assembly.xml =>
transport/azure-transport/src/main/assembly/transport-assembly.xml (60%)
copy agent/service/src/main/assembly/agent-bin-assembly.xml =>
transport/box-transport/src/main/assembly/transport-assembly.xml (60%)
copy agent/service/src/main/assembly/agent-bin-assembly.xml =>
transport/dropbox-transport/src/main/assembly/transport-assembly.xml (60%)
copy agent/service/src/main/assembly/agent-bin-assembly.xml =>
transport/ftp-transport/src/main/assembly/tran
[airavata-mft] 02/05: Local incoming streamer added
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
commit e536a7d2f0ba13c0ecf438225a8219f2e2e2a28d
Author: Praneeth Chityala
AuthorDate: Sat Mar 18 01:35:21 2023 -0400
Local incoming streamer added
---
.../airavata/mft/core/ConnectorResolver.java | 6 +--
.../local/LocalIncomingStreamingConnector.java | 57 ++
2 files changed, 60 insertions(+), 3 deletions(-)
diff --git
a/core/src/main/java/org/apache/airavata/mft/core/ConnectorResolver.java
b/core/src/main/java/org/apache/airavata/mft/core/ConnectorResolver.java
index 13824d3..ea355bf 100644
--- a/core/src/main/java/org/apache/airavata/mft/core/ConnectorResolver.java
+++ b/core/src/main/java/org/apache/airavata/mft/core/ConnectorResolver.java
@@ -45,9 +45,9 @@ public final class ConnectorResolver {
case "GCS":
className =
"org.apache.airavata.mft.transport.gcp.GCSIncomingStreamingConnector";
break;
-//case "LOCAL":
-//className =
"org.apache.airavata.mft.transport.local.LocalIncomingStreamingConnector";
-//break;
+case "LOCAL":
+className =
"org.apache.airavata.mft.transport.local.LocalIncomingStreamingConnector";
+break;
}
if (className != null) {
diff --git
a/transport/local-transport/src/main/java/org/apache/airavata/mft/transport/local/LocalIncomingStreamingConnector.java
b/transport/local-transport/src/main/java/org/apache/airavata/mft/transport/local/LocalIncomingStreamingConnector.java
new file mode 100644
index 000..290cccb
--- /dev/null
+++
b/transport/local-transport/src/main/java/org/apache/airavata/mft/transport/local/LocalIncomingStreamingConnector.java
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.airavata.mft.transport.local;
+
+
+import org.apache.airavata.mft.core.api.ConnectorConfig;
+import org.apache.airavata.mft.core.api.IncomingStreamingConnector;
+
+import java.io.InputStream;
+import java.io.FileInputStream;
+import java.io.File;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class LocalIncomingStreamingConnector implements
IncomingStreamingConnector{
+
+private String resourcePath;
+
+private static final Logger logger =
LoggerFactory.getLogger(LocalIncomingStreamingConnector.class);
+
+@Override
+public void init(ConnectorConfig connectorConfig) throws Exception {
+this.resourcePath = connectorConfig.getResourcePath();
+}
+
+@Override
+public void complete() throws Exception {
+logger.info("File {} successfully received", this.resourcePath);
+}
+
+@Override
+public void failed() throws Exception {
+logger.error("Failed while receiving file {}", this.resourcePath);
+}
+
+@Override
+public InputStream fetchInputStream() throws Exception {
+InputStream from = new FileInputStream(new File(this.resourcePath));
+
+return from;
+}
+}
[airavata-mft] 04/05: S3 Outgoing bufferStreaming update
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
commit 29c05e15b5f2c9c57ad916a1de214c2a7d99f9fc
Author: Praneeth Chityala
AuthorDate: Sat Mar 18 01:36:59 2023 -0400
S3 Outgoing bufferStreaming update
---
.../org/apache/airavata/mft/transport/s3/S3OutgoingConnector.java | 8 +---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git
a/transport/s3-transport/src/main/java/org/apache/airavata/mft/transport/s3/S3OutgoingConnector.java
b/transport/s3-transport/src/main/java/org/apache/airavata/mft/transport/s3/S3OutgoingConnector.java
index 4803b32..6dfa022 100644
---
a/transport/s3-transport/src/main/java/org/apache/airavata/mft/transport/s3/S3OutgoingConnector.java
+++
b/transport/s3-transport/src/main/java/org/apache/airavata/mft/transport/s3/S3OutgoingConnector.java
@@ -36,7 +36,9 @@ import
org.apache.airavata.mft.resource.stubs.s3.storage.S3StorageGetRequest;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import java.io.BufferedInputStream;
import java.io.File;
+import java.io.FileInputStream;
import java.io.InputStream;
import java.nio.file.Files;
import java.nio.file.Paths;
@@ -61,7 +63,6 @@ public class S3OutgoingConnector implements
OutgoingChunkedConnector {
public void init(ConnectorConfig cc) throws Exception {
this.resourcePath = cc.getResourcePath();
-this.resourceLength = cc.getMetadata().getFile().getResourceSize();
s3Storage = cc.getStorage().getS3();
@@ -92,7 +93,8 @@ public class S3OutgoingConnector implements
OutgoingChunkedConnector {
.withPartNumber(chunkId + 1)
.withFileOffset(0)
//.withMD5Digest(Md5Utils.md5AsBase64(new
File(uploadFile)))
-.withFile(file)
+//.withFile(file)
+.withInputStream(new BufferedInputStream(new
FileInputStream(file), Math.min(16 * 1024 * 1024, (int) ( endByte -
startByte
.withPartSize(file.length());
UploadPartResult uploadResult = s3Client.uploadPart(uploadRequest);
@@ -137,6 +139,6 @@ public class S3OutgoingConnector implements
OutgoingChunkedConnector {
@Override
public void failed() throws Exception {
-
+logger.error("S3 failed to upload chunk to bucket {} for resource path
{}", s3Storage.getBucketName(), resourcePath);
}
}
[airavata-mft] 05/05: consul update to accept all
This is an automated email from the ASF dual-hosted git repository. dimuthuupe pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/airavata-mft.git commit badd42c810e3b58a1d266b204b39b12ab77293fe Author: Praneeth Chityala AuthorDate: Sat Mar 18 01:37:36 2023 -0400 consul update to accept all --- scripts/start-consul.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/start-consul.sh b/scripts/start-consul.sh index 74a79f2..fcc4a4e 100755 --- a/scripts/start-consul.sh +++ b/scripts/start-consul.sh @@ -27,7 +27,7 @@ case $1 in curl -O https://releases.hashicorp.com/consul/1.7.1/consul_1.7.1_darwin_amd64.zip unzip -o consul_1.7.1_darwin_amd64.zip -d ../airavata-mft/consul rm consul_1.7.1_darwin_amd64.zip -nohup ../airavata-mft/consul/consul agent -dev > $LOG_FILE 2>&1 & +nohup ../airavata-mft/consul/consul agent -dev -client 0.0.0.0 > $LOG_FILE 2>&1 & echo $! > $PID_PATH_NAME echo "Consul started" else
[airavata-mft] branch master updated (9c9e48c -> badd42c)
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
from 9c9e48c Turning on tcp keep alive for S3 clients
new 27dca07 gRPC ports config
new e536a7d Local incoming streamer added
new 33be009 Local incoming chunked update
new 29c05e1 S3 Outgoing bufferStreaming update
new badd42c consul update to accept all
The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../airavata/mft/core/ConnectorResolver.java | 6 +-
python-cli/mft_cli/airavata_mft_cli/config.py | 7 ++
python-cli/mft_cli/airavata_mft_cli/operations.py | 51 +++---
.../mft_cli/airavata_mft_cli/storage/__init__.py | 17 +++--
.../mft_cli/airavata_mft_cli/storage/azure.py | 17 +++--
python-cli/mft_cli/airavata_mft_cli/storage/gcs.py | 17 +++--
.../mft_cli/airavata_mft_cli/storage/local.py | 17 +++--
python-cli/mft_cli/airavata_mft_cli/storage/s3.py | 17 +++--
.../mft_cli/airavata_mft_cli/storage/swift.py | 17 +++--
scripts/start-consul.sh| 2 +-
.../local/LocalIncomingChunkedConnector.java | 81 +++---
...r.java => LocalIncomingStreamingConnector.java} | 59 ++--
.../mft/transport/s3/S3OutgoingConnector.java | 8 ++-
13 files changed, 147 insertions(+), 169 deletions(-)
create mode 100644 python-cli/mft_cli/airavata_mft_cli/config.py
copy
transport/local-transport/src/main/java/org/apache/airavata/mft/transport/local/{LocalIncomingChunkedConnector.java
=> LocalIncomingStreamingConnector.java} (53%)
[airavata-mft] 01/05: gRPC ports config
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
commit 27dca078dd7582646c8b9232beb58f526a1603f9
Author: Praneeth Chityala
AuthorDate: Sat Mar 18 01:32:57 2023 -0400
gRPC ports config
---
python-cli/mft_cli/airavata_mft_cli/config.py | 7 +++
python-cli/mft_cli/airavata_mft_cli/operations.py | 51 --
.../mft_cli/airavata_mft_cli/storage/__init__.py | 17 +---
.../mft_cli/airavata_mft_cli/storage/azure.py | 17 +---
python-cli/mft_cli/airavata_mft_cli/storage/gcs.py | 17 +---
.../mft_cli/airavata_mft_cli/storage/local.py | 17 +---
python-cli/mft_cli/airavata_mft_cli/storage/s3.py | 17 +---
.../mft_cli/airavata_mft_cli/storage/swift.py | 17 +---
8 files changed, 94 insertions(+), 66 deletions(-)
diff --git a/python-cli/mft_cli/airavata_mft_cli/config.py
b/python-cli/mft_cli/airavata_mft_cli/config.py
new file mode 100644
index 000..b5be669
--- /dev/null
+++ b/python-cli/mft_cli/airavata_mft_cli/config.py
@@ -0,0 +1,7 @@
+transfer_api_port = 7004
+transfer_api_secured = False
+resource_service_host = "localhost"
+resource_service_port = 7002
+resource_service_secured = False
+secret_service_host = "localhost"
+secret_service_port = 7003
\ No newline at end of file
diff --git a/python-cli/mft_cli/airavata_mft_cli/operations.py
b/python-cli/mft_cli/airavata_mft_cli/operations.py
index e6b9aec..40fbe64 100644
--- a/python-cli/mft_cli/airavata_mft_cli/operations.py
+++ b/python-cli/mft_cli/airavata_mft_cli/operations.py
@@ -23,15 +23,18 @@ from airavata_mft_sdk import MFTTransferApi_pb2
from rich.console import Console
from rich.table import Table
import time
+import sys
+sys.path.append('.')
+from . import config as configcli
def fetch_storage_and_secret_ids(storage_name):
- client = mft_client.MFTClient(transfer_api_port = 7003,
-transfer_api_secured = False,
-resource_service_host = "localhost",
-resource_service_port = 7003,
-resource_service_secured = False,
-secret_service_host = "localhost",
-secret_service_port = 7003)
+ client = mft_client.MFTClient(transfer_api_port =
configcli.transfer_api_port,
+transfer_api_secured =
configcli.transfer_api_secured,
+resource_service_host =
configcli.resource_service_host,
+resource_service_port =
configcli.resource_service_port,
+resource_service_secured =
configcli.resource_service_secured,
+secret_service_host =
configcli.secret_service_host,
+secret_service_port =
configcli.secret_service_port)
search_req = StorageCommon_pb2.StorageSearchRequest(storageName=storage_name)
storages = client.common_api.searchStorages(search_req)
@@ -68,13 +71,13 @@ def get_resource_metadata(storage_path, recursive_search =
False):
resourcePath =
resource_path)
resource_medata_req =
MFTTransferApi_pb2.FetchResourceMetadataRequest(idRequest = id_req)
- client = mft_client.MFTClient(transfer_api_port = 7003,
-transfer_api_secured = False,
-resource_service_host = "localhost",
-resource_service_port = 7003,
-resource_service_secured = False,
-secret_service_host = "localhost",
-secret_service_port = 7003)
+ client = mft_client.MFTClient(transfer_api_port =
configcli.transfer_api_port,
+transfer_api_secured =
configcli.transfer_api_secured,
+resource_service_host =
configcli.resource_service_host,
+resource_service_port =
configcli.resource_service_port,
+resource_service_secured =
configcli.resource_service_secured,
+secret_service_host =
configcli.secret_service_host,
+secret_service_port =
configcli.secret_service_port)
metadata_resp = client.transfer_api.resourceMetadata(resource_medata_req)
return metadata_resp
@@ -167,18 +170,18 @@ def copy(source, destination):
" files to be transferred. Total volume is " +
str(total_volume)
+ " bytes. Do you want to start the transfer? ",
True)
- client = mft_client.MFTClient(transfe
[airavata-mft] 03/05: Local incoming chunked update
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
commit 33be009066c43d32db9c89da96dd6444643fd231
Author: Praneeth Chityala
AuthorDate: Sat Mar 18 01:36:13 2023 -0400
Local incoming chunked update
---
.../local/LocalIncomingChunkedConnector.java | 81 +++---
1 file changed, 39 insertions(+), 42 deletions(-)
diff --git
a/transport/local-transport/src/main/java/org/apache/airavata/mft/transport/local/LocalIncomingChunkedConnector.java
b/transport/local-transport/src/main/java/org/apache/airavata/mft/transport/local/LocalIncomingChunkedConnector.java
index 91f5de6..dd388ab 100644
---
a/transport/local-transport/src/main/java/org/apache/airavata/mft/transport/local/LocalIncomingChunkedConnector.java
+++
b/transport/local-transport/src/main/java/org/apache/airavata/mft/transport/local/LocalIncomingChunkedConnector.java
@@ -21,22 +21,24 @@ package org.apache.airavata.mft.transport.local;
import org.apache.airavata.mft.core.api.ConnectorConfig;
import org.apache.airavata.mft.core.api.IncomingChunkedConnector;
-import java.io.InputStream;
-import java.io.FileInputStream;
-import java.io.FileOutputStream;
-import java.io.File;
+import java.io.*;
+import java.nio.file.Files;
+import java.nio.file.Path;
+
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class LocalIncomingChunkedConnector implements IncomingChunkedConnector
{
private String resourcePath;
+private long resourceSize;
private static final Logger logger =
LoggerFactory.getLogger(LocalIncomingChunkedConnector.class);
@Override
public void init(ConnectorConfig connectorConfig) throws Exception {
this.resourcePath = connectorConfig.getResourcePath();
+this.resourceSize =
connectorConfig.getMetadata().getFile().getResourceSize();
}
@Override
@@ -53,45 +55,40 @@ public class LocalIncomingChunkedConnector implements
IncomingChunkedConnector {
@Override
public void downloadChunk(int chunkId, long startByte, long endByte,
String downloadFile) throws Exception {
-FileInputStream from = new FileInputStream(new
File(this.resourcePath));
-FileOutputStream to = new FileOutputStream(new File(downloadFile));
-
-final int buffLen = 1024;
-
-byte[] buf = new byte[buffLen];
-
-from.skip(startByte);
-
-long fileSize = endByte - startByte + 1;
-
-while (true) {
-int bufSize = 0;
-
-if (buffLen < fileSize) {
-bufSize = buffLen;
-} else {
-bufSize = (int) fileSize;
-}
-
-bufSize = (int) from.read(buf, 0, bufSize);
-
-if (bufSize < 0) {
-break;
-}
-
-to.write(buf, 0, bufSize);
-to.flush();
-
-fileSize -= bufSize;
-
-if (fileSize == 0L) {
-break;
+logger.info("Downloading chunk {} with start byte {} and end byte {}
to file {} from resource path {}",
+chunkId, startByte, endByte, downloadFile, this.resourcePath);
+
+//#use this code on a DMA enabled device
+//if (resourceSize <= endByte - startByte) {
+//Files.copy(Path.of(this.resourcePath), Path.of(downloadFile));
+//} else {
+//try (FileInputStream from = new
FileInputStream(this.resourcePath);
+// FileOutputStream to = new FileOutputStream(downloadFile)) {
+//from.getChannel().transferTo(startByte, endByte - startByte,
to.getChannel());
+//} catch (Exception e) {
+//logger.error("Unexpected error occurred while downloading
chunk {} to file {} from resource path {}",
+//chunkId, downloadFile, this.resourcePath, e);
+//throw e;
+//}
+//}
+
+int buffLen = 1024 * 1024 * 16;
+try (BufferedInputStream bis = new BufferedInputStream(new
FileInputStream(this.resourcePath),buffLen);
+ BufferedOutputStream bos = new BufferedOutputStream(new
FileOutputStream(downloadFile))) {
+byte[] buffer = new byte[buffLen];
+int read = 0;
+long totalRead = bis.skip(startByte);
+while ((read = bis.read(buffer,0,Math.min(buffLen, (int) (endByte
- totalRead > 0) {
+bos.write(buffer, 0, read);
+totalRead += read;
}
+bis.close();
+bos.close();
+} catch (Exception e) {
+logger.error("Unexpected error occurred while downloading chunk {}
to file {} from resource path {}",
+chunkId, downloadFile, this.resourcePath, e);
+throw e;
}
-
-from.close();
-to.close();
-
}
[airavata] branch develop updated: Removing realtime monitoring from condor jobs
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch develop
in repository https://gitbox.apache.org/repos/asf/airavata.git
The following commit(s) were added to refs/heads/develop by this push:
new 8a95dc0738 Removing realtime monitoring from condor jobs
8a95dc0738 is described below
commit 8a95dc0738c947adb166e85e3fa93cf689b41003
Author: DImuthuUpe
AuthorDate: Fri Mar 17 12:32:43 2023 -0400
Removing realtime monitoring from condor jobs
---
.../airavata/helix/impl/task/submission/JobSubmissionTask.java | 5 -
.../configuration/server/src/main/resources/HTCONDOR_Groovy.template | 2 +-
2 files changed, 5 insertions(+), 2 deletions(-)
diff --git
a/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/JobSubmissionTask.java
b/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/JobSubmissionTask.java
index 70d6299aec..5b815b87fe 100644
---
a/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/JobSubmissionTask.java
+++
b/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/JobSubmissionTask.java
@@ -32,6 +32,7 @@ import
org.apache.airavata.helix.impl.task.submission.config.JobFactory;
import
org.apache.airavata.helix.impl.task.submission.config.JobManagerConfiguration;
import org.apache.airavata.helix.impl.task.submission.config.RawCommandInfo;
import org.apache.airavata.model.appcatalog.computeresource.ResourceJobManager;
+import
org.apache.airavata.model.appcatalog.computeresource.ResourceJobManagerType;
import org.apache.airavata.model.job.JobModel;
import org.apache.airavata.model.status.JobStatus;
import org.apache.commons.io.FileUtils;
@@ -59,7 +60,9 @@ public abstract class JobSubmissionTask extends AiravataTask {
JobManagerConfiguration jobManagerConfiguration =
JobFactory.getJobManagerConfiguration(JobFactory.getResourceJobManager(
getRegistryServiceClient(),
getTaskContext().getJobSubmissionProtocol(),
getTaskContext().getPreferredJobSubmissionInterface()));
-addMonitoringCommands(groovyMapData);
+if
(getTaskContext().getResourceJobManager().getResourceJobManagerType() !=
ResourceJobManagerType.HTCONDOR) {
+addMonitoringCommands(groovyMapData);
+}
String scriptAsString =
groovyMapData.loadFromFile(jobManagerConfiguration.getJobDescriptionTemplateName());
logger.info("Generated job submission script : " + scriptAsString);
diff --git
a/modules/configuration/server/src/main/resources/HTCONDOR_Groovy.template
b/modules/configuration/server/src/main/resources/HTCONDOR_Groovy.template
index 9e4cb974a9..c0f6501949 100644
--- a/modules/configuration/server/src/main/resources/HTCONDOR_Groovy.template
+++ b/modules/configuration/server/src/main/resources/HTCONDOR_Groovy.template
@@ -23,7 +23,7 @@
if (inputFiles != null && inputFiles.size() > 0) out.print
'transfer_input_files = '
if (inputFiles != null && inputFiles.size() > 0) for(file in inputFiles)
out.print file + ', '
def preJobInputs = checkCommand(preJobCommands, 'transfer_input_files =')
- if (preJobInputs) out.print ',' + preJobInputs.substring(22)
+ if (preJobInputs) out.print preJobInputs.substring(22)
if (inputFiles != null && inputFiles.size() > 0) out.print '\n'
if (moduleCommands != null) for(mc in moduleCommands) out.print mc +'\n'
[airavata-mft] branch master updated: Turning on tcp keep alive for S3 clients
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
The following commit(s) were added to refs/heads/master by this push:
new 9c9e48c Turning on tcp keep alive for S3 clients
9c9e48c is described below
commit 9c9e48c962feeb6f17dc3c4246757c349a1a0e2a
Author: DImuthuUpe
AuthorDate: Thu Mar 16 11:36:50 2023 -0400
Turning on tcp keep alive for S3 clients
---
.../src/main/java/org/apache/airavata/mft/transport/s3/S3Util.java | 5 +
1 file changed, 5 insertions(+)
diff --git
a/transport/s3-transport/src/main/java/org/apache/airavata/mft/transport/s3/S3Util.java
b/transport/s3-transport/src/main/java/org/apache/airavata/mft/transport/s3/S3Util.java
index ecd178d..9495ec8 100644
---
a/transport/s3-transport/src/main/java/org/apache/airavata/mft/transport/s3/S3Util.java
+++
b/transport/s3-transport/src/main/java/org/apache/airavata/mft/transport/s3/S3Util.java
@@ -17,6 +17,7 @@
package org.apache.airavata.mft.transport.s3;
+import com.amazonaws.ClientConfiguration;
import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
@@ -93,10 +94,14 @@ public class S3Util {
s3Secret.getSessionToken());
}
+ClientConfiguration clientConfiguration = new ClientConfiguration();
+clientConfiguration.setUseTcpKeepAlive(true);
+
AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
.withEndpointConfiguration(new
AwsClientBuilder.EndpointConfiguration(
s3Storage.getEndpoint(), s3Storage.getRegion()))
.withCredentials(new AWSStaticCredentialsProvider(awsCreds))
+.withClientConfiguration(clientConfiguration)
.disableChunkedEncoding()
.build();
[airavata-mft] branch master updated: Using random path for chunk files
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
The following commit(s) were added to refs/heads/master by this push:
new c2311f7 Using random path for chunk files
c2311f7 is described below
commit c2311f74a13e385da6fbf79a783656bc2ae61bfb
Author: DImuthuUpe
AuthorDate: Tue Mar 7 09:55:32 2023 -0500
Using random path for chunk files
---
.../src/main/java/org/apache/airavata/mft/agent/TransportMediator.java | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git
a/agent/service/src/main/java/org/apache/airavata/mft/agent/TransportMediator.java
b/agent/service/src/main/java/org/apache/airavata/mft/agent/TransportMediator.java
index f009772..b67298d 100644
---
a/agent/service/src/main/java/org/apache/airavata/mft/agent/TransportMediator.java
+++
b/agent/service/src/main/java/org/apache/airavata/mft/agent/TransportMediator.java
@@ -30,6 +30,7 @@ import java.io.File;
import java.io.InputStream;
import java.io.OutputStream;
import java.util.Optional;
+import java.util.UUID;
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicLong;
@@ -277,7 +278,7 @@ public class TransportMediator {
InputStream inputStream =
downloader.downloadChunk(chunkIdx, startPos, endPos);
uploader.uploadChunk(chunkIdx, startPos, endPos,
inputStream);
} else {
-String tempFile = tempDataDir + File.separator +
transferId + "-" + chunkIdx;
+String tempFile = tempDataDir + File.separator +
UUID.randomUUID().toString() + "-" + chunkIdx;
downloader.downloadChunk(chunkIdx, startPos, endPos,
tempFile);
uploader.uploadChunk(chunkIdx, startPos, endPos, tempFile);
new File(tempFile).delete();
[airavata-mft] branch master updated: S3 Connection Caching
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
The following commit(s) were added to refs/heads/master by this push:
new 149d816 S3 Connection Caching
149d816 is described below
commit 149d816e8fabf66c679e0ec23fcfac554d5c76f6
Author: DImuthuUpe
AuthorDate: Tue Mar 7 09:31:37 2023 -0500
S3 Connection Caching
---
.../airavata/mft/agent/TransferOrchestrator.java | 2 +
.../airavata/mft/core/api/ConnectorConfig.java | 17
.../mft/transport/s3/S3IncomingConnector.java | 17 +---
.../mft/transport/s3/S3MetadataCollector.java | 31 +-
.../mft/transport/s3/S3OutgoingConnector.java | 96 ++-
.../apache/airavata/mft/transport/s3/S3Util.java | 106 +
6 files changed, 182 insertions(+), 87 deletions(-)
diff --git
a/agent/service/src/main/java/org/apache/airavata/mft/agent/TransferOrchestrator.java
b/agent/service/src/main/java/org/apache/airavata/mft/agent/TransferOrchestrator.java
index 2eed2a0..dcdf01f 100644
---
a/agent/service/src/main/java/org/apache/airavata/mft/agent/TransferOrchestrator.java
+++
b/agent/service/src/main/java/org/apache/airavata/mft/agent/TransferOrchestrator.java
@@ -158,6 +158,7 @@ public class TransferOrchestrator {
.withSecret(sourceSecret)
.withStorage(sourceStorage)
.withResourcePath(endpointPath.getSourcePath())
+.withChunkSize(chunkedSize)
.withMetadata(srcMetadata).build();
ConnectorConfig dstCC =
ConnectorConfig.ConnectorConfigBuilder.newBuilder()
@@ -165,6 +166,7 @@ public class TransferOrchestrator {
.withStorage(destStorage)
.withSecret(destSecret)
.withResourcePath(endpointPath.getDestinationPath())
+.withChunkSize(chunkedSize)
.withMetadata(srcMetadata).build();
updateStatus.accept(endpointPath, new TransferState()
diff --git
a/core/src/main/java/org/apache/airavata/mft/core/api/ConnectorConfig.java
b/core/src/main/java/org/apache/airavata/mft/core/api/ConnectorConfig.java
index c87b6a6..8cd7538 100644
--- a/core/src/main/java/org/apache/airavata/mft/core/api/ConnectorConfig.java
+++ b/core/src/main/java/org/apache/airavata/mft/core/api/ConnectorConfig.java
@@ -28,6 +28,8 @@ public class ConnectorConfig {
private String resourcePath;
private ResourceMetadata metadata;
+private long chunkSize;
+
public String getTransferId() {
return transferId;
}
@@ -68,6 +70,13 @@ public class ConnectorConfig {
this.metadata = metadata;
}
+public long getChunkSize() {
+return chunkSize;
+}
+
+public void setChunkSize(long chunkSize) {
+this.chunkSize = chunkSize;
+}
public static final class ConnectorConfigBuilder {
private String transferId;
@@ -76,6 +85,8 @@ public class ConnectorConfig {
private String resourcePath;
private ResourceMetadata metadata;
+private long chunkSize;
+
private ConnectorConfigBuilder() {
}
@@ -108,6 +119,11 @@ public class ConnectorConfig {
return this;
}
+public ConnectorConfigBuilder withChunkSize(long chunkSize) {
+this.chunkSize = chunkSize;
+return this;
+}
+
public ConnectorConfig build() {
ConnectorConfig connectorConfig = new ConnectorConfig();
connectorConfig.setTransferId(transferId);
@@ -115,6 +131,7 @@ public class ConnectorConfig {
connectorConfig.setSecret(secret);
connectorConfig.setResourcePath(resourcePath);
connectorConfig.setMetadata(metadata);
+connectorConfig.setChunkSize(chunkSize);
return connectorConfig;
}
}
diff --git
a/transport/s3-transport/src/main/java/org/apache/airavata/mft/transport/s3/S3IncomingConnector.java
b/transport/s3-transport/src/main/java/org/apache/airavata/mft/transport/s3/S3IncomingConnector.java
index 8de3541..0f49480 100644
---
a/transport/s3-transport/src/main/java/org/apache/airavata/mft/transport/s3/S3IncomingConnector.java
+++
b/transport/s3-transport/src/main/java/org/apache/airavata/mft/transport/s3/S3IncomingConnector.java
@@ -54,21 +54,8 @@ public class S3IncomingConnector implements
IncomingChunkedConnector, IncomingSt
S3Secret s3Secret = cc.getSecret().getS3();
-AWSCredentials awsCreds;
-
-if (s3Secret.getSessionToken() == null ||
s3Secret.getSessionToken().equals("")) {
-awsCreds = new BasicAWSCredentials(s3Secret.getAccessKey(),
s3Secret.getSecretKey());
-} else {
-awsCreds = new BasicSessionCredentials(s3Secret.ge
[airavata] branch develop updated: Merging transfer input files from pre job commands for HTCondor script
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch develop
in repository https://gitbox.apache.org/repos/asf/airavata.git
The following commit(s) were added to refs/heads/develop by this push:
new 5836029d0e Merging transfer input files from pre job commands for
HTCondor script
5836029d0e is described below
commit 5836029d0e0237552afa1d2751e491aa734062b1
Author: DImuthuUpe
AuthorDate: Fri Mar 3 11:36:06 2023 -0500
Merging transfer input files from pre job commands for HTCondor script
---
.../server/src/main/resources/HTCONDOR_Groovy.template| 8 +++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git
a/modules/configuration/server/src/main/resources/HTCONDOR_Groovy.template
b/modules/configuration/server/src/main/resources/HTCONDOR_Groovy.template
index 3625d0b065..9e4cb974a9 100644
--- a/modules/configuration/server/src/main/resources/HTCONDOR_Groovy.template
+++ b/modules/configuration/server/src/main/resources/HTCONDOR_Groovy.template
@@ -1,5 +1,8 @@
# HTCondor job submission script generated by Apache Airavata
<%
+
+ def checkCommand = {items, prefix -> for (it in items) if
(it.startsWith(prefix)) return it}
+
if (executablePath != null && executablePath != "") out.print 'executable
= ' + executablePath + '\n'
if (inputs != null && inputs.size() > 0) out.print 'arguments = \"'
if (inputs != null && inputs.size() > 0) for(input in inputs) out.print
input + ' '
@@ -19,9 +22,12 @@
out.print 'should_transfer_files = Yes\nwhen_to_transfer_output = ON_EXIT\n'
if (inputFiles != null && inputFiles.size() > 0) out.print
'transfer_input_files = '
if (inputFiles != null && inputFiles.size() > 0) for(file in inputFiles)
out.print file + ', '
+ def preJobInputs = checkCommand(preJobCommands, 'transfer_input_files =')
+ if (preJobInputs) out.print ',' + preJobInputs.substring(22)
+
if (inputFiles != null && inputFiles.size() > 0) out.print '\n'
if (moduleCommands != null) for(mc in moduleCommands) out.print mc +'\n'
- if (preJobCommands != null) for(pjc in preJobCommands) out.print pjc +'\n'
+ if (preJobCommands != null) for(pjc in preJobCommands) if
(!pjc.startsWith("transfer_input_files =")) out.print pjc +'\n'
out.print 'queue\n'
if (postJobCommands != null) for(pjc in postJobCommands) out.print pjc
+'\n'
%>
[airavata-mft] branch master updated: Connection pooling for swift
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
The following commit(s) were added to refs/heads/master by this push:
new a8fb679 Connection pooling for swift
a8fb679 is described below
commit a8fb6790da5aeb463494fc106781c705408b97ca
Author: Dimuthu Wannipurage
AuthorDate: Sun Feb 26 12:03:01 2023 -0500
Connection pooling for swift
---
.../transport/swift/SwiftIncomingConnector.java| 14 ++---
.../transport/swift/SwiftMetadataCollector.java| 10 ++--
.../transport/swift/SwiftOutgoingConnector.java| 13 ++--
.../airavata/mft/transport/swift/SwiftUtil.java| 69 ++
4 files changed, 71 insertions(+), 35 deletions(-)
diff --git
a/transport/swift-transport/src/main/java/org/apache/airavata/mft/transport/swift/SwiftIncomingConnector.java
b/transport/swift-transport/src/main/java/org/apache/airavata/mft/transport/swift/SwiftIncomingConnector.java
index 5f9a415..4c9ab88 100644
---
a/transport/swift-transport/src/main/java/org/apache/airavata/mft/transport/swift/SwiftIncomingConnector.java
+++
b/transport/swift-transport/src/main/java/org/apache/airavata/mft/transport/swift/SwiftIncomingConnector.java
@@ -20,13 +20,8 @@ package org.apache.airavata.mft.transport.swift;
import org.apache.airavata.mft.core.api.ConnectorConfig;
import org.apache.airavata.mft.core.api.IncomingChunkedConnector;
import org.apache.airavata.mft.credential.stubs.swift.SwiftSecret;
-import org.apache.airavata.mft.credential.stubs.swift.SwiftSecretGetRequest;
import org.apache.airavata.mft.resource.stubs.swift.storage.SwiftStorage;
-import
org.apache.airavata.mft.resource.stubs.swift.storage.SwiftStorageGetRequest;
-import org.jclouds.ContextBuilder;
import org.jclouds.http.options.GetOptions;
-import org.jclouds.openstack.keystone.auth.config.CredentialTypes;
-import org.jclouds.openstack.keystone.config.KeystoneProperties;
import org.jclouds.openstack.swift.v1.SwiftApi;
import org.jclouds.openstack.swift.v1.domain.SwiftObject;
import org.jclouds.openstack.swift.v1.features.ObjectApi;
@@ -36,7 +31,6 @@ import org.slf4j.LoggerFactory;
import java.io.File;
import java.io.InputStream;
import java.nio.file.StandardCopyOption;
-import java.util.Properties;
public class SwiftIncomingConnector implements IncomingChunkedConnector {
@@ -52,15 +46,15 @@ public class SwiftIncomingConnector implements
IncomingChunkedConnector {
SwiftStorage swiftStorage = cc.getStorage().getSwift();
this.resourcePath = cc.getResourcePath();
SwiftSecret swiftSecret = cc.getSecret().getSwift();
-swiftApi = SwiftUtil.createSwiftApi(swiftSecret, swiftStorage);
+swiftApi = SwiftUtil.getInstance().leaseSwiftApi(swiftSecret);
objectApi = swiftApi.getObjectApi(swiftStorage.getRegion(),
swiftStorage.getContainer());
}
@Override
public void complete() throws Exception {
-if (swiftApi != null) {
-swiftApi.close();
-}
+//if (swiftApi != null) {
+//swiftApi.close();
+//}
}
@Override
diff --git
a/transport/swift-transport/src/main/java/org/apache/airavata/mft/transport/swift/SwiftMetadataCollector.java
b/transport/swift-transport/src/main/java/org/apache/airavata/mft/transport/swift/SwiftMetadataCollector.java
index ad4fa85..9073133 100644
---
a/transport/swift-transport/src/main/java/org/apache/airavata/mft/transport/swift/SwiftMetadataCollector.java
+++
b/transport/swift-transport/src/main/java/org/apache/airavata/mft/transport/swift/SwiftMetadataCollector.java
@@ -28,8 +28,6 @@ import org.jclouds.openstack.swift.v1.domain.ObjectList;
import org.jclouds.openstack.swift.v1.domain.SwiftObject;
import org.jclouds.openstack.swift.v1.features.ObjectApi;
-import java.util.Properties;
-
public class SwiftMetadataCollector implements MetadataCollector {
boolean initialized = false;
private SwiftStorage swiftStorage;
@@ -52,7 +50,7 @@ public class SwiftMetadataCollector implements
MetadataCollector {
public ResourceMetadata getResourceMetadata(String resourcePath, boolean
recursiveSearch) throws Exception {
checkInitialized();
-SwiftApi swiftApi = SwiftUtil.createSwiftApi(swiftSecret,
swiftStorage);
+SwiftApi swiftApi = SwiftUtil.getInstance().leaseSwiftApi(swiftSecret);
try {
ResourceMetadata.Builder resourceBuilder =
ResourceMetadata.newBuilder();
@@ -105,7 +103,7 @@ public class SwiftMetadataCollector implements
MetadataCollector {
}
return resourceBuilder.build();
} finally{
-swiftApi.close();
+//swiftApi.close();
}
}
@@ -113,7 +111,7 @@ public class SwiftMetadataCollector implements
MetadataCollector {
public Boolean isAvailable(String resourcePath) throws Exception
[airavata-mft] 03/03: Latest python sdk and cli release
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
commit c8fd34282f68d5285679fc55b6ec8b5433268a56
Author: Dimuthu Wannipurage
AuthorDate: Tue Feb 21 15:37:49 2023 -0500
Latest python sdk and cli release
---
python-cli/README.md | 2 +-
python-cli/mft_cli/pyproject.toml | 4 ++--
python-sdk/setup.cfg | 2 +-
3 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/python-cli/README.md b/python-cli/README.md
index b076417..3a2639c 100644
--- a/python-cli/README.md
+++ b/python-cli/README.md
@@ -37,7 +37,7 @@ Install dependencies
```
pip install grpcio==1.46.3
pip install grpcio-tools==1.46.3
-pip install airavata_mft_sdk==0.0.1-alpha24
+pip install airavata_mft_sdk==0.0.1-alpha26
```
Build the binary
diff --git a/python-cli/mft_cli/pyproject.toml
b/python-cli/mft_cli/pyproject.toml
index d630c00..630f44b 100644
--- a/python-cli/mft_cli/pyproject.toml
+++ b/python-cli/mft_cli/pyproject.toml
@@ -18,7 +18,7 @@
[tool.poetry]
name = "airavata-mft-cli"
-version = "0.1.7"
+version = "0.1.8"
description = "Command Line Client for Airavata MFT data transfer software"
authors = ["Apache Airavata "]
readme = "README.md"
@@ -32,7 +32,7 @@ typer = {extras = ["all"], version = "^0.7.0"}
pick = {version= "2.2.0"}
grpcio= [{version="1.46.3", markers = "platform_machine !=
'arm64'"},{version="1.47.0rc1", markers = "platform_machine == 'arm64'"}]
grpcio-tools = [{version="1.46.3", markers = "platform_machine !=
'arm64'"},{version="1.47.0rc1", markers = "platform_machine == 'arm64'"}]
-airavata_mft_sdk= {version="0.0.1-alpha24"}
+airavata_mft_sdk= {version="0.0.1-alpha26"}
[build-system]
requires = ["poetry-core"]
diff --git a/python-sdk/setup.cfg b/python-sdk/setup.cfg
index 519de26..f9be735 100644
--- a/python-sdk/setup.cfg
+++ b/python-sdk/setup.cfg
@@ -16,7 +16,7 @@
# under the License.
[metadata]
name = airavata_mft_sdk
-version = 0.0.1-alpha24
+version = 0.0.1-alpha26
author = Airavata MFT Developers
author_email = d...@airavata.apache.org
description = Python SDK for Apache Airavata Managed File Transfers (MFT)
[airavata-mft] branch master updated (58570dc -> c8fd342)
This is an automated email from the ASF dual-hosted git repository. dimuthuupe pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airavata-mft.git from 58570dc Refactoring API new 3eadd94 Support to delete old transfer status new 17efc59 Minor bug fixes new c8fd342 Latest python sdk and cli release The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../airavata/mft/agent/TransportMediator.java | 3 +- .../airavata/mft/api/handler/MFTApiHandler.java| 16 +++- api/stub/src/main/proto/MFTTransferApi.proto | 9 ++ .../apache/airavata/mft/admin/MFTConsulClient.java | 6 ++ python-cli/README.md | 2 +- python-cli/mft_cli/pyproject.toml | 4 +- python-sdk/setup.cfg | 2 +- .../src/airavata_mft_sdk/MFTTransferApi_pb2.py | 90 .../airavata_mft_sdk/MFTTransferApi_pb2_grpc.py| 97 +++--- 9 files changed, 103 insertions(+), 126 deletions(-)
[airavata-mft] 02/03: Minor bug fixes
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
commit 17efc597b85b158ed82739f2a2b387e3ba70d742
Author: Dimuthu Wannipurage
AuthorDate: Tue Feb 21 15:37:27 2023 -0500
Minor bug fixes
---
.../org/apache/airavata/mft/agent/TransportMediator.java | 3 ++-
.../apache/airavata/mft/api/handler/MFTApiHandler.java | 16 +++-
2 files changed, 17 insertions(+), 2 deletions(-)
diff --git
a/agent/service/src/main/java/org/apache/airavata/mft/agent/TransportMediator.java
b/agent/service/src/main/java/org/apache/airavata/mft/agent/TransportMediator.java
index 3fc15bd..f009772 100644
---
a/agent/service/src/main/java/org/apache/airavata/mft/agent/TransportMediator.java
+++
b/agent/service/src/main/java/org/apache/airavata/mft/agent/TransportMediator.java
@@ -140,6 +140,7 @@ public class TransportMediator {
for (int i = 0; i < chunkIdx; i++) {
Future future = completionService.take();
+future.get();
}
inConnector.complete();
@@ -283,7 +284,7 @@ public class TransportMediator {
}
return chunkIdx;
} catch (Exception e) {
-logger.error("Failed to transfer ", e);
+logger.error("Failed to transfer transfer id {}", transferId,
e);
throw e;
}
}
diff --git
a/api/service/src/main/java/org/apache/airavata/mft/api/handler/MFTApiHandler.java
b/api/service/src/main/java/org/apache/airavata/mft/api/handler/MFTApiHandler.java
index e595281..caecae3 100644
---
a/api/service/src/main/java/org/apache/airavata/mft/api/handler/MFTApiHandler.java
+++
b/api/service/src/main/java/org/apache/airavata/mft/api/handler/MFTApiHandler.java
@@ -200,7 +200,7 @@ public class MFTApiHandler extends
MFTTransferServiceGrpc.MFTTransferServiceImpl
(completedFiles.size() + failedFiles.size() +
pendingFiles.size()));
} else if (!completedFiles.isEmpty()) {
stateBuilder.setState("COMPLETED");
-stateBuilder.setState("All file transfers completed");
+stateBuilder.setDescription("All file transfers
completed");
stateBuilder.setPercentage((completedFiles.size() +
failedFiles.size()) * 1.0 /
(completedFiles.size() + failedFiles.size() +
pendingFiles.size()));
}
@@ -454,6 +454,20 @@ public class MFTApiHandler extends
MFTTransferServiceGrpc.MFTTransferServiceImpl
}
}
+@Override
+public void removeTransfer(TransferRemoveRequest request,
StreamObserver responseObserver) {
+try {
+mftConsulClient.removeTransfer(request.getTransferId());
+
responseObserver.onNext(TransferRemoveResponse.newBuilder().setSuccess(true).build());
+responseObserver.onCompleted();
+} catch (Exception e) {
+logger.error("Error while removing transfer {}",
request.getTransferId() , e);
+responseObserver.onError(Status.INTERNAL
+.withDescription("Failed to remove the transfer. " +
e.getMessage())
+.asException());
+}
+}
+
private String derriveTargetAgent(GetResourceMetadataRequest
directRequest) throws Exception {
String targetAgent = "";
[airavata-mft] 01/03: Support to delete old transfer status
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
commit 3eadd942052d44c606f9120c73415e681afe97d7
Author: Dimuthu Wannipurage
AuthorDate: Tue Feb 21 15:37:05 2023 -0500
Support to delete old transfer status
---
api/stub/src/main/proto/MFTTransferApi.proto | 9 ++
.../apache/airavata/mft/admin/MFTConsulClient.java | 6 ++
.../src/airavata_mft_sdk/MFTTransferApi_pb2.py | 90
.../airavata_mft_sdk/MFTTransferApi_pb2_grpc.py| 97 +++---
4 files changed, 82 insertions(+), 120 deletions(-)
diff --git a/api/stub/src/main/proto/MFTTransferApi.proto
b/api/stub/src/main/proto/MFTTransferApi.proto
index 3629a08..83641d4 100644
--- a/api/stub/src/main/proto/MFTTransferApi.proto
+++ b/api/stub/src/main/proto/MFTTransferApi.proto
@@ -94,6 +94,14 @@ message FetchResourceMetadataRequest {
org.apache.airavata.mft.common.AuthToken mftAuthorizationToken = 3;
}
+message TransferRemoveRequest {
+string transferId = 1;
+}
+
+message TransferRemoveResponse {
+bool success = 1;
+}
+
service MFTTransferService {
rpc submitTransfer(TransferApiRequest) returns (TransferApiResponse);
@@ -106,4 +114,5 @@ service MFTTransferService {
rpc resourceMetadata(FetchResourceMetadataRequest) returns
(org.apache.airavata.mft.agent.stub.ResourceMetadata);
+rpc removeTransfer(TransferRemoveRequest) returns (TransferRemoveResponse);
}
\ No newline at end of file
diff --git
a/common/common-clients/src/main/java/org/apache/airavata/mft/admin/MFTConsulClient.java
b/common/common-clients/src/main/java/org/apache/airavata/mft/admin/MFTConsulClient.java
index d2e0252..faf14f3 100644
---
a/common/common-clients/src/main/java/org/apache/airavata/mft/admin/MFTConsulClient.java
+++
b/common/common-clients/src/main/java/org/apache/airavata/mft/admin/MFTConsulClient.java
@@ -26,6 +26,7 @@ import com.orbitz.consul.ConsulException;
import com.orbitz.consul.KeyValueClient;
import com.orbitz.consul.SessionClient;
import com.orbitz.consul.model.kv.Value;
+import com.orbitz.consul.option.DeleteOptions;
import com.orbitz.consul.option.PutOptions;
import org.apache.airavata.mft.admin.models.AgentInfo;
import org.apache.airavata.mft.admin.models.TransferState;
@@ -445,6 +446,11 @@ public class MFTConsulClient {
}
}
+public void removeTransfer(String transferId) {
+kvClient.deleteKey(TRANSFER_STATE_PATH + transferId,
DeleteOptions.RECURSE);
+kvClient.deleteKey(TRANSFER_PROCESSED_PATH + transferId,
DeleteOptions.RECURSE);
+}
+
public List getLiveAgentInfos() throws MFTConsulClientException
{
List liveAgentIds = getLiveAgentIds();
return liveAgentIds.stream().map(id ->
getAgentInfo(id).get()).collect(Collectors.toList());
diff --git a/python-sdk/src/airavata_mft_sdk/MFTTransferApi_pb2.py
b/python-sdk/src/airavata_mft_sdk/MFTTransferApi_pb2.py
index 6476263..b197321 100644
--- a/python-sdk/src/airavata_mft_sdk/MFTTransferApi_pb2.py
+++ b/python-sdk/src/airavata_mft_sdk/MFTTransferApi_pb2.py
@@ -16,7 +16,7 @@ import airavata_mft_sdk.CredCommon_pb2 as CredCommon__pb2
import airavata_mft_sdk.MFTAgentStubs_pb2 as MFTAgentStubs__pb2
-DESCRIPTOR =
_descriptor_pool.Default().AddSerializedFile(b'\n\x14MFTTransferApi.proto\x12#org.apache.airavata.mft.api.service\x1a\x10\x43redCommon.proto\x1a\x13MFTAgentStubs.proto\"\x9b\x01\n\x10\x43\x61llbackEndpoint\x12P\n\x04type\x18\x01
\x01(\x0e\x32\x42.org.apache.airavata.mft.api.service.CallbackEndpoint.CallbackType\x12\x10\n\x08\x65ndpoint\x18\x02
\x01(\t\"#\n\x0c\x43\x61llbackType\x12\x08\n\x04HTTP\x10\x00\x12\t\n\x05KAFKA\x10\x01\"<\n\rEndpointPaths\x12\x12\n\nsourcePath\x18\x
[...]
+DESCRIPTOR =
_descriptor_pool.Default().AddSerializedFile(b'\n\x14MFTTransferApi.proto\x12#org.apache.airavata.mft.api.service\x1a\x10\x43redCommon.proto\x1a\x13MFTAgentStubs.proto\"\x9b\x01\n\x10\x43\x61llbackEndpoint\x12P\n\x04type\x18\x01
\x01(\x0e\x32\x42.org.apache.airavata.mft.api.service.CallbackEndpoint.CallbackType\x12\x10\n\x08\x65ndpoint\x18\x02
\x01(\t\"#\n\x0c\x43\x61llbackType\x12\x08\n\x04HTTP\x10\x00\x12\t\n\x05KAFKA\x10\x01\"<\n\rEndpointPaths\x12\x12\n\nsourcePath\x18\x
[...]
@@ -25,16 +25,14 @@ _ENDPOINTPATHS =
DESCRIPTOR.message_types_by_name['EndpointPaths']
_TRANSFERAPIREQUEST = DESCRIPTOR.message_types_by_name['TransferApiRequest']
_TRANSFERAPIREQUEST_TARGETAGENTSENTRY =
_TRANSFERAPIREQUEST.nested_types_by_name['TargetAgentsEntry']
_TRANSFERAPIRESPONSE = DESCRIPTOR.message_types_by_name['TransferApiResponse']
-_HTTPUPLOADAPIREQUEST =
DESCRIPTOR.message_types_by_name['HttpUploadApiRequest']
-_HTTPUPLOADAPIRESPONSE =
DESCRIPTOR.message_types_by_name['HttpUploadApiResponse']
-_HTTPDOWNLOADAPIREQUEST =
DESCRIPTOR.message_types_by_name['HttpDownloa
[airavata-mft] branch master updated: Refactoring API
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
The following commit(s) were added to refs/heads/master by this push:
new 58570dc Refactoring API
58570dc is described below
commit 58570dcab36ef0965749ff3b2e48d3289dbb095f
Author: DImuthuUpe
AuthorDate: Tue Feb 21 00:09:05 2023 -0500
Refactoring API
---
agent/stub/src/main/proto/MFTAgentStubs.proto | 1 -
.../airavata/mft/api/handler/MFTApiHandler.java| 67 --
api/stub/src/main/proto/MFTTransferApi.proto | 30 --
3 files changed, 10 insertions(+), 88 deletions(-)
diff --git a/agent/stub/src/main/proto/MFTAgentStubs.proto
b/agent/stub/src/main/proto/MFTAgentStubs.proto
index 451722b..121a2ad 100644
--- a/agent/stub/src/main/proto/MFTAgentStubs.proto
+++ b/agent/stub/src/main/proto/MFTAgentStubs.proto
@@ -120,6 +120,5 @@ message GetResourceMetadataRequest {
StorageWrapper storage = 2;
SecretWrapper secret = 3;
bool recursiveSearch = 4;
-string targetAgent = 5;
}
diff --git
a/api/service/src/main/java/org/apache/airavata/mft/api/handler/MFTApiHandler.java
b/api/service/src/main/java/org/apache/airavata/mft/api/handler/MFTApiHandler.java
index 9574eb1..e595281 100644
---
a/api/service/src/main/java/org/apache/airavata/mft/api/handler/MFTApiHandler.java
+++
b/api/service/src/main/java/org/apache/airavata/mft/api/handler/MFTApiHandler.java
@@ -131,59 +131,6 @@ public class MFTApiHandler extends
MFTTransferServiceGrpc.MFTTransferServiceImpl
}
}
-@Override
-public void submitHttpUpload(HttpUploadApiRequest request,
StreamObserver responseObserver) {
-super.submitHttpUpload(request, responseObserver);
-}
-
-@Override
-public void submitHttpDownload(HttpDownloadApiRequest request,
StreamObserver responseObserver) {
-try {
-// TODO : Automatically derive agent if the target agent is empty
-
-logger.info("Processing submit http download for resource path
{}", request.getResourcePath());
-
-String targetAgent = derriveTargetAgent(request.getTargetAgent());
-
-SyncRPCRequest.SyncRPCRequestBuilder requestBuilder =
SyncRPCRequest.SyncRPCRequestBuilder.builder()
-.withAgentId(targetAgent)
-.withMessageId(UUID.randomUUID().toString())
-.withMethod("submitHttpDownload")
-.withParameter("resourcePath", request.getResourcePath())
-.withParameter("sourceStorageId",
request.getSourceStorageId())
-.withParameter("sourceToken", request.getSourceSecretId())
-.withParameter("mftAuthorizationToken",
JsonFormat.printer().print(request.getMftAuthorizationToken()));
-
-SyncRPCResponse rpcResponse =
agentRPCClient.sendSyncRequest(requestBuilder.build());
-
-switch (rpcResponse.getResponseStatus()) {
-case SUCCESS:
-String url = rpcResponse.getResponseAsStr();
-HttpDownloadApiResponse downloadResponse =
HttpDownloadApiResponse.newBuilder()
-.setUrl(url)
-.setTargetAgent(request.getTargetAgent()).build();
-responseObserver.onNext(downloadResponse);
-responseObserver.onCompleted();
-return;
-case FAIL:
-logger.error("Errored while processing the download
request to resource path {}. Error msg : {}",
-request.getResourcePath(),
rpcResponse.getErrorAsStr());
-
-responseObserver.onError(Status.INTERNAL
-.withDescription("Errored while processing the the
fetch file metadata response. Error msg : " +
-rpcResponse.getErrorAsStr())
-.asException());
-}
-
-} catch (Exception e) {
-logger.error("Error while submitting http download request to
resource path {}",
-request.getResourcePath() , e);
-responseObserver.onError(Status.INTERNAL
-.withDescription("Failed to submit http download request.
" + e.getMessage())
-.asException());
-}
-}
-
@Override
public void getAllTransferStates(TransferStateApiRequest request,
StreamObserver responseObserver) {
try {
@@ -328,7 +275,6 @@ public class MFTApiHandler extends
MFTTransferServiceGrpc.MFTTransferServiceImpl
directReqBuilder
.setStorage(StorageWrapper.newBuilder().setLocal(localStorage).build());
-direc
[airavata-mft] branch master updated (e0b4d31 -> a0a4a15)
This is an automated email from the ASF dual-hosted git repository. dimuthuupe pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/airavata-mft.git from e0b4d31 Updating pending transfer count in agents new ac85bd2 Local Transport Changes new 1b33f49 merger conflicts resolved new 68b104a Local Transport OutgoingChunk new 3832259 Local Agent selection fixed new f3a9d67 Local storage type fix new a0a4a15 Merge pull request #90 from praneethchityala/local-transport The 383 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: agent/stub/src/main/proto/MFTAgentStubs.proto | 1 + .../airavata/mft/api/handler/MFTApiHandler.java| 6 +- .../airavata/mft/core/ConnectorResolver.java | 10 +- python-cli/mft_cli/airavata_mft_cli/operations.py | 2 + .../mft_cli/airavata_mft_cli/storage/__init__.py | 4 + .../mft_cli/airavata_mft_cli/storage/local.py | 43 .../local/LocalIncomingChunkedConnector.java | 106 .../transport/local/LocalMetadataCollector.java| 4 +- .../local/LocalOutgoingChunkedConnector.java | 110 + 9 files changed, 282 insertions(+), 4 deletions(-) create mode 100644 python-cli/mft_cli/airavata_mft_cli/storage/local.py create mode 100644 transport/local-transport/src/main/java/org/apache/airavata/mft/transport/local/LocalIncomingChunkedConnector.java create mode 100644 transport/local-transport/src/main/java/org/apache/airavata/mft/transport/local/LocalOutgoingChunkedConnector.java
[airavata-mft] branch master updated: Updating pending transfer count in agents
This is an automated email from the ASF dual-hosted git repository.
dimuthuupe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/airavata-mft.git
The following commit(s) were added to refs/heads/master by this push:
new e0b4d31 Updating pending transfer count in agents
e0b4d31 is described below
commit e0b4d31c4514c4a81eff910b8cbe9cacb7ff97fb
Author: Dimuthu Wannipurage
AuthorDate: Fri Feb 10 14:19:31 2023 -0500
Updating pending transfer count in agents
---
.../airavata/mft/agent/TransferOrchestrator.java | 13 +
.../apache/airavata/mft/admin/MFTConsulClient.java | 68 --
.../mft/controller/TransferDispatcher.java | 16 ++---
3 files changed, 71 insertions(+), 26 deletions(-)
diff --git
a/agent/service/src/main/java/org/apache/airavata/mft/agent/TransferOrchestrator.java
b/agent/service/src/main/java/org/apache/airavata/mft/agent/TransferOrchestrator.java
index 5395876..2eed2a0 100644
---
a/agent/service/src/main/java/org/apache/airavata/mft/agent/TransferOrchestrator.java
+++
b/agent/service/src/main/java/org/apache/airavata/mft/agent/TransferOrchestrator.java
@@ -17,6 +17,7 @@
package org.apache.airavata.mft.agent;
+import org.apache.airavata.mft.admin.MFTConsulClient;
import org.apache.airavata.mft.admin.models.TransferState;
import org.apache.airavata.mft.agent.stub.*;
import org.apache.airavata.mft.core.MetadataCollectorResolver;
@@ -25,6 +26,7 @@ import org.apache.airavata.mft.core.api.MetadataCollector;
import org.apache.commons.lang3.exception.ExceptionUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import org.springframework.beans.factory.annotation.Autowired;
import javax.annotation.PostConstruct;
import javax.annotation.PreDestroy;
@@ -61,6 +63,12 @@ public class TransferOrchestrator {
@org.springframework.beans.factory.annotation.Value("${agent.temp.data.dir}")
private String tempDataDir = "/tmp";
+@org.springframework.beans.factory.annotation.Value("${agent.id}")
+private String agentId;
+
+@Autowired
+private MFTConsulClient mftConsulClient;
+
@PostConstruct
public void init() {
transferRequestExecutor =
Executors.newFixedThreadPool(concurrentTransfers);
@@ -68,6 +76,7 @@ public class TransferOrchestrator {
concurrentTransfers,
concurrentChunkedThreads,
chunkedSize, doChunkStream);
+mftConsulClient.updateAgentPendingTransferCount(agentId, 0);
logger.info("Transfer orchestrator initialized");
}
@@ -81,6 +90,8 @@ public class TransferOrchestrator {
BiConsumer updateStatus,
BiConsumer
createTransferHook) {
long totalPending =
totalPendingTransfers.addAndGet(request.getEndpointPathsCount());
+mftConsulClient.updateAgentPendingTransferCount(agentId, totalPending);
+
logger.info("Total pending files to transfer {}", totalPending);
for (EndpointPaths endpointPath : request.getEndpointPathsList()) {
@@ -100,6 +111,8 @@ public class TransferOrchestrator {
long running = totalRunningTransfers.incrementAndGet();
long pending = totalPendingTransfers.decrementAndGet();
+mftConsulClient.updateAgentPendingTransferCount(agentId, pending);
+
logger.info("Received request {}. Total Running {}. Total Pending
{}", transferId, running, pending);
updateStatus.accept(endpointPath, new TransferState()
diff --git
a/common/common-clients/src/main/java/org/apache/airavata/mft/admin/MFTConsulClient.java
b/common/common-clients/src/main/java/org/apache/airavata/mft/admin/MFTConsulClient.java
index 273e6c4..d2e0252 100644
---
a/common/common-clients/src/main/java/org/apache/airavata/mft/admin/MFTConsulClient.java
+++
b/common/common-clients/src/main/java/org/apache/airavata/mft/admin/MFTConsulClient.java
@@ -71,6 +71,7 @@ public class MFTConsulClient {
public static final String TRANSFER_PROCESSED_PATH =
"mft/transfer/processed/";
public static final String AGENTS_TRANSFER_REQUEST_MESSAGE_PATH =
"mft/agents/transfermessages/";
public static final String AGENTS_SCHEDULED_PATH = "mft/agents/scheduled/";
+public static final String AGENTS_PENDING_TRANSFER_COUNT_PATH =
"mft/agents/pendingtransfers/";
public static final String CONTROLLER_STATE_MESSAGE_PATH =
"mft/controller/messages/states/";
public static final String CONTROLLER_TRANSFER_MESSAGE_PATH =
"mft/controller/messages/transfers/";
@@ -115,6 +116,32 @@ public class MFTConsulClient {
public MFTConsulClient() {
}
+public synchronized void updateAgentPendingTransferCount(String agentId,
long transferCount) {
+try {
+kvClient.putValue(AGENTS
