RE: HTCondor Extension Help

2020-10-08 Thread Daniel Shchur 
Hi Dimuthu,

That clears up a lot of confusion. Thank you!

- Daniel Shchur

From: DImuthu Upeksha
Sent: Thursday, October 8, 2020 4:58 PM
To: Airavata Dev
Cc: daniellthibod...@gmail.com; Evan 
Palmer
Subject: Re: HTCondor Extension Help

Hi Daniel,

That's correct, we assume all job manager configurations have unique set of 
command types like job submission, cancellation, status checking etc. It's up 
to the end user to specify the exact command for each command type. For 
example, SLURM job submission command is sbatch. So the user has to specify 
that though the portal which will eventually be stored in the database.

[cid:image002.png@01D69D94.65FDC9A0]
Thanks
Dimuthu


On Thu, Oct 8, 2020 at 5:14 PM Daniel Shchur 
mailto:daniel.shc...@huskers.unl.edu>> wrote:
Hi Dimuthu,

So, to be clear,

The relevant job submission commands are registered at the client-side? As in, 
the front end interface and are not hardcoded into the backend?

- Daniel Shchur

From: DImuthu Upeksha
Sent: Thursday, October 8, 2020 4:12 PM
To: Airavata Dev
Cc: daniellthibod...@gmail.com; Evan 
Palmer
Subject: Re: HTCondor Extension Help

Hi Daniel,

If you look at [2], it's where the protocol specific configuration object is 
created. Each implementation is a subclass of the JobManagerConfiguration 
interface [3]. So each implementation might have different execution commands 
and those are stored in the database and you can register them through PGA. 
Please look at the screenshot I sent earlier. Once helix figured out the right 
Configuration implementation, it invokes it at the [4]. Please let me know if 
this is clear

[2] 
https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/config/JobFactory.java#L106
[3] 
https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/config/JobManagerConfiguration.java
[4] 
https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/JobSubmissionTask.java#L78

Thanks
Dimuthu

On Thu, Oct 8, 2020 at 4:58 PM Daniel Shchur 
mailto:daniel.shc...@huskers.unl.edu>> wrote:
Hi all,

We have been trying to figure out where the helix system actually specifies job 
submission protocol commands (e.g. sbatch, squeue, scancel). The only direct 
references we could find are part of the client-sdk 
“registerSampleApplication”. Everything else, when referencing 
“ResourceJobConfiguration” seems to be fairly circular, or goes to dead-end 
files that may not be used anymore.

Question: is there a direct command mapping for the “Map” that is called within the code, or is this set elsewhere? We haven’t 
found any references to where Airavata would know how to submit a SLURM job so 
that we can let it know how to submit an HTCondor job.

Any help on this would be greatly appreciated. Thanks!

- Daniel Shchur

From: DImuthu Upeksha
Sent: Monday, October 5, 2020 9:50 AM
To: Airavata Dev
Cc: daniellthibod...@gmail.com; Evan 
Palmer
Subject: Re: HTCondor Extension Help

Sorry I didn't send the image I mentioned earlier



On Sun, Oct 4, 2020 at 10:51 PM DImuthu Upeksha 
mailto:dimuthu.upeks...@gmail.com>> wrote:
Hi Daniel,

Your

RE: HTCondor Extension Help

2020-10-08 Thread Daniel Shchur 
Hi Dimuthu,

So, to be clear,

The relevant job submission commands are registered at the client-side? As in, 
the front end interface and are not hardcoded into the backend?

- Daniel Shchur

From: DImuthu Upeksha
Sent: Thursday, October 8, 2020 4:12 PM
To: Airavata Dev
Cc: daniellthibod...@gmail.com; Evan 
Palmer
Subject: Re: HTCondor Extension Help

Hi Daniel,

If you look at [2], it's where the protocol specific configuration object is 
created. Each implementation is a subclass of the JobManagerConfiguration 
interface [3]. So each implementation might have different execution commands 
and those are stored in the database and you can register them through PGA. 
Please look at the screenshot I sent earlier. Once helix figured out the right 
Configuration implementation, it invokes it at the [4]. Please let me know if 
this is clear

[2] 
https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/config/JobFactory.java#L106
[3] 
https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/config/JobManagerConfiguration.java
[4] 
https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/JobSubmissionTask.java#L78

Thanks
Dimuthu

On Thu, Oct 8, 2020 at 4:58 PM Daniel Shchur 
mailto:daniel.shc...@huskers.unl.edu>> wrote:
Hi all,

We have been trying to figure out where the helix system actually specifies job 
submission protocol commands (e.g. sbatch, squeue, scancel). The only direct 
references we could find are part of the client-sdk 
“registerSampleApplication”. Everything else, when referencing 
“ResourceJobConfiguration” seems to be fairly circular, or goes to dead-end 
files that may not be used anymore.

Question: is there a direct command mapping for the “Map” that is called within the code, or is this set elsewhere? We haven’t 
found any references to where Airavata would know how to submit a SLURM job so 
that we can let it know how to submit an HTCondor job.

Any help on this would be greatly appreciated. Thanks!

- Daniel Shchur

From: DImuthu Upeksha
Sent: Monday, October 5, 2020 9:50 AM
To: Airavata Dev
Cc: daniellthibod...@gmail.com; Evan 
Palmer
Subject: Re: HTCondor Extension Help

Sorry I didn't send the image I mentioned earlier



On Sun, Oct 4, 2020 at 10:51 PM DImuthu Upeksha 
mailto:dimuthu.upeks...@gmail.com>> wrote:
Hi Daniel,

Your PR looks good to me. I think the best place for you to start is by setting 
up the PGA portal (the old portal we used before the django one) and adding a 
resource manager type for the compute resource as you can see in the attached 
image. You can set up the pga locally using this [1] guide. Once you can 
register the resource manager type using the pga portal, you can use the django 
portal to submit jobs. I believe that you don't need to change any code in the 
django portal.

[1]

Re: HTCondor Extension Help

2020-10-08 Thread DImuthu Upeksha 
Hi Daniel,

If you look at [2], it's where the protocol specific configuration object
is created. Each implementation is a subclass of the
JobManagerConfiguration interface [3]. So each implementation might have
different execution commands and those are stored in the database and you
can register them through PGA. Please look at the screenshot I sent
earlier. Once helix figured out the right Configuration implementation, it
invokes it at the [4]. Please let me know if this is clear

[2]
https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/config/JobFactory.java#L106
[3]
https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/config/JobManagerConfiguration.java
[4]
https://github.com/apache/airavata/blob/develop/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/task/submission/JobSubmissionTask.java#L78

Thanks
Dimuthu

On Thu, Oct 8, 2020 at 4:58 PM Daniel Shchur 
wrote:

> Hi all,
>
>
>
> We have been trying to figure out where the helix system actually
> specifies job submission protocol commands (e.g. sbatch, squeue, scancel).
> The only direct references we could find are part of the client-sdk
> “registerSampleApplication”. Everything else, when referencing
> “ResourceJobConfiguration” seems to be fairly circular, or goes to dead-end
> files that may not be used anymore.
>
>
>
> Question: is there a direct command mapping for the
> “Map” that is called within the code, or is this
> set elsewhere? We haven’t found any references to where Airavata would know
> how to submit a SLURM job so that we can let it know how to submit an
> HTCondor job.
>
> Any help on this would be greatly appreciated. Thanks!
>
>
>
> - Daniel Shchur
>
>
>
> *From: *DImuthu Upeksha 
> *Sent: *Monday, October 5, 2020 9:50 AM
> *To: *Airavata Dev 
> *Cc: *daniellthibod...@gmail.com; Evan Palmer
> 
> *Subject: *Re: HTCondor Extension Help
>
>
>
> Sorry I didn't send the image I mentioned earlier
>
>
>
>
>
>
>
> On Sun, Oct 4, 2020 at 10:51 PM DImuthu Upeksha <
> dimuthu.upeks...@gmail.com> wrote:
>
> Hi Daniel,
>
>
>
> Your PR looks good to me. I think the best place for you to start is by
> setting up the PGA portal (the old portal we used before the django one)
> and adding a resource manager type for the compute resource as you can see
> in the attached image. You can set up the pga locally using this [1] guide.
> Once you can register the resource manager type using the pga portal, you
> can use the django portal to submit jobs. I believe that you don't need to
> change any code in the django portal.
>
>
>
> [1]
> https://github.com/apache/airavata/tree/master/modules/ide-integration#optional-starting-super-admin-portal-pga
> 
>
>
>
> Thanks
>
> Dimuthu
>
>
>
> On Sun, Oct 4, 2020 at 6:12 PM Daniel Shchur <
> daniel.shc...@huskers.unl.edu> wrote:
>
> Hi Dimuthu and Suresh,
>
>
>
> Thanks for getting back to us. We went ahead and made a work in progress pull
> request
> 
> for our changes we have created so far. We also took a look at the
> job-monitor section and will work towards creating additions to support
> HTCondor.
>
>
>
> Do any of you have any suggestions on what the best way is to test
> submissions to HTCondor? We looked into using the Airavata API and the
> Django portal and are curious what you would recommend, and how to do it
> using your recommendation.
>
>
>
> Thanks!
>
>
>
> - Daniel Shchur
>
>
>
> *From: *DImuthu Upeksha 
> *Sent: *Friday, October 2, 2020 5:22 PM
> *To: *Airavata Dev 
> *Cc: *daniellthibod...@gmail.com; Evan Palmer
> 
> *Subject: *Re: HTCondor Extension Help
>
>
>
> Hi Daniel,
>
>
>
> This is great progress. I think you are on the right track. Did you manage
> to set up the ide integration for the development of Airavata. If you have
> already done that, you can test these features. However I recommend you to
> send pull requests to a separate branch (htcondor?) and then I can provide
> you feedback on the next steps.
>
>
>
> Thanks
>
> Dimuthu
>
>
>
>
>
> On Thu, Oct 1, 2020 at 4:57 PM Suresh Marru  wrote:
>
> Hi Daniel,
>
>
>
> Thanks for this nice summary for keeping the dev list apprised of your
> progress.
>
>
>
> One additional item is to discuss

RE: HTCondor Extension Help

2020-10-08 Thread Daniel Shchur 
Hi all,

We have been trying to figure out where the helix system actually specifies job 
submission protocol commands (e.g. sbatch, squeue, scancel). The only direct 
references we could find are part of the client-sdk 
“registerSampleApplication”. Everything else, when referencing 
“ResourceJobConfiguration” seems to be fairly circular, or goes to dead-end 
files that may not be used anymore.

Question: is there a direct command mapping for the “Map” that is called within the code, or is this set elsewhere? We haven’t 
found any references to where Airavata would know how to submit a SLURM job so 
that we can let it know how to submit an HTCondor job.

Any help on this would be greatly appreciated. Thanks!

- Daniel Shchur

From: DImuthu Upeksha
Sent: Monday, October 5, 2020 9:50 AM
To: Airavata Dev
Cc: daniellthibod...@gmail.com; Evan 
Palmer
Subject: Re: HTCondor Extension Help

Sorry I didn't send the image I mentioned earlier



On Sun, Oct 4, 2020 at 10:51 PM DImuthu Upeksha 
mailto:dimuthu.upeks...@gmail.com>> wrote:
Hi Daniel,

Your PR looks good to me. I think the best place for you to start is by setting 
up the PGA portal (the old portal we used before the django one) and adding a 
resource manager type for the compute resource as you can see in the attached 
image. You can set up the pga locally using this [1] guide. Once you can 
register the resource manager type using the pga portal, you can use the django 
portal to submit jobs. I believe that you don't need to change any code in the 
django portal.

[1] 
https://github.com/apache/airavata/tree/master/modules/ide-integration#optional-starting-super-admin-portal-pga

Thanks
Dimuthu

On Sun, Oct 4, 2020 at 6:12 PM Daniel Shchur 
mailto:daniel.shc...@huskers.unl.edu>> wrote:
Hi Dimuthu and Suresh,

Thanks for getting back to us. We went ahead and made a work in progress pull 
request
 for our changes we have created so far. We also took a look at the job-monitor 
section and will work towards creating additions to support HTCondor.

Do any of you have any suggestions on what the best way is to test submissions 
to HTCondor? We looked into using the Airavata API and the Django portal and 
are curious what you would recommend, and how to do it using your 
recommendation.

Thanks!

- Daniel Shchur

From: DImuthu Upeksha
Sent: Friday, October 2, 2020 5:22 PM
To: Airavata Dev
Cc: daniellthibod...@gmail.com; Evan 
Palmer
Subject: Re: HTCondor Extension Help

Hi Daniel,

This is great progress. I think you are on the right track. Did you manage to 
set up the ide integration for the development of Airavata. If you have already 
done that, you can test these features. However I recommend you to send pull 
requests to a separate branch (htcondor?) and then I can provide you feedback 
on the next steps.

Thanks
Dimuthu


On Thu, Oct 1, 2020 at 4:57 PM Suresh Marru 
mailto:sma...@apache.org>> wrote:
Hi Daniel,

Thanks for this nice summary for keeping the dev list apprised of your progress.

One additional item is to discuss and plan on job monitoring approaches, you 
can look at current references here - 
https://github.com/apache/airavata/tree/master/modules/job-monitor

Suresh

On Oct 1, 2020, at 4:52 PM, Daniel Shchur 
mailto:daniel.shc...@huskers.unl.edu>> wrote:

Hi all,

We are working on the HTCondor extension for Airavata. We have gone through the 
code and started to implement additions based on the other job workflow systems 
such as SLURM. We 
forked
 the project and created a new

[GitHub] [airavata] DImuthuUpe commented on pull request #263: [WIP] HTCondor Support Implementation

2020-10-08 Thread GitBox 


DImuthuUpe commented on pull request #263:
URL: https://github.com/apache/airavata/pull/263#issuecomment-705819419


   This is great progress. Have you manage to register HTCondor resource type 
from the pga? And please send the PR to airavata:htcondor branch.



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org