Re: HTCondor Extension Help

[DImuthu Upeksha]

Hi Daniel,

Your PR looks good to me. I think the best place for you to start is by
setting up the PGA portal (the old portal we used before the django one)
and adding a resource manager type for the compute resource as you can see
in the attached image. You can set up the pga locally using this [1] guide.
Once you can register the resource manager type using the pga portal, you
can use the django portal to submit jobs. I believe that you don't need to
change any code in the django portal.

[1]
https://github.com/apache/airavata/tree/master/modules/ide-integration#optional-starting-super-admin-portal-pga

Thanks
Dimuthu

On Sun, Oct 4, 2020 at 6:12 PM Daniel Shchur 
wrote:

> Hi Dimuthu and Suresh,
>
>
>
> Thanks for getting back to us. We went ahead and made a work in progress pull
> request  for our changes we
> have created so far. We also took a look at the job-monitor section and
> will work towards creating additions to support HTCondor.
>
>
>
> Do any of you have any suggestions on what the best way is to test
> submissions to HTCondor? We looked into using the Airavata API and the
> Django portal and are curious what you would recommend, and how to do it
> using your recommendation.
>
>
>
> Thanks!
>
>
>
> - Daniel Shchur
>
>
>
> *From: *DImuthu Upeksha 
> *Sent: *Friday, October 2, 2020 5:22 PM
> *To: *Airavata Dev 
> *Cc: *daniellthibod...@gmail.com; Evan Palmer
> 
> *Subject: *Re: HTCondor Extension Help
>
>
>
> Hi Daniel,
>
>
>
> This is great progress. I think you are on the right track. Did you manage
> to set up the ide integration for the development of Airavata. If you have
> already done that, you can test these features. However I recommend you to
> send pull requests to a separate branch (htcondor?) and then I can provide
> you feedback on the next steps.
>
>
>
> Thanks
>
> Dimuthu
>
>
>
>
>
> On Thu, Oct 1, 2020 at 4:57 PM Suresh Marru  wrote:
>
> Hi Daniel,
>
>
>
> Thanks for this nice summary for keeping the dev list apprised of your
> progress.
>
>
>
> One additional item is to discuss and plan on job monitoring approaches,
> you can look at current references here -
> https://github.com/apache/airavata/tree/master/modules/job-monitor
> 
>
>
>
>
> Suresh
>
>
>
> On Oct 1, 2020, at 4:52 PM, Daniel Shchur 
> wrote:
>
>
>
> Hi all,
>
>
>
> We are working on the HTCondor extension for Airavata. We have gone
> through the code and started to implement additions based on the other job
> workflow systems such as SLURM. We forked
> 
>  the
> project and created a new branch (htcondor) to host our changes. We will
> make a pull request into the main Airavata repository for it once we have
> finished the implementation. A relevant issue has also been created here
> 
> .
>
>
>
> We would like help to know exactly where additions need to be made in case
> we missed a few spots. At this point, we know that:
>
>- A groovy template needs to be made (finished)
>- A job configuration class needs to be made (WIP)
>- A job parser class needs to be made (WIP)
>- And enums need to be modified throughout the project so that systems
>know of HTCondor as an option. (finished)
>
>
>
> If any of you could direct us to the right places, that would be greatly
> appreciated.
>
>
>
> Also, if there’s more documentation or information about the code, such as
> how the command map should work or should be implemented (configuration
> class?), that would be appreciated.
>
>
>
> Thank you!
>
>
>
> - Daniel Shchur
>
>
>
>
>

RE: HTCondor Extension Help

[Daniel Shchur]

Hi Dimuthu and Suresh,

Thanks for getting back to us. We went ahead and made a work in progress pull 
request for our changes we have 
created so far. We also took a look at the job-monitor section and will work 
towards creating additions to support HTCondor.

Do any of you have any suggestions on what the best way is to test submissions 
to HTCondor? We looked into using the Airavata API and the Django portal and 
are curious what you would recommend, and how to do it using your 
recommendation.

Thanks!

- Daniel Shchur

From: DImuthu Upeksha
Sent: Friday, October 2, 2020 5:22 PM
To: Airavata Dev
Cc: daniellthibod...@gmail.com; Evan 
Palmer
Subject: Re: HTCondor Extension Help

Hi Daniel,

This is great progress. I think you are on the right track. Did you manage to 
set up the ide integration for the development of Airavata. If you have already 
done that, you can test these features. However I recommend you to send pull 
requests to a separate branch (htcondor?) and then I can provide you feedback 
on the next steps.

Thanks
Dimuthu


On Thu, Oct 1, 2020 at 4:57 PM Suresh Marru 
mailto:sma...@apache.org>> wrote:
Hi Daniel,

Thanks for this nice summary for keeping the dev list apprised of your progress.

One additional item is to discuss and plan on job monitoring approaches, you 
can look at current references here - 
https://github.com/apache/airavata/tree/master/modules/job-monitor

Suresh


On Oct 1, 2020, at 4:52 PM, Daniel Shchur 
mailto:daniel.shc...@huskers.unl.edu>> wrote:

Hi all,

We are working on the HTCondor extension for Airavata. We have gone through the 
code and started to implement additions based on the other job workflow systems 
such as SLURM. We 
forked
 the project and created a new branch (htcondor) to host our changes. We will 
make a pull request into the main Airavata repository for it once we have 
finished the implementation. A relevant issue has also been created 
here.

We would like help to know exactly where additions need to be made in case we 
missed a few spots. At this point, we know that:

  *   A groovy template needs to be made (finished)
  *   A job configuration class needs to be made (WIP)
  *   A job parser class needs to be made (WIP)
  *   And enums need to be modified throughout the project so that systems know 
of HTCondor as an option. (finished)

If any of you could direct us to the right places, that would be greatly 
appreciated.

Also, if there’s more documentation or information about the code, such as how 
the command map should work or should be implemented (configuration class?), 
that would be appreciated.

Thank you!

- Daniel Shchur

[GitHub] [airavata] evanp18 opened a new pull request #263: [WIP] HTCondor Support Implementation

[GitBox]

evanp18 opened a new pull request #263:
URL: https://github.com/apache/airavata/pull/263


   HTCondor is a framework that enables job submission onto the Open Science 
Grid. Extending the Airavata Helix implementation to incorporate HTCondor as a 
job submission framework will allow Airavata to also submit jobs to the Open 
Science Grid. This pull request includes the development of this extension. The 
tasks to implement this are as follows:
   
   [ ] Integration of an HTCondor Groovy template
   [ ] Integration of an HTCondor output parser class
   [ ] Integration of an HTCondor configuration class
   [ ] Integration of an HTCondor command map
   [ ] Integration of an HTCondor job monitor
   [ ] Unit test implementation where necessary



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org