Re: [DISCUSS] Apache Airavata release 0.17 - RC1

[DImuthu Upeksha]

Hi Suresh,

Build succeeded in JDK 1.8 with Maven 3.5.2 on Mac OS. INSTALL file in the
binary distribution looks little bit out dated.

Thanks
Dimuthu

On Thu, Mar 21, 2019 at 1:07 AM Suresh Marru  wrote:

> Discussion thread for vote on Apache Airavata 0.17 release candidate.
>
> If you have any questions or feedback or to post results of validating the 
> release, please reply to this thread. Once you verify the release, please 
> post your vote to the VOTE thread.
>
> For reference, the Apache release guide  - 
> http://www.apache.org/dev/release.html
>
> Some tips to validate the release before you vote:
>
> * Download the binary version and run the 5 minute or 10 minute tutorial as 
> described in README and website.
> * Download the source files from compressed files and release tag and build 
> (which includes tests).
> * Verify the distribution for the required LICENSE and NOTICE files
> * Verify if all the staged files are signed and the signature is verifiable.
> * Verify if the signing key in the project's KEYS file is hosted on a public 
> server
>
> Thanks for your time in validating the release and voting,
> Suresh
> (On Behalf of Airavata PMC)
>
>
>

Re: Sporadic delays in task execution

[DImuthu Upeksha]

Hi Junkai,

We are using 0.8.1

Dimuthu

On Thu, Mar 21, 2019 at 12:14 AM Xue Junkai  wrote:

> Hi Dimuthu,
>
> What's the version of Helix you are using?
>
> Best,
>
> Junkai
>
> On Wed, Mar 20, 2019 at 8:54 PM DImuthu Upeksha <
> dimuthu.upeks...@gmail.com>
> wrote:
>
> > Hi Helix Dev,
> >
> > We are again seeing this delay in task execution. Please have a look at
> the
> > screencast [1] of logs printed in participant (top shell) and controller
> > (bottom shell). When I record this, there were about 90 - 100 workflows
> > pending to be executed. As you can see some tasks were suddenly executed
> > and then participant freezed for about 30 seconds before executing next
> set
> > of tasks. I can see some WARN logs on controller log. I feel like this 30
> > second delay is some sort of a pattern. What do you think as the reason
> for
> > this? I can provide you more information by turning on verbose logs on
> > controller if you want.
> >
> > [1] https://youtu.be/3EUdSxnIxVw
> >
> > Thanks
> > Dimuthu
> >
> > On Thu, Oct 4, 2018 at 4:46 PM DImuthu Upeksha <
> dimuthu.upeks...@gmail.com
> > >
> > wrote:
> >
> > > Hi Junkai,
> > >
> > > I'm CCing Airavata dev list as this is directly related to the project.
> > >
> > > I just went through the zookeeper path like / Name>/EXTERNALVIEW,
> > > //CONFIGS/RESOURCE as I have noticed that helix
> controller
> > is
> > > periodically monitoring for the children of those paths even though all
> > the
> > > Workflows have moved into a saturated state like COMPLETED and STOPPED.
> > In
> > > our case, we have a lot of completed workflows piled up in those
> paths. I
> > > believe that helix is clearing up those resources after some TTL. What
> I
> > > did was writing an external spectator [1] that continuously monitors
> for
> > > saturated workflows and clearing up resources before controller does
> that
> > > after a TTL. After that, we didn't see such delays in workflow
> execution
> > > and everything seems to be running smoothly. However we are
> continuously
> > > monitoring our deployments for any form of adverse effect introduced by
> > > that improvement.
> > >
> > > Please let us know if we are doing something wrong in this improvement
> or
> > > is there any better way to achieve this directly through helix task
> > > framework.
> > >
> > > [1]
> > >
> >
> https://github.com/apache/airavata/blob/staging/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/controller/WorkflowCleanupAgent.java
> > >
> > > Thanks
> > > Dimuthu
> > >
> > > On Tue, Oct 2, 2018 at 1:12 PM Xue Junkai 
> wrote:
> > >
> > >> Could you please check the log of how long for each pipeline stage
> > takes?
> > >>
> > >> Also, did you set expiry for workflows? Are they piled up for long
> time?
> > >> How long for each workflow completes?
> > >>
> > >> best,
> > >>
> > >> Junkai
> > >>
> > >> On Wed, Sep 26, 2018 at 8:52 AM DImuthu Upeksha <
> > >> dimuthu.upeks...@gmail.com>
> > >> wrote:
> > >>
> > >> > Hi Junkai,
> > >> >
> > >> > Average load is like 10 - 20 workflows per minutes. In some cases
> it's
> > >> less
> > >> > than that However based on the observations, I feel like it does not
> > >> depend
> > >> > on the load and it is sporadic. Is there a particular log lines
> that I
> > >> can
> > >> > filter in controller and participant to capture the timeline of
> > >> workflow so
> > >> > that I can figure out which which component is malfunctioning? We
> use
> > >> helix
> > >> > v 0.8.1.
> > >> >
> > >> > Thanks
> > >> > Dimuthu
> > >> >
> > >> > On Tue, Sep 25, 2018 at 5:19 PM Xue Junkai 
> > >> wrote:
> > >> >
> > >> > > Hi Dimuthu,
> > >> > >
> > >> > > At which rate, you are keep submitting workflows? Usually,
> Workflow
> > >> > > scheduling is very fast. And which version of Helix you are using?
> > >> > >
> > >> > > Best,
> > >> > >
> > >> > > Junkai
> > >> > >
> > >> > > On Tue, Sep 25, 2018 at 8:58 AM DImuthu Upeksha <
> > >> > > dimuthu.upeks...@gmail.com>
> > >> > > wrote:
> > >> > >
> > >> > > > Hi Folks,
> > >> > > >
> > >> > > > We have noticed some delays between workflow submission and
> actual
> > >> > > picking
> > >> > > > up by participants and seems like that delay is somewhat
> constant
> > >> > around
> > >> > > 2-
> > >> > > > 3 minutes. We used to continuously submit workflows and after 2
> -3
> > >> > > minutes,
> > >> > > > a bulk of workflows are picked by participant and execute them.
> > >> Then it
> > >> > > > remain silent for next 2 -3 minutes event we submit more
> > workflows.
> > >> > It's
> > >> > > > like participant picking up workflows in discrete time
> intervals.
> > >> I'm
> > >> > not
> > >> > > > sure whether this is an issue of controller or the participant.
> Do
> > >> you
> > >> > > have
> > >> > > > any experience with this sort of behavior?
> > >> > > >
> > >> > > > Thanks
> > >> > > > Dimuthu
> > >> > > >
> > >> > >
> > >> > >
> > >> > > --
> > >> > > Junkai Xue
> > >> > >
> > >> >
> > >>
> > >>
> 

Re: Removing Stale Branches

[DImuthu Upeksha]

+1

On Thu, Mar 21, 2019 at 1:24 AM Suresh Marru  wrote:

> Hi All,
>
> There are quite a few branches in airavata main repo which seems to be
> definitely not relevant anymore. If we are absolutely sure about them, can
> we remove them?
>
>   origin/AIRAVATA-2500 - This one is merged into master, remove?
>   origin/AIRAVATA-2517-keycloak-mysql-connections - remove?
>   origin/AIRAVATA-2620 - remove?
>   origin/BuildRTWithoutModel - Keep for now
>   origin/airavata-0.15-release-branch - remove?
>
> origin/airavata-2938-change-db-initialization-in-registry-server-to-use-registry-refactoring-code
> - remove?
>   origin/airavata-docker - keep
>   origin/airavata-gov-registry - keep
>   origin/archive - keep
>   origin/auroraMesosIntegration - keep
>   origin/cluster-monitoring - keep
>   origin/data-manager - keep
>   origin/data-model-pre-1.0-discussion - keep
>   origin/deprecated-gfac-modules - keep
>   origin/develop - keep
>   origin/file-management.- keep
>   origin/gfac-storm - keep
>   origin/git-app-catalog - keep
>   origin/group-based-auth - remove since it is merged?
>   origin/grouper-integration - keep
>   origin/helix-integration - remove since it is merged?
>   origin/hotfix-AIRAVATA-2096 - remove?
>   origin/jupyter-integration - keep
>   origin/lahiru/AIRAVATA-2017 - remove?
>   origin/lahiru/AIRAVATA-2065 - remove?
>   origin/lahiru/AIRAVATA-2107 - remove?
>   origin/lahiru/airavata-docker - keep
>   origin/master - keep
>   origin/master-staging-merge - keep
>   origin/mongo-registry - keep
>   origin/new-workflow-design - keep
>   origin/nextcloud - keep
>   origin/orchestratorTaskBreakdown - keep
>   origin/queue-gfac - remove?
>   origin/queue-gfac-rabbitmq - remove?
>   origin/registry-refactoring - remove since it is merged?
>   origin/revert-203-develop - remove
>   origin/shameera/build-fix - remove?
>   origin/staging - keep
>   origin/staging-temp - remove?
>   origin/thrift-0.10.0-upgrade - remove?
>   origin/user-profile - remove since it is merged?
>   origin/workflow-support - keep for now
>
> Thoughts?
> Suresh

Removing Stale Branches

[Suresh Marru]

Hi All,

There are quite a few branches in airavata main repo which seems to be 
definitely not relevant anymore. If we are absolutely sure about them, can we 
remove them? 

  origin/AIRAVATA-2500 - This one is merged into master, remove? 
  origin/AIRAVATA-2517-keycloak-mysql-connections - remove?
  origin/AIRAVATA-2620 - remove? 
  origin/BuildRTWithoutModel - Keep for now
  origin/airavata-0.15-release-branch - remove?
  
origin/airavata-2938-change-db-initialization-in-registry-server-to-use-registry-refactoring-code
 - remove?
  origin/airavata-docker - keep 
  origin/airavata-gov-registry - keep 
  origin/archive - keep 
  origin/auroraMesosIntegration - keep 
  origin/cluster-monitoring - keep 
  origin/data-manager - keep 
  origin/data-model-pre-1.0-discussion - keep
  origin/deprecated-gfac-modules - keep 
  origin/develop - keep
  origin/file-management.- keep
  origin/gfac-storm - keep
  origin/git-app-catalog - keep
  origin/group-based-auth - remove since it is merged?
  origin/grouper-integration - keep
  origin/helix-integration - remove since it is merged? 
  origin/hotfix-AIRAVATA-2096 - remove?
  origin/jupyter-integration - keep
  origin/lahiru/AIRAVATA-2017 - remove?
  origin/lahiru/AIRAVATA-2065 - remove?
  origin/lahiru/AIRAVATA-2107 - remove?
  origin/lahiru/airavata-docker - keep
  origin/master - keep
  origin/master-staging-merge - keep 
  origin/mongo-registry - keep 
  origin/new-workflow-design - keep 
  origin/nextcloud - keep 
  origin/orchestratorTaskBreakdown - keep
  origin/queue-gfac - remove?
  origin/queue-gfac-rabbitmq - remove?
  origin/registry-refactoring - remove since it is merged?
  origin/revert-203-develop - remove
  origin/shameera/build-fix - remove?
  origin/staging - keep
  origin/staging-temp - remove?
  origin/thrift-0.10.0-upgrade - remove?
  origin/user-profile - remove since it is merged?
  origin/workflow-support - keep for now

Thoughts?
Suresh

[VOTE] Apache Airavata release 0.17 - RC1

[Suresh Marru]

Apache Airavata PMC is pleased to call for a vote on the following Apache 
Airavata 0.17 release candidate artifacts:

Detailed change log/release notes:
https://github.com/apache/airavata/blob/airavata-0.17/RELEASE_NOTES 

All Release Artifacts:
https://dist.apache.org/repos/dist/dev/airavata/0.17/RC1/ 


PGP release keys (signed using 617DDBAD):
https://dist.apache.org/repos/dist/dev/airavata/KEYS 


Specific URL’s:

GIT source tag (305cccab1d8eab8aff28e0fa06ebe9f01ffdde2e):
https://github.com/apache/airavata/tree/airavata-0.17 


Source release:
https://dist.apache.org/repos/dist/dev/airavata/0.17/RC1/airavata-0.17-source-release.zip
 


Binary Artifacts:
https://dist.apache.org/repos/dist/dev/airavata/0.17/RC1/apache-airavata-server-0.17-bin.tar.gz
 

 
https://dist.apache.org/repos/dist/dev/airavata/0.17/RC1/apache-airavata-server-0.17-bin.zip
 


Maven staging repo:
https://repository.apache.org/content/repositories/orgapacheairavata-1008/ 


Please verify the artifacts and vote. The vote will be open for 72 hours.

[ ] +1  approve
[ ] +0  no opinion
[ ] -1  disapprove (and reason why)

[DISCUSS] Apache Airavata release 0.17 - RC1

[Suresh Marru]

Discussion thread for vote on Apache Airavata 0.17 release candidate.

If you have any questions or feedback or to post results of validating the 
release, please reply to this thread. Once you verify the release, please post 
your vote to the VOTE thread.  

For reference, the Apache release guide  - 
http://www.apache.org/dev/release.html

Some tips to validate the release before you vote:

* Download the binary version and run the 5 minute or 10 minute tutorial as 
described in README and website.
* Download the source files from compressed files and release tag and build 
(which includes tests). 
* Verify the distribution for the required LICENSE and NOTICE files
* Verify if all the staged files are signed and the signature is verifiable. 
* Verify if the signing key in the project's KEYS file is hosted on a public 
server

Thanks for your time in validating the release and voting,
Suresh 
(On Behalf of Airavata PMC)

Re: Sporadic delays in task execution

[Xue Junkai]

Hi Dimuthu,

What's the version of Helix you are using?

Best,

Junkai

On Wed, Mar 20, 2019 at 8:54 PM DImuthu Upeksha 
wrote:

> Hi Helix Dev,
>
> We are again seeing this delay in task execution. Please have a look at the
> screencast [1] of logs printed in participant (top shell) and controller
> (bottom shell). When I record this, there were about 90 - 100 workflows
> pending to be executed. As you can see some tasks were suddenly executed
> and then participant freezed for about 30 seconds before executing next set
> of tasks. I can see some WARN logs on controller log. I feel like this 30
> second delay is some sort of a pattern. What do you think as the reason for
> this? I can provide you more information by turning on verbose logs on
> controller if you want.
>
> [1] https://youtu.be/3EUdSxnIxVw
>
> Thanks
> Dimuthu
>
> On Thu, Oct 4, 2018 at 4:46 PM DImuthu Upeksha  >
> wrote:
>
> > Hi Junkai,
> >
> > I'm CCing Airavata dev list as this is directly related to the project.
> >
> > I just went through the zookeeper path like //EXTERNALVIEW,
> > //CONFIGS/RESOURCE as I have noticed that helix controller
> is
> > periodically monitoring for the children of those paths even though all
> the
> > Workflows have moved into a saturated state like COMPLETED and STOPPED.
> In
> > our case, we have a lot of completed workflows piled up in those paths. I
> > believe that helix is clearing up those resources after some TTL. What I
> > did was writing an external spectator [1] that continuously monitors for
> > saturated workflows and clearing up resources before controller does that
> > after a TTL. After that, we didn't see such delays in workflow execution
> > and everything seems to be running smoothly. However we are continuously
> > monitoring our deployments for any form of adverse effect introduced by
> > that improvement.
> >
> > Please let us know if we are doing something wrong in this improvement or
> > is there any better way to achieve this directly through helix task
> > framework.
> >
> > [1]
> >
> https://github.com/apache/airavata/blob/staging/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/controller/WorkflowCleanupAgent.java
> >
> > Thanks
> > Dimuthu
> >
> > On Tue, Oct 2, 2018 at 1:12 PM Xue Junkai  wrote:
> >
> >> Could you please check the log of how long for each pipeline stage
> takes?
> >>
> >> Also, did you set expiry for workflows? Are they piled up for long time?
> >> How long for each workflow completes?
> >>
> >> best,
> >>
> >> Junkai
> >>
> >> On Wed, Sep 26, 2018 at 8:52 AM DImuthu Upeksha <
> >> dimuthu.upeks...@gmail.com>
> >> wrote:
> >>
> >> > Hi Junkai,
> >> >
> >> > Average load is like 10 - 20 workflows per minutes. In some cases it's
> >> less
> >> > than that However based on the observations, I feel like it does not
> >> depend
> >> > on the load and it is sporadic. Is there a particular log lines that I
> >> can
> >> > filter in controller and participant to capture the timeline of
> >> workflow so
> >> > that I can figure out which which component is malfunctioning? We use
> >> helix
> >> > v 0.8.1.
> >> >
> >> > Thanks
> >> > Dimuthu
> >> >
> >> > On Tue, Sep 25, 2018 at 5:19 PM Xue Junkai 
> >> wrote:
> >> >
> >> > > Hi Dimuthu,
> >> > >
> >> > > At which rate, you are keep submitting workflows? Usually, Workflow
> >> > > scheduling is very fast. And which version of Helix you are using?
> >> > >
> >> > > Best,
> >> > >
> >> > > Junkai
> >> > >
> >> > > On Tue, Sep 25, 2018 at 8:58 AM DImuthu Upeksha <
> >> > > dimuthu.upeks...@gmail.com>
> >> > > wrote:
> >> > >
> >> > > > Hi Folks,
> >> > > >
> >> > > > We have noticed some delays between workflow submission and actual
> >> > > picking
> >> > > > up by participants and seems like that delay is somewhat constant
> >> > around
> >> > > 2-
> >> > > > 3 minutes. We used to continuously submit workflows and after 2 -3
> >> > > minutes,
> >> > > > a bulk of workflows are picked by participant and execute them.
> >> Then it
> >> > > > remain silent for next 2 -3 minutes event we submit more
> workflows.
> >> > It's
> >> > > > like participant picking up workflows in discrete time intervals.
> >> I'm
> >> > not
> >> > > > sure whether this is an issue of controller or the participant. Do
> >> you
> >> > > have
> >> > > > any experience with this sort of behavior?
> >> > > >
> >> > > > Thanks
> >> > > > Dimuthu
> >> > > >
> >> > >
> >> > >
> >> > > --
> >> > > Junkai Xue
> >> > >
> >> >
> >>
> >>
> >> --
> >> Junkai Xue
> >>
> >
>


-- 
Junkai Xue

Re: Sporadic delays in task execution

[DImuthu Upeksha]

Hi Helix Dev,

We are again seeing this delay in task execution. Please have a look at the
screencast [1] of logs printed in participant (top shell) and controller
(bottom shell). When I record this, there were about 90 - 100 workflows
pending to be executed. As you can see some tasks were suddenly executed
and then participant freezed for about 30 seconds before executing next set
of tasks. I can see some WARN logs on controller log. I feel like this 30
second delay is some sort of a pattern. What do you think as the reason for
this? I can provide you more information by turning on verbose logs on
controller if you want.

[1] https://youtu.be/3EUdSxnIxVw

Thanks
Dimuthu

On Thu, Oct 4, 2018 at 4:46 PM DImuthu Upeksha 
wrote:

> Hi Junkai,
>
> I'm CCing Airavata dev list as this is directly related to the project.
>
> I just went through the zookeeper path like //EXTERNALVIEW,
> //CONFIGS/RESOURCE as I have noticed that helix controller is
> periodically monitoring for the children of those paths even though all the
> Workflows have moved into a saturated state like COMPLETED and STOPPED. In
> our case, we have a lot of completed workflows piled up in those paths. I
> believe that helix is clearing up those resources after some TTL. What I
> did was writing an external spectator [1] that continuously monitors for
> saturated workflows and clearing up resources before controller does that
> after a TTL. After that, we didn't see such delays in workflow execution
> and everything seems to be running smoothly. However we are continuously
> monitoring our deployments for any form of adverse effect introduced by
> that improvement.
>
> Please let us know if we are doing something wrong in this improvement or
> is there any better way to achieve this directly through helix task
> framework.
>
> [1]
> https://github.com/apache/airavata/blob/staging/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/controller/WorkflowCleanupAgent.java
>
> Thanks
> Dimuthu
>
> On Tue, Oct 2, 2018 at 1:12 PM Xue Junkai  wrote:
>
>> Could you please check the log of how long for each pipeline stage takes?
>>
>> Also, did you set expiry for workflows? Are they piled up for long time?
>> How long for each workflow completes?
>>
>> best,
>>
>> Junkai
>>
>> On Wed, Sep 26, 2018 at 8:52 AM DImuthu Upeksha <
>> dimuthu.upeks...@gmail.com>
>> wrote:
>>
>> > Hi Junkai,
>> >
>> > Average load is like 10 - 20 workflows per minutes. In some cases it's
>> less
>> > than that However based on the observations, I feel like it does not
>> depend
>> > on the load and it is sporadic. Is there a particular log lines that I
>> can
>> > filter in controller and participant to capture the timeline of
>> workflow so
>> > that I can figure out which which component is malfunctioning? We use
>> helix
>> > v 0.8.1.
>> >
>> > Thanks
>> > Dimuthu
>> >
>> > On Tue, Sep 25, 2018 at 5:19 PM Xue Junkai 
>> wrote:
>> >
>> > > Hi Dimuthu,
>> > >
>> > > At which rate, you are keep submitting workflows? Usually, Workflow
>> > > scheduling is very fast. And which version of Helix you are using?
>> > >
>> > > Best,
>> > >
>> > > Junkai
>> > >
>> > > On Tue, Sep 25, 2018 at 8:58 AM DImuthu Upeksha <
>> > > dimuthu.upeks...@gmail.com>
>> > > wrote:
>> > >
>> > > > Hi Folks,
>> > > >
>> > > > We have noticed some delays between workflow submission and actual
>> > > picking
>> > > > up by participants and seems like that delay is somewhat constant
>> > around
>> > > 2-
>> > > > 3 minutes. We used to continuously submit workflows and after 2 -3
>> > > minutes,
>> > > > a bulk of workflows are picked by participant and execute them.
>> Then it
>> > > > remain silent for next 2 -3 minutes event we submit more workflows.
>> > It's
>> > > > like participant picking up workflows in discrete time intervals.
>> I'm
>> > not
>> > > > sure whether this is an issue of controller or the participant. Do
>> you
>> > > have
>> > > > any experience with this sort of behavior?
>> > > >
>> > > > Thanks
>> > > > Dimuthu
>> > > >
>> > >
>> > >
>> > > --
>> > > Junkai Xue
>> > >
>> >
>>
>>
>> --
>> Junkai Xue
>>
>