Re: Sporadic delays in task execution

[Hunter Lee]

Let me add a caveat to my previous email. Although it comes with
scalability improvements, there are currently a few known issues with the
latest version. We'd encourage you to check back to make sure your current
usage isn't affected.

Hunter

On Fri, Mar 22, 2019 at 12:35 PM Hunter Lee  wrote:

> No problem. If you have further questions, let us know what kind of load
> you're putting on Helix as well. The newest version of Helix contains Task
> Framework 2.0, and has greater scalability in scheduling tasks, so you
> might want to consider using the newest version as well.
>
> Hunter
>
> On Fri, Mar 22, 2019 at 8:59 AM DImuthu Upeksha <
> dimuthu.upeks...@gmail.com> wrote:
>
>> Hi Lee,
>>
>> Thanks for the trick. I didn't know that we can poke the controller like
>> that :) However now we can see that tasks are moving smoothly in our
>> staging setup. This behavior can be seen from time to time and get
>> resolved
>> automatically in few hours. I can't find a particular pattern however my
>> best guess is that this happens when the load is high. I will put some
>> load
>> on testing setup and see if I can reproduce this issue and try your
>> instructions then get back to you
>>
>> Thanks
>> Dimuthu
>>
>> On Thu, Mar 21, 2019 at 5:27 PM Hunter Lee  wrote:
>>
>> > Hi Dimuthu,
>> >
>> > What Junkai meant by touching the IdealState is this:
>> >
>> > 1) use Zooinspector to log into ZK
>> > 2) Locate the IDEALSTATES/ path
>> > 3) grab any ZNode under that path and try to modify (just add a
>> > whitespace) and save
>> > 4) This will trigger a ZK callback which should tell Helix Controller to
>> > rebalance/schedule things
>> >
>> > On Thu, Mar 21, 2019 at 11:30 AM DImuthu Upeksha <
>> > dimuthu.upeks...@gmail.com> wrote:
>> >
>> >> Hi Junkai,
>> >>
>> >> What do you mean by touching ideal state to trigger an event? I didn't
>> >> quite get what you said. Is that like creating some path in zookeeper?
>> >> Workflows are eventually scheduled but the problem is, it is very slow
>> due
>> >> to that 30s freeze.
>> >>
>> >> Thanks
>> >> Dimuthu
>> >>
>> >> On Thu, Mar 21, 2019 at 2:26 PM Xue Junkai 
>> wrote:
>> >>
>> >> > Can you try one thing? Touch the ideal state to trigger an event. If
>> >> > workflows are not scheduled, it should scheduling has problem.
>> >> >
>> >> > Best,
>> >> >
>> >> > Junkai
>> >> >
>> >> > On Wed, Mar 20, 2019 at 10:31 PM DImuthu Upeksha <
>> >> > dimuthu.upeks...@gmail.com> wrote:
>> >> >
>> >> >> Hi Junkai,
>> >> >>
>> >> >> We are using 0.8.1
>> >> >>
>> >> >> Dimuthu
>> >> >>
>> >> >> On Thu, Mar 21, 2019 at 12:14 AM Xue Junkai 
>> >> wrote:
>> >> >>
>> >> >> > Hi Dimuthu,
>> >> >> >
>> >> >> > What's the version of Helix you are using?
>> >> >> >
>> >> >> > Best,
>> >> >> >
>> >> >> > Junkai
>> >> >> >
>> >> >> > On Wed, Mar 20, 2019 at 8:54 PM DImuthu Upeksha <
>> >> >> > dimuthu.upeks...@gmail.com>
>> >> >> > wrote:
>> >> >> >
>> >> >> > > Hi Helix Dev,
>> >> >> > >
>> >> >> > > We are again seeing this delay in task execution. Please have a
>> >> look
>> >> >> at
>> >> >> > the
>> >> >> > > screencast [1] of logs printed in participant (top shell) and
>> >> >> controller
>> >> >> > > (bottom shell). When I record this, there were about 90 - 100
>> >> >> workflows
>> >> >> > > pending to be executed. As you can see some tasks were suddenly
>> >> >> executed
>> >> >> > > and then participant freezed for about 30 seconds before
>> executing
>> >> >> next
>> >> >> > set
>> >> >> > > of tasks. I can see some WARN logs on controller log. I feel
>> like
>> >> >> this 30
>> >> >> > > second delay is some sort of a pattern. What do you think as the
>> >> >> reason
>> >> >> > for
>> >> >> > > this? I can provide you more information by turning on verbose
>> >> logs on
>> >> >> > > controller if you want.
>> >> >> > >
>> >> >> > > [1] https://youtu.be/3EUdSxnIxVw
>> >> >> > >
>> >> >> > > Thanks
>> >> >> > > Dimuthu
>> >> >> > >
>> >> >> > > On Thu, Oct 4, 2018 at 4:46 PM DImuthu Upeksha <
>> >> >> > dimuthu.upeks...@gmail.com
>> >> >> > > >
>> >> >> > > wrote:
>> >> >> > >
>> >> >> > > > Hi Junkai,
>> >> >> > > >
>> >> >> > > > I'm CCing Airavata dev list as this is directly related to the
>> >> >> project.
>> >> >> > > >
>> >> >> > > > I just went through the zookeeper path like /> >> >> > Name>/EXTERNALVIEW,
>> >> >> > > > //CONFIGS/RESOURCE as I have noticed that helix
>> >> >> > controller
>> >> >> > > is
>> >> >> > > > periodically monitoring for the children of those paths even
>> >> though
>> >> >> all
>> >> >> > > the
>> >> >> > > > Workflows have moved into a saturated state like COMPLETED and
>> >> >> STOPPED.
>> >> >> > > In
>> >> >> > > > our case, we have a lot of completed workflows piled up in
>> those
>> >> >> > paths. I
>> >> >> > > > believe that helix is clearing up those resources after some
>> TTL.
>> >> >> What
>> >> >> > I
>> >> >> > > > did was writing an external spectator [1] that continuously
>> >> monitors
>> >> >> > for
>> >> >> > > > saturated workflows 

Re: Sporadic delays in task execution

[Hunter Lee]

No problem. If you have further questions, let us know what kind of load
you're putting on Helix as well. The newest version of Helix contains Task
Framework 2.0, and has greater scalability in scheduling tasks, so you
might want to consider using the newest version as well.

Hunter

On Fri, Mar 22, 2019 at 8:59 AM DImuthu Upeksha 
wrote:

> Hi Lee,
>
> Thanks for the trick. I didn't know that we can poke the controller like
> that :) However now we can see that tasks are moving smoothly in our
> staging setup. This behavior can be seen from time to time and get resolved
> automatically in few hours. I can't find a particular pattern however my
> best guess is that this happens when the load is high. I will put some load
> on testing setup and see if I can reproduce this issue and try your
> instructions then get back to you
>
> Thanks
> Dimuthu
>
> On Thu, Mar 21, 2019 at 5:27 PM Hunter Lee  wrote:
>
> > Hi Dimuthu,
> >
> > What Junkai meant by touching the IdealState is this:
> >
> > 1) use Zooinspector to log into ZK
> > 2) Locate the IDEALSTATES/ path
> > 3) grab any ZNode under that path and try to modify (just add a
> > whitespace) and save
> > 4) This will trigger a ZK callback which should tell Helix Controller to
> > rebalance/schedule things
> >
> > On Thu, Mar 21, 2019 at 11:30 AM DImuthu Upeksha <
> > dimuthu.upeks...@gmail.com> wrote:
> >
> >> Hi Junkai,
> >>
> >> What do you mean by touching ideal state to trigger an event? I didn't
> >> quite get what you said. Is that like creating some path in zookeeper?
> >> Workflows are eventually scheduled but the problem is, it is very slow
> due
> >> to that 30s freeze.
> >>
> >> Thanks
> >> Dimuthu
> >>
> >> On Thu, Mar 21, 2019 at 2:26 PM Xue Junkai 
> wrote:
> >>
> >> > Can you try one thing? Touch the ideal state to trigger an event. If
> >> > workflows are not scheduled, it should scheduling has problem.
> >> >
> >> > Best,
> >> >
> >> > Junkai
> >> >
> >> > On Wed, Mar 20, 2019 at 10:31 PM DImuthu Upeksha <
> >> > dimuthu.upeks...@gmail.com> wrote:
> >> >
> >> >> Hi Junkai,
> >> >>
> >> >> We are using 0.8.1
> >> >>
> >> >> Dimuthu
> >> >>
> >> >> On Thu, Mar 21, 2019 at 12:14 AM Xue Junkai 
> >> wrote:
> >> >>
> >> >> > Hi Dimuthu,
> >> >> >
> >> >> > What's the version of Helix you are using?
> >> >> >
> >> >> > Best,
> >> >> >
> >> >> > Junkai
> >> >> >
> >> >> > On Wed, Mar 20, 2019 at 8:54 PM DImuthu Upeksha <
> >> >> > dimuthu.upeks...@gmail.com>
> >> >> > wrote:
> >> >> >
> >> >> > > Hi Helix Dev,
> >> >> > >
> >> >> > > We are again seeing this delay in task execution. Please have a
> >> look
> >> >> at
> >> >> > the
> >> >> > > screencast [1] of logs printed in participant (top shell) and
> >> >> controller
> >> >> > > (bottom shell). When I record this, there were about 90 - 100
> >> >> workflows
> >> >> > > pending to be executed. As you can see some tasks were suddenly
> >> >> executed
> >> >> > > and then participant freezed for about 30 seconds before
> executing
> >> >> next
> >> >> > set
> >> >> > > of tasks. I can see some WARN logs on controller log. I feel like
> >> >> this 30
> >> >> > > second delay is some sort of a pattern. What do you think as the
> >> >> reason
> >> >> > for
> >> >> > > this? I can provide you more information by turning on verbose
> >> logs on
> >> >> > > controller if you want.
> >> >> > >
> >> >> > > [1] https://youtu.be/3EUdSxnIxVw
> >> >> > >
> >> >> > > Thanks
> >> >> > > Dimuthu
> >> >> > >
> >> >> > > On Thu, Oct 4, 2018 at 4:46 PM DImuthu Upeksha <
> >> >> > dimuthu.upeks...@gmail.com
> >> >> > > >
> >> >> > > wrote:
> >> >> > >
> >> >> > > > Hi Junkai,
> >> >> > > >
> >> >> > > > I'm CCing Airavata dev list as this is directly related to the
> >> >> project.
> >> >> > > >
> >> >> > > > I just went through the zookeeper path like / >> >> > Name>/EXTERNALVIEW,
> >> >> > > > //CONFIGS/RESOURCE as I have noticed that helix
> >> >> > controller
> >> >> > > is
> >> >> > > > periodically monitoring for the children of those paths even
> >> though
> >> >> all
> >> >> > > the
> >> >> > > > Workflows have moved into a saturated state like COMPLETED and
> >> >> STOPPED.
> >> >> > > In
> >> >> > > > our case, we have a lot of completed workflows piled up in
> those
> >> >> > paths. I
> >> >> > > > believe that helix is clearing up those resources after some
> TTL.
> >> >> What
> >> >> > I
> >> >> > > > did was writing an external spectator [1] that continuously
> >> monitors
> >> >> > for
> >> >> > > > saturated workflows and clearing up resources before controller
> >> does
> >> >> > that
> >> >> > > > after a TTL. After that, we didn't see such delays in workflow
> >> >> > execution
> >> >> > > > and everything seems to be running smoothly. However we are
> >> >> > continuously
> >> >> > > > monitoring our deployments for any form of adverse effect
> >> >> introduced by
> >> >> > > > that improvement.
> >> >> > > >
> >> >> > > > Please let us know if we are doing something wrong in this
> >> >> 

Re: [VOTE] Apache Airavata release 0.17 - RC1

[Pierce, Marlon]

+1

 

From: "dimuthu.upeks...@gmail.com" 
Reply-To: dev 
Date: Thursday, March 21, 2019 at 9:33 AM
To: dev 
Subject: Re: [VOTE] Apache Airavata release 0.17 - RC1

 

+1

 

On Thu, Mar 21, 2019 at 1:07 AM Suresh Marru  wrote:

Apache Airavata PMC is pleased to call for a vote on the following Apache 
Airavata 0.17 release candidate artifacts:
 
Detailed change log/release notes:
https://github.com/apache/airavata/blob/airavata-0.17/RELEASE_NOTES
All Release Artifacts:
https://dist.apache.org/repos/dist/dev/airavata/0.17/RC1/
PGP release keys (signed using 617DDBAD):
https://dist.apache.org/repos/dist/dev/airavata/KEYS
Specific URL’s:
GIT source tag (305cccab1d8eab8aff28e0fa06ebe9f01ffdde2e):
https://github.com/apache/airavata/tree/airavata-0.17
Source release:
https://dist.apache.org/repos/dist/dev/airavata/0.17/RC1/airavata-0.17-source-release.zip
Binary Artifacts:
https://dist.apache.org/repos/dist/dev/airavata/0.17/RC1/apache-airavata-server-0.17-bin.tar.gz
 
https://dist.apache.org/repos/dist/dev/airavata/0.17/RC1/apache-airavata-server-0.17-bin.zip
Maven staging repo:
https://repository.apache.org/content/repositories/orgapacheairavata-1008/
 
Please verify the artifacts and vote. The vote will be open for 72 hours.
 
[ ] +1  approve
[ ] +0  no opinion
[ ] -1  disapprove (and reason why)
 



smime.p7s
Description: S/MIME cryptographic signature

Re: [DISCUSS] Apache Airavata release 0.17 - RC1

[Pierce, Marlon]

I verified the release signatures and built the code on Mac OSX 10.14.x with 
Java 1.8 and Maven 3.6, so I’m +1 for the release, voted on the [VOTE) thread.

 

Marlon

 

 

From: "dimuthu.upeks...@gmail.com" 
Reply-To: dev 
Date: Thursday, March 21, 2019 at 1:55 AM
To: dev 
Subject: Re: [DISCUSS] Apache Airavata release 0.17 - RC1

 

Hi Suresh, 

 

Build succeeded in JDK 1.8 with Maven 3.5.2 on Mac OS. INSTALL file in the 
binary distribution looks little bit out dated. 

 

Thanks

Dimuthu

 

On Thu, Mar 21, 2019 at 1:07 AM Suresh Marru  wrote:

Discussion thread for vote on Apache Airavata 0.17 release candidate.
 
If you have any questions or feedback or to post results of validating the 
release, please reply to this thread. Once you verify the release, please post 
your vote to the VOTE thread.  
 
For reference, the Apache release guide  - 
http://www.apache.org/dev/release.html
 
Some tips to validate the release before you vote:
 
* Download the binary version and run the 5 minute or 10 minute tutorial as 
described in README and website.
* Download the source files from compressed files and release tag and build 
(which includes tests). 
* Verify the distribution for the required LICENSE and NOTICE files
* Verify if all the staged files are signed and the signature is verifiable. 
* Verify if the signing key in the project's KEYS file is hosted on a public 
server
 
Thanks for your time in validating the release and voting,
Suresh 
(On Behalf of Airavata PMC)
 



smime.p7s
Description: S/MIME cryptographic signature

Re: [VOTE] Apache Airavata release 0.17 - RC1

[Christie, Marcus Aaron]

> On Mar 21, 2019, at 1:07 AM, Suresh Marru  wrote:
> 
> Apache Airavata PMC is pleased to call for a vote on the following Apache 
> Airavata 0.17 release candidate artifacts:
> 

+1

smime.p7s
Description: S/MIME cryptographic signature

Re: [DISCUSS] Apache Airavata release 0.17 - RC1

[Suresh Marru]

Hi Marcus,

Thanks for the due diligence. I ignored to update the binary files, to avoid 
this in the future, I created a JIRA to maintain a single file for these and 
clearly distinguish the source and binary information in the single file. I 
blocked to change this before next release - 
https://issues.apache.org/jira/browse/AIRAVATA-3001 


Cheers,
Suresh

> On Mar 22, 2019, at 2:07 PM, Christie, Marcus Aaron  wrote:
> 
> 
> 
>> On Mar 21, 2019, at 1:54 AM, DImuthu Upeksha > > wrote:
>> 
>> Build succeeded in JDK 1.8 with Maven 3.5.2 on Mac OS.
> 
> Build succeeded for me too. I deleted my ~/.m2/ directory to make sure the 
> maven dependencies are there too.
> 
> $ mvn -version
> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 
> 2015-11-10T11:41:47-05:00)
> Maven home: /Users/machrist/lib/maven
> Java version: 1.8.0_121, vendor: Oracle Corporation
> Java home: 
> /Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "mac os x", version: "10.14.3", arch: "x86_64", family: "mac"

Re: [DISCUSS] Apache Airavata release 0.17 - RC1

[Christie, Marcus Aaron]

> On Mar 21, 2019, at 1:54 AM, DImuthu Upeksha  
> wrote:
> 
> Build succeeded in JDK 1.8 with Maven 3.5.2 on Mac OS.

Build succeeded for me too. I deleted my ~/.m2/ directory to make sure the 
maven dependencies are there too.

$ mvn -version
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 
2015-11-10T11:41:47-05:00)
Maven home: /Users/machrist/lib/maven
Java version: 1.8.0_121, vendor: Oracle Corporation
Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "10.14.3", arch: "x86_64", family: "mac"

smime.p7s
Description: S/MIME cryptographic signature

Re: [DISCUSS] Apache Airavata release 0.17 - RC1

[Christie, Marcus Aaron]

Hi Suresh,

First of all, major kudos on making this release!

Here's my feedback, I don't think any of these are blockers

- INSTALL says version "0.14"
- NOTICE says "Apache Airavata
Copyright 2014 The Apache Software Foundation". Should that be Copyright 
2014-2019? [1]

Thanks for the signature verification notes. Regarding that, I had to install 
the signing key

gpg --keyserver pgpkeys.mit.edu --recv-key 617DDBAD

Then I was able to verify the signatures.


Thanks,

Marcus


[1] http://www.apache.org/legal/src-headers.html#notice

> On Mar 22, 2019, at 12:03 PM, Suresh Marru  wrote:
> 
> Hi All,
> 
> If you are looking to verify release signatures, here is a quick tip:
> 
> * download all release artifacts into a folder or do a svn checkout of the 
> release - `svn co https://dist.apache.org/repos/dist/dev/airavata/0.17/RC1/ 
> `
> 
> * Verify signatures by a small script something like:
> 
> for file in `find . -type f -iname '*.asc'`
>   do
>   gpg --verify ${file} 
>   done
> The output will indicate the You'll need to look at the output to ensure it 
> contains only good signatures -
> 
> gpg: Good signature from ... gpg: Signature made …
> 
> Once you verify the signature, next step is to uncompress the source and 
> verify it builds fine. 
> 
> This links provide guidance on release verifications:
> 
> https://www.apache.org/info/verification 
> 
> 
> Cheers,
> Suresh
> 
>> On Mar 21, 2019, at 1:07 AM, Suresh Marru > > wrote:
>> 
>> Discussion thread for vote on Apache Airavata 0.17 release candidate.
>> 
>> If you have any questions or feedback or to post results of validating the 
>> release, please reply to this thread. Once you verify the release, please 
>> post your vote to the VOTE thread.  
>> 
>> For reference, the Apache release guide  - 
>> http://www.apache.org/dev/release.html 
>> 
>> 
>> Some tips to validate the release before you vote:
>> 
>> * Download the binary version and run the 5 minute or 10 minute tutorial as 
>> described in README and website.
>> * Download the source files from compressed files and release tag and build 
>> (which includes tests). 
>> * Verify the distribution for the required LICENSE and NOTICE files
>> * Verify if all the staged files are signed and the signature is verifiable. 
>> * Verify if the signing key in the project's KEYS file is hosted on a public 
>> server
>> 
>> Thanks for your time in validating the release and voting,
>> Suresh 
>> (On Behalf of Airavata PMC)
>> 
> 



smime.p7s
Description: S/MIME cryptographic signature

Re: Removing Stale Branches

[Suresh Marru]

Thanks Dimuthu, Marcus for reviewing. I removed the ones which were marked 
remove below. 

Suresh

> On Mar 22, 2019, at 9:20 AM, Christie, Marcus Aaron  wrote:
> 
> +1
> 
> I don't see any that you've marked with remove that need to be kept.
> 
>> On Mar 21, 2019, at 1:23 AM, Suresh Marru  wrote:
>> 
>> Hi All,
>> 
>> There are quite a few branches in airavata main repo which seems to be 
>> definitely not relevant anymore. If we are absolutely sure about them, can 
>> we remove them? 
>> 
>> origin/AIRAVATA-2500 - This one is merged into master, remove? 
>> origin/AIRAVATA-2517-keycloak-mysql-connections - remove?
>> origin/AIRAVATA-2620 - remove? 
>> origin/BuildRTWithoutModel - Keep for now
>> origin/airavata-0.15-release-branch - remove?
>> origin/airavata-2938-change-db-initialization-in-registry-server-to-use-registry-refactoring-code
>>  - remove?
>> origin/airavata-docker - keep 
>> origin/airavata-gov-registry - keep 
>> origin/archive - keep 
>> origin/auroraMesosIntegration - keep 
>> origin/cluster-monitoring - keep 
>> origin/data-manager - keep 
>> origin/data-model-pre-1.0-discussion - keep
>> origin/deprecated-gfac-modules - keep 
>> origin/develop - keep
>> origin/file-management.- keep
>> origin/gfac-storm - keep
>> origin/git-app-catalog - keep
>> origin/group-based-auth - remove since it is merged?
>> origin/grouper-integration - keep
>> origin/helix-integration - remove since it is merged? 
>> origin/hotfix-AIRAVATA-2096 - remove?
>> origin/jupyter-integration - keep
>> origin/lahiru/AIRAVATA-2017 - remove?
>> origin/lahiru/AIRAVATA-2065 - remove?
>> origin/lahiru/AIRAVATA-2107 - remove?
>> origin/lahiru/airavata-docker - keep
>> origin/master - keep
>> origin/master-staging-merge - keep 
>> origin/mongo-registry - keep 
>> origin/new-workflow-design - keep 
>> origin/nextcloud - keep 
>> origin/orchestratorTaskBreakdown - keep
>> origin/queue-gfac - remove?
>> origin/queue-gfac-rabbitmq - remove?
>> origin/registry-refactoring - remove since it is merged?
>> origin/revert-203-develop - remove
>> origin/shameera/build-fix - remove?
>> origin/staging - keep
>> origin/staging-temp - remove?
>> origin/thrift-0.10.0-upgrade - remove?
>> origin/user-profile - remove since it is merged?
>> origin/workflow-support - keep for now
>> 
>> Thoughts?
>> Suresh
> 

Re: Sporadic delays in task execution

[DImuthu Upeksha]

Hi Lee,

Thanks for the trick. I didn't know that we can poke the controller like
that :) However now we can see that tasks are moving smoothly in our
staging setup. This behavior can be seen from time to time and get resolved
automatically in few hours. I can't find a particular pattern however my
best guess is that this happens when the load is high. I will put some load
on testing setup and see if I can reproduce this issue and try your
instructions then get back to you

Thanks
Dimuthu

On Thu, Mar 21, 2019 at 5:27 PM Hunter Lee  wrote:

> Hi Dimuthu,
>
> What Junkai meant by touching the IdealState is this:
>
> 1) use Zooinspector to log into ZK
> 2) Locate the IDEALSTATES/ path
> 3) grab any ZNode under that path and try to modify (just add a
> whitespace) and save
> 4) This will trigger a ZK callback which should tell Helix Controller to
> rebalance/schedule things
>
> On Thu, Mar 21, 2019 at 11:30 AM DImuthu Upeksha <
> dimuthu.upeks...@gmail.com> wrote:
>
>> Hi Junkai,
>>
>> What do you mean by touching ideal state to trigger an event? I didn't
>> quite get what you said. Is that like creating some path in zookeeper?
>> Workflows are eventually scheduled but the problem is, it is very slow due
>> to that 30s freeze.
>>
>> Thanks
>> Dimuthu
>>
>> On Thu, Mar 21, 2019 at 2:26 PM Xue Junkai  wrote:
>>
>> > Can you try one thing? Touch the ideal state to trigger an event. If
>> > workflows are not scheduled, it should scheduling has problem.
>> >
>> > Best,
>> >
>> > Junkai
>> >
>> > On Wed, Mar 20, 2019 at 10:31 PM DImuthu Upeksha <
>> > dimuthu.upeks...@gmail.com> wrote:
>> >
>> >> Hi Junkai,
>> >>
>> >> We are using 0.8.1
>> >>
>> >> Dimuthu
>> >>
>> >> On Thu, Mar 21, 2019 at 12:14 AM Xue Junkai 
>> wrote:
>> >>
>> >> > Hi Dimuthu,
>> >> >
>> >> > What's the version of Helix you are using?
>> >> >
>> >> > Best,
>> >> >
>> >> > Junkai
>> >> >
>> >> > On Wed, Mar 20, 2019 at 8:54 PM DImuthu Upeksha <
>> >> > dimuthu.upeks...@gmail.com>
>> >> > wrote:
>> >> >
>> >> > > Hi Helix Dev,
>> >> > >
>> >> > > We are again seeing this delay in task execution. Please have a
>> look
>> >> at
>> >> > the
>> >> > > screencast [1] of logs printed in participant (top shell) and
>> >> controller
>> >> > > (bottom shell). When I record this, there were about 90 - 100
>> >> workflows
>> >> > > pending to be executed. As you can see some tasks were suddenly
>> >> executed
>> >> > > and then participant freezed for about 30 seconds before executing
>> >> next
>> >> > set
>> >> > > of tasks. I can see some WARN logs on controller log. I feel like
>> >> this 30
>> >> > > second delay is some sort of a pattern. What do you think as the
>> >> reason
>> >> > for
>> >> > > this? I can provide you more information by turning on verbose
>> logs on
>> >> > > controller if you want.
>> >> > >
>> >> > > [1] https://youtu.be/3EUdSxnIxVw
>> >> > >
>> >> > > Thanks
>> >> > > Dimuthu
>> >> > >
>> >> > > On Thu, Oct 4, 2018 at 4:46 PM DImuthu Upeksha <
>> >> > dimuthu.upeks...@gmail.com
>> >> > > >
>> >> > > wrote:
>> >> > >
>> >> > > > Hi Junkai,
>> >> > > >
>> >> > > > I'm CCing Airavata dev list as this is directly related to the
>> >> project.
>> >> > > >
>> >> > > > I just went through the zookeeper path like /> >> > Name>/EXTERNALVIEW,
>> >> > > > //CONFIGS/RESOURCE as I have noticed that helix
>> >> > controller
>> >> > > is
>> >> > > > periodically monitoring for the children of those paths even
>> though
>> >> all
>> >> > > the
>> >> > > > Workflows have moved into a saturated state like COMPLETED and
>> >> STOPPED.
>> >> > > In
>> >> > > > our case, we have a lot of completed workflows piled up in those
>> >> > paths. I
>> >> > > > believe that helix is clearing up those resources after some TTL.
>> >> What
>> >> > I
>> >> > > > did was writing an external spectator [1] that continuously
>> monitors
>> >> > for
>> >> > > > saturated workflows and clearing up resources before controller
>> does
>> >> > that
>> >> > > > after a TTL. After that, we didn't see such delays in workflow
>> >> > execution
>> >> > > > and everything seems to be running smoothly. However we are
>> >> > continuously
>> >> > > > monitoring our deployments for any form of adverse effect
>> >> introduced by
>> >> > > > that improvement.
>> >> > > >
>> >> > > > Please let us know if we are doing something wrong in this
>> >> improvement
>> >> > or
>> >> > > > is there any better way to achieve this directly through helix
>> task
>> >> > > > framework.
>> >> > > >
>> >> > > > [1]
>> >> > > >
>> >> > >
>> >> >
>> >>
>> https://github.com/apache/airavata/blob/staging/modules/airavata-helix/helix-spectator/src/main/java/org/apache/airavata/helix/impl/controller/WorkflowCleanupAgent.java
>> >> > > >
>> >> > > > Thanks
>> >> > > > Dimuthu
>> >> > > >
>> >> > > > On Tue, Oct 2, 2018 at 1:12 PM Xue Junkai 
>> >> > wrote:
>> >> > > >
>> >> > > >> Could you please check the log of how long for each pipeline
>> stage
>> >> > > takes?
>> >> > > >>
>> >> > > >> Also, 

Re: [DISCUSS] Apache Airavata release 0.17 - RC1

[Suresh Marru]

Hi All,

If you are looking to verify release signatures, here is a quick tip:

* download all release artifacts into a folder or do a svn checkout of the 
release - `svn co https://dist.apache.org/repos/dist/dev/airavata/0.17/RC1/ 
`

* Verify signatures by a small script something like:

for file in `find . -type f -iname '*.asc'`
  do
  gpg --verify ${file} 
  done
The output will indicate the You'll need to look at the output to ensure it 
contains only good signatures -

gpg: Good signature from ... gpg: Signature made …

Once you verify the signature, next step is to uncompress the source and verify 
it builds fine. 

This links provide guidance on release verifications:

https://www.apache.org/info/verification 


Cheers,
Suresh

> On Mar 21, 2019, at 1:07 AM, Suresh Marru  wrote:
> 
> Discussion thread for vote on Apache Airavata 0.17 release candidate.
> 
> If you have any questions or feedback or to post results of validating the 
> release, please reply to this thread. Once you verify the release, please 
> post your vote to the VOTE thread.  
> 
> For reference, the Apache release guide  - 
> http://www.apache.org/dev/release.html 
> 
> 
> Some tips to validate the release before you vote:
> 
> * Download the binary version and run the 5 minute or 10 minute tutorial as 
> described in README and website.
> * Download the source files from compressed files and release tag and build 
> (which includes tests). 
> * Verify the distribution for the required LICENSE and NOTICE files
> * Verify if all the staged files are signed and the signature is verifiable. 
> * Verify if the signing key in the project's KEYS file is hosted on a public 
> server
> 
> Thanks for your time in validating the release and voting,
> Suresh 
> (On Behalf of Airavata PMC)
> 

Re: Removing Stale Branches

[Christie, Marcus Aaron]

+1

I don't see any that you've marked with remove that need to be kept.

> On Mar 21, 2019, at 1:23 AM, Suresh Marru  wrote:
> 
> Hi All,
> 
> There are quite a few branches in airavata main repo which seems to be 
> definitely not relevant anymore. If we are absolutely sure about them, can we 
> remove them? 
> 
>  origin/AIRAVATA-2500 - This one is merged into master, remove? 
>  origin/AIRAVATA-2517-keycloak-mysql-connections - remove?
>  origin/AIRAVATA-2620 - remove? 
>  origin/BuildRTWithoutModel - Keep for now
>  origin/airavata-0.15-release-branch - remove?
>  
> origin/airavata-2938-change-db-initialization-in-registry-server-to-use-registry-refactoring-code
>  - remove?
>  origin/airavata-docker - keep 
>  origin/airavata-gov-registry - keep 
>  origin/archive - keep 
>  origin/auroraMesosIntegration - keep 
>  origin/cluster-monitoring - keep 
>  origin/data-manager - keep 
>  origin/data-model-pre-1.0-discussion - keep
>  origin/deprecated-gfac-modules - keep 
>  origin/develop - keep
>  origin/file-management.- keep
>  origin/gfac-storm - keep
>  origin/git-app-catalog - keep
>  origin/group-based-auth - remove since it is merged?
>  origin/grouper-integration - keep
>  origin/helix-integration - remove since it is merged? 
>  origin/hotfix-AIRAVATA-2096 - remove?
>  origin/jupyter-integration - keep
>  origin/lahiru/AIRAVATA-2017 - remove?
>  origin/lahiru/AIRAVATA-2065 - remove?
>  origin/lahiru/AIRAVATA-2107 - remove?
>  origin/lahiru/airavata-docker - keep
>  origin/master - keep
>  origin/master-staging-merge - keep 
>  origin/mongo-registry - keep 
>  origin/new-workflow-design - keep 
>  origin/nextcloud - keep 
>  origin/orchestratorTaskBreakdown - keep
>  origin/queue-gfac - remove?
>  origin/queue-gfac-rabbitmq - remove?
>  origin/registry-refactoring - remove since it is merged?
>  origin/revert-203-develop - remove
>  origin/shameera/build-fix - remove?
>  origin/staging - keep
>  origin/staging-temp - remove?
>  origin/thrift-0.10.0-upgrade - remove?
>  origin/user-profile - remove since it is merged?
>  origin/workflow-support - keep for now
> 
> Thoughts?
> Suresh



smime.p7s
Description: S/MIME cryptographic signature