[jira] [Commented] (TWILL-213) Increase of instances while starting up may lead to ignored retries and instance increases
[
https://issues.apache.org/jira/browse/TWILL-213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856669#comment-15856669
]
Martin Serrano commented on TWILL-213:
--
I've updated the description with what I think is happening.
> Increase of instances while starting up may lead to ignored retries and
> instance increases
> --
>
> Key: TWILL-213
> URL: https://issues.apache.org/jira/browse/TWILL-213
> Project: Apache Twill
> Issue Type: Bug
> Components: yarn
>Affects Versions: 0.9.0
>Reporter: Martin Serrano
>
> As seen in the test development for TWILL-181, if the number of instances for
> a container is increased before the {{ApplicationMasterService}} has observed
> the original request as being satisfied, the instance increase and any
> subsequent retries will be blocked. This is because in {{launchRunnable}}:
> {code}
> TwillContainerLauncher launcher = new TwillContainerLauncher(
> twillSpec.getRunnables().get(runnableName),
> processLauncher.getContainerInfo(), launchContext,
> ZKClients.namespace(zkClient, getZKNamespace(runnableName)),
> containerCount, jvmOpts, reservedMemory, getSecureStoreLocation());
> runningContainers.start(runnableName,
> processLauncher.getContainerInfo(), launcher);
> // Need to call complete to workaround bug in YARN AMRMClient
> if (provisionRequest.containerAcquired()) {
> amClient.completeContainerRequest(provisionRequest.getRequestId());
> }
> /*
>* The provisionRequest will either contain a single container
> (ALLOCATE_ONE_INSTANCE_AT_A_TIME), or all the
>* containers to satisfy the expectedContainers count. In the later
> case, the provision request is complete once
>* all the containers have run at which point we poll() to remove the
> provisioning request.
>*/
> if (expectedContainers.getExpected(runnableName) ==
> runningContainers.count(runnableName) ||
>
> provisioning.peek().getType().equals(AllocationSpecification.Type.ALLOCATE_ONE_INSTANCE_AT_A_TIME))
> {
> provisioning.poll();
> }
> {code}
> There is a race condition. The sequence:
> * *Thread A*: {{runningContainers.start}} is called and 2 instances are
> started
> * *Thread B*: The runnable from {{createSetInstanceRunnable}} executes, sees
> the 2 instances are started and updates the expected count to 3.
> * *Thread A*: Gets to if check comparing expectedContainers (3) to
> runningContainers.count (2). Since this fails, {{poll}} is not called and
> this provision request is not satisfied.
> Subsequent calls will try to provision the 3rd container because it seems
> like the first provision request is not yet satisfied.
> The {{MaxRetriesTestRun.maxRetriesWithIncreasedInstances}} method can be used
> to reproduce this case intermittently by changing the {{allRunning.await}}
> check to something that does a countdown latch {{onRunning}} as
> {{EchoServerTestRun}} does.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TWILL-191) Add coding style preferences for Eclipse to site
[ https://issues.apache.org/jira/browse/TWILL-191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856596#comment-15856596 ] ASF GitHub Bot commented on TWILL-191: -- Github user hsaputra commented on the issue: https://github.com/apache/twill/pull/30 Cool! Looking forward to them =) > Add coding style preferences for Eclipse to site > > > Key: TWILL-191 > URL: https://issues.apache.org/jira/browse/TWILL-191 > Project: Apache Twill > Issue Type: Task >Reporter: Martin Serrano >Assignee: Martin Serrano >Priority: Minor > Fix For: 0.10.0 > > > As mentioned on the dev list it would be good to have preferences files for > the major IDEs. This ticket is to track the Eclipse version of those > preferences. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] twill issue #30: (TWILL-191), add eclipse settings to site
Github user hsaputra commented on the issue: https://github.com/apache/twill/pull/30 Cool! Looking forward to them =) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (TWILL-191) Add coding style preferences for Eclipse to site
[ https://issues.apache.org/jira/browse/TWILL-191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856589#comment-15856589 ] ASF GitHub Bot commented on TWILL-191: -- Github user serranom commented on the issue: https://github.com/apache/twill/pull/30 i've now set up my twill git repo to have serranom and my github email. so future commits will be correct. > Add coding style preferences for Eclipse to site > > > Key: TWILL-191 > URL: https://issues.apache.org/jira/browse/TWILL-191 > Project: Apache Twill > Issue Type: Task >Reporter: Martin Serrano >Assignee: Martin Serrano >Priority: Minor > Fix For: 0.10.0 > > > As mentioned on the dev list it would be good to have preferences files for > the major IDEs. This ticket is to track the Eclipse version of those > preferences. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TWILL-191) Add coding style preferences for Eclipse to site
[ https://issues.apache.org/jira/browse/TWILL-191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856572#comment-15856572 ] ASF GitHub Bot commented on TWILL-191: -- Github user hsaputra commented on the issue: https://github.com/apache/twill/pull/30 Could you check your Git identity via `git config --global user.name` and `git config --global user.email` ? > Add coding style preferences for Eclipse to site > > > Key: TWILL-191 > URL: https://issues.apache.org/jira/browse/TWILL-191 > Project: Apache Twill > Issue Type: Task >Reporter: Martin Serrano >Assignee: Martin Serrano >Priority: Minor > Fix For: 0.10.0 > > > As mentioned on the dev list it would be good to have preferences files for > the major IDEs. This ticket is to track the Eclipse version of those > preferences. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] twill issue #30: (TWILL-191), add eclipse settings to site
Github user hsaputra commented on the issue: https://github.com/apache/twill/pull/30 Could you check your Git identity via `git config --global user.name` and `git config --global user.email` ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (TWILL-191) Add coding style preferences for Eclipse to site
[ https://issues.apache.org/jira/browse/TWILL-191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856554#comment-15856554 ] ASF GitHub Bot commented on TWILL-191: -- Github user serranom closed the pull request at: https://github.com/apache/twill/pull/30 > Add coding style preferences for Eclipse to site > > > Key: TWILL-191 > URL: https://issues.apache.org/jira/browse/TWILL-191 > Project: Apache Twill > Issue Type: Task >Reporter: Martin Serrano >Assignee: Martin Serrano >Priority: Minor > Fix For: 0.10.0 > > > As mentioned on the dev list it would be good to have preferences files for > the major IDEs. This ticket is to track the Eclipse version of those > preferences. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] twill pull request #30: (TWILL-191), add eclipse settings to site
Github user serranom closed the pull request at: https://github.com/apache/twill/pull/30 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (TWILL-191) Add coding style preferences for Eclipse to site
[ https://issues.apache.org/jira/browse/TWILL-191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856550#comment-15856550 ] ASF GitHub Bot commented on TWILL-191: -- Github user chtyim commented on the issue: https://github.com/apache/twill/pull/30 I've merged the changes. Please close the PR (the asfgit only close PR with pushes that go to master branch). > Add coding style preferences for Eclipse to site > > > Key: TWILL-191 > URL: https://issues.apache.org/jira/browse/TWILL-191 > Project: Apache Twill > Issue Type: Task >Reporter: Martin Serrano >Assignee: Martin Serrano >Priority: Minor > Fix For: 0.10.0 > > > As mentioned on the dev list it would be good to have preferences files for > the major IDEs. This ticket is to track the Eclipse version of those > preferences. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] twill issue #30: (TWILL-191), add eclipse settings to site
Github user serranom commented on the issue: https://github.com/apache/twill/pull/30 oh... yeah, i've been using my personal github but committing from work. do i need to do something for my PR for you now or is this just a good idea? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] twill issue #30: (TWILL-191), add eclipse settings to site
Github user hsaputra commented on the issue: https://github.com/apache/twill/pull/30 Hi @serranom , I just noticed that the commits you made use different account than your pull request. I would advise to check it to make sure your contributions easily tracked in Github --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (TWILL-181) Control the maximum number of retries for failed application starts
[ https://issues.apache.org/jira/browse/TWILL-181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15856512#comment-15856512 ] ASF GitHub Bot commented on TWILL-181: -- Github user asfgit closed the pull request at: https://github.com/apache/twill/pull/23 > Control the maximum number of retries for failed application starts > --- > > Key: TWILL-181 > URL: https://issues.apache.org/jira/browse/TWILL-181 > Project: Apache Twill > Issue Type: Improvement > Components: yarn >Affects Versions: 0.7.0-incubating >Reporter: Martin Serrano >Assignee: Martin Serrano > Fix For: 0.10.0 > > > If an application consistently exits with a non-zero code, twill will > attempt to restart indefinitely. I ran into this issue and a list search > also reveals [others| http://markmail.org/message/dehx7r6tpqgcmjh4]. > There should be a mechanism to specify the maximum number of retries until > the application fails. Ideally by default there would be a non-infinite > maximum. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[GitHub] twill pull request #23: (TWILL-181) allow setting the maximum number of retr...
Github user asfgit closed the pull request at: https://github.com/apache/twill/pull/23 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Updated] (TWILL-214) RelaxLocality no response
[
https://issues.apache.org/jira/browse/TWILL-214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
imad updated TWILL-214:
---
Affects Version/s: 0.9.0
> RelaxLocality no response
> --
>
> Key: TWILL-214
> URL: https://issues.apache.org/jira/browse/TWILL-214
> Project: Apache Twill
> Issue Type: Bug
> Components: yarn
>Affects Versions: 0.9.0
> Environment: hadoop2.6.0-cdh5.8.0
>Reporter: imad
>
> i want assign some runner on some host ,but when i change RelaxLocality is
> false,it was cant assign any container , is more config not swith when i
> change RelaxLocality?
> String requestId = amClient.addContainerRequest(capability,
> newContainers)
> .addHosts(hosts)
> .addRacks(racks)
> .setPriority(0)
> .setRelaxLocality(false)
> .apply();
> next is my app:
>@Override
> public TwillSpecification configure() {
> return TwillSpecification.Builder.with()
> .setName("presto")
> .withRunnable()
> .add("master", new PrestoRunnable(), master).noLocalFiles()
> .add("slave", new PrestoRunnable(), slave).noLocalFiles()
> .withPlacementPolicy()
> .add(Hosts.of("10.129.64.11"), "master")
> // .add(Hosts.of(nodeReports.get(1).getHttpAddress()), "slave")
> .add(TwillSpecification.PlacementPolicy.Type.DISTRIBUTED, "slave")
> .withOrder().begin("master").nextWhenStarted("slave")
> .build();
> }
> }
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (TWILL-214) RelaxLocality no response
[
https://issues.apache.org/jira/browse/TWILL-214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
imad updated TWILL-214:
---
Component/s: yarn
> RelaxLocality no response
> --
>
> Key: TWILL-214
> URL: https://issues.apache.org/jira/browse/TWILL-214
> Project: Apache Twill
> Issue Type: Bug
> Components: yarn
>Affects Versions: 0.9.0
> Environment: hadoop2.6.0-cdh5.8.0
>Reporter: imad
>
> i want assign some runner on some host ,but when i change RelaxLocality is
> false,it was cant assign any container , is more config not swith when i
> change RelaxLocality?
> String requestId = amClient.addContainerRequest(capability,
> newContainers)
> .addHosts(hosts)
> .addRacks(racks)
> .setPriority(0)
> .setRelaxLocality(false)
> .apply();
> next is my app:
>@Override
> public TwillSpecification configure() {
> return TwillSpecification.Builder.with()
> .setName("presto")
> .withRunnable()
> .add("master", new PrestoRunnable(), master).noLocalFiles()
> .add("slave", new PrestoRunnable(), slave).noLocalFiles()
> .withPlacementPolicy()
> .add(Hosts.of("10.129.64.11"), "master")
> // .add(Hosts.of(nodeReports.get(1).getHttpAddress()), "slave")
> .add(TwillSpecification.PlacementPolicy.Type.DISTRIBUTED, "slave")
> .withOrder().begin("master").nextWhenStarted("slave")
> .build();
> }
> }
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (TWILL-214) RelaxLocality no response
imad created TWILL-214: -- Summary: RelaxLocality no response Key: TWILL-214 URL: https://issues.apache.org/jira/browse/TWILL-214 Project: Apache Twill Issue Type: Bug Environment: hadoop2.6.0-cdh5.8.0 Reporter: imad i want assign some runner on some host ,but when i change RelaxLocality is false,it was cant assign any container , is more config not swith when i change RelaxLocality? -- This message was sent by Atlassian JIRA (v6.3.15#6346)
