Re: forking off releases/branch-0.60 to work with hadoop-2.6

[Sumit Mohanty]

I am proposing that SLIDER-620 should be part of 0.60 release. While this
is a debugging improvement, it seems to be a critical one especially when
the error happens due to some bad configuration.

I am working on the fix and testing it. ETA is Nov 6th morning.

Let me know if there are any concerns.

-Sumit

On Wed, Nov 5, 2014 at 11:21 PM, Gour Saha  wrote:

> Steve,
>
> The changes for SLIDER-555 (AM log4j) has been merged into develop and
> releases/slider-0.60.
>
> -Gour
>
> On Wed, Nov 5, 2014 at 12:50 PM, Steve Loughran 
> wrote:
>
> > FYI, there's going to be an RC of Hadoop 2.6 this weekend. To celebrate
> > this I'm creating a slider-0.60 release which will be in sync.
> >
> > 1. The branch already exists: releases/branch-0.60  please try and
> > stabilize this. new features into develop/
> >
> > 2. I did one last-minute feature addition to slider today, before this
> fork
> > : SLIDER-619 
> >
> > the registry --list and --listconf commands support the --out argument to
> > take a file; if set it saves the output to a text file. This is for
> > testing.
> >
> > Gour: I know you want to get your work on server-side logging in: once
> you
> > are happy with it commit to develop/ and then cherry pick over the 0.60
> > branch
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: forking off releases/branch-0.60 to work with hadoop-2.6

[Gour Saha]

Steve,

The changes for SLIDER-555 (AM log4j) has been merged into develop and
releases/slider-0.60.

-Gour

On Wed, Nov 5, 2014 at 12:50 PM, Steve Loughran 
wrote:

> FYI, there's going to be an RC of Hadoop 2.6 this weekend. To celebrate
> this I'm creating a slider-0.60 release which will be in sync.
>
> 1. The branch already exists: releases/branch-0.60  please try and
> stabilize this. new features into develop/
>
> 2. I did one last-minute feature addition to slider today, before this fork
> : SLIDER-619 
>
> the registry --list and --listconf commands support the --out argument to
> take a file; if set it saves the output to a text file. This is for
> testing.
>
> Gour: I know you want to get your work on server-side logging in: once you
> are happy with it commit to develop/ and then cherry pick over the 0.60
> branch
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

[jira] [Commented] (SLIDER-555) AM needs to get log aggregation friendly log4j

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SLIDER-555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199930#comment-14199930
 ] 

ASF subversion and git services commented on SLIDER-555:


Commit f6a3467b071656b38910e39f39a5a85d37d7abd2 in incubator-slider's branch 
refs/heads/releases/slider-0.60 from [~gsaha]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=f6a3467 ]

SLIDER-555 AM needs to get log aggregation friendly log4j


> AM needs to get log aggregation friendly log4j
> --
>
> Key: SLIDER-555
> URL: https://issues.apache.org/jira/browse/SLIDER-555
> Project: Slider
>  Issue Type: Bug
>  Components: appmaster
>Reporter: Steve Loughran
>Assignee: Gour Saha
> Fix For: Slider 0.60
>
>
> The AM needs to get log4j settings



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (SLIDER-555) AM needs to get log aggregation friendly log4j

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SLIDER-555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199931#comment-14199931
 ] 

ASF subversion and git services commented on SLIDER-555:


Commit 83c3e061606e688aa101af5dda2c3dc1534c9148 in incubator-slider's branch 
refs/heads/releases/slider-0.60 from [~sumitmohanty]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=83c3e06 ]

SLIDER-555 AM needs to get log aggregation friendly log4j (gsaha via smohanty)


> AM needs to get log aggregation friendly log4j
> --
>
> Key: SLIDER-555
> URL: https://issues.apache.org/jira/browse/SLIDER-555
> Project: Slider
>  Issue Type: Bug
>  Components: appmaster
>Reporter: Steve Loughran
>Assignee: Gour Saha
> Fix For: Slider 0.60
>
>
> The AM needs to get log4j settings



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (SLIDER-620) Execute command should allow polling for daemons that may stop within seconds

[Sumit Mohanty (JIRA)]

Sumit Mohanty created SLIDER-620:


 Summary: Execute command should allow polling for daemons that may 
stop within seconds
 Key: SLIDER-620
 URL: https://issues.apache.org/jira/browse/SLIDER-620
 Project: Slider
  Issue Type: Bug
  Components: agent
Affects Versions: Slider 0.50
Reporter: Sumit Mohanty
Assignee: Sumit Mohanty
Priority: Critical
 Fix For: Slider 0.60


When component starts fail immediately then there is no good logging to capture 
that failure. These processes are usually long running and thus Slider does not 
wait for completion. But if there is an error (e.g. bad JAVA_HOME) and process 
fails immediately then log does not capture this error. This makes it very 
difficult to debug and requires users to manually run the command.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (SLIDER-620) Execute command should allow polling for daemons that may stop within seconds

[Sumit Mohanty (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SLIDER-620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sumit Mohanty updated SLIDER-620:
-
Sprint: Slider November #1

> Execute command should allow polling for daemons that may stop within seconds
> -
>
> Key: SLIDER-620
> URL: https://issues.apache.org/jira/browse/SLIDER-620
> Project: Slider
>  Issue Type: Bug
>  Components: agent
>Affects Versions: Slider 0.50
>Reporter: Sumit Mohanty
>Assignee: Sumit Mohanty
>Priority: Critical
> Fix For: Slider 0.60
>
>
> When component starts fail immediately then there is no good logging to 
> capture that failure. These processes are usually long running and thus 
> Slider does not wait for completion. But if there is an error (e.g. bad 
> JAVA_HOME) and process fails immediately then log does not capture this 
> error. This makes it very difficult to debug and requires users to manually 
> run the command.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Maven repository?

[hsy...@gmail.com]

This is git repository? What I mean is if I want to write some code depends
on slider library. How would I include the dependency in pom.xml?

Thanks!


On Wed, Nov 5, 2014 at 7:23 PM, Ted Yu  wrote:

> Slider maven repo is here:
>
> https://git-wip-us.apache.org/repos/asf/incubator-slider.git
>
>
> You can checkout develop branch.
>
>
> Cheers
>
>
> On Wed, Nov 5, 2014 at 6:36 PM, hsy...@gmail.com  wrote:
>
> > Is there a public maven repository that I can checkout the slider
> library?
> >
> > Best,
> > Siyuan
> >
>

Re: Maven repository?

[Ted Yu]

Slider maven repo is here:

https://git-wip-us.apache.org/repos/asf/incubator-slider.git


You can checkout develop branch.


Cheers


On Wed, Nov 5, 2014 at 6:36 PM, hsy...@gmail.com  wrote:

> Is there a public maven repository that I can checkout the slider library?
>
> Best,
> Siyuan
>

Maven repository?

[hsy...@gmail.com]

Is there a public maven repository that I can checkout the slider library?

Best,
Siyuan

[jira] [Commented] (SLIDER-555) AM needs to get log aggregation friendly log4j

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SLIDER-555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199652#comment-14199652
 ] 

ASF subversion and git services commented on SLIDER-555:


Commit 81ee39b4804da19751f415de33e8626212f17245 in incubator-slider's branch 
refs/heads/develop from [~sumitmohanty]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=81ee39b ]

SLIDER-555 AM needs to get log aggregation friendly log4j (gsaha via smohanty)


> AM needs to get log aggregation friendly log4j
> --
>
> Key: SLIDER-555
> URL: https://issues.apache.org/jira/browse/SLIDER-555
> Project: Slider
>  Issue Type: Bug
>  Components: appmaster
>Reporter: Steve Loughran
>Assignee: Gour Saha
> Fix For: Slider 0.60
>
>
> The AM needs to get log4j settings



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (SLIDER-555) AM needs to get log aggregation friendly log4j

[Gour Saha (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SLIDER-555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gour Saha resolved SLIDER-555.
--
Resolution: Fixed

> AM needs to get log aggregation friendly log4j
> --
>
> Key: SLIDER-555
> URL: https://issues.apache.org/jira/browse/SLIDER-555
> Project: Slider
>  Issue Type: Bug
>  Components: appmaster
>Reporter: Steve Loughran
>Assignee: Gour Saha
> Fix For: Slider 0.60
>
>
> The AM needs to get log4j settings



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (SLIDER-555) AM needs to get log aggregation friendly log4j

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SLIDER-555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199603#comment-14199603
 ] 

ASF subversion and git services commented on SLIDER-555:


Commit a29d825e68b2f51dab80257ce3f4501cc2ce3e88 in incubator-slider's branch 
refs/heads/develop from [~gsaha]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=a29d825 ]

SLIDER-555 AM needs to get log aggregation friendly log4j


> AM needs to get log aggregation friendly log4j
> --
>
> Key: SLIDER-555
> URL: https://issues.apache.org/jira/browse/SLIDER-555
> Project: Slider
>  Issue Type: Bug
>  Components: appmaster
>Reporter: Steve Loughran
>Assignee: Gour Saha
> Fix For: Slider 0.60
>
>
> The AM needs to get log4j settings



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Is it able to make AM to restart from previous state?

[hsy...@gmail.com]

Steve,

I found out from the code that everything is kept in history folder in
hdfs.

You mentioned that if I add new component, the history layout would be
discarded. What if I add more component instances in configuration? Do you
try to launch instance from previous node and add new instance from new
node?
What if you decrease the instance number?

Thanks!

Best,
Siyuan

On Wed, Nov 5, 2014 at 1:44 PM, hsy...@gmail.com  wrote:

> Thanks Steve,
>
> Is No 1  a new feature in YARN (Not released yet)?
>
> And you mentioned slider saves the location in history files. What are the
> history files and where is it stored? Is it in HDFS?
>
> If the one of the previous machines is gone, will it try to get resource
> from new labeled machine?
>
> Thanks
>
>
> On Wed, Nov 5, 2014 at 12:46 PM, Steve Loughran 
> wrote:
>
>> On 5 November 2014 20:21, hsy...@gmail.com  wrote:
>>
>> > Hi guys,
>> >
>> > I noticed in the code when a container fails it will try to relaunch
>> from
>> > the same node. My question is if I restart whole application(Ex. AM got
>> > killed, or manually restart the app). Does slider try to launch all
>> > containers from the nodes where it was running?
>> >
>> >
>> 1. If the AM crashes then YARN will restart it. The containers will keep
>> working. When the AM comes back up it will work out its state and all
>> running containers will stay live. Any containers that were part way
>> through starting will be released and new ones requested (there's no
>> record
>> of what state they were in, so a clean destroy is simpler)
>>
>>
>> If you stop/start the app then it asks for the nodes back on the same
>> machines they were on. It saves the locations (look in the history subdir)
>> to see the history files.
>>
>> Slider tries to read the last entry, going back to previous ones if the
>> last one doesn't load. It then asks YARN for containers on those machines.
>> There's no guarantee you get them though.
>>
>> Looking at the history code last week I noticed one little quirk: it
>> doesn't reload the histories if the number of component types has
>> increased. It just indexes the entries; more entries means it doesn't know
>> how to handle them.
>>
>> To avoid this problem define all your components from the outset, setting
>> the instances count 0 for ones you don't currently want
>>
>> Thanks!
>> >
>> > Best,
>> > Siyuan
>> >
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified
>> that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender
>> immediately
>> and delete it from your system. Thank You.
>>
>
>

Re: Is it able to make AM to restart from previous state?

[hsy...@gmail.com]

Thanks Steve,

Is No 1  a new feature in YARN (Not released yet)?

And you mentioned slider saves the location in history files. What are the
history files and where is it stored? Is it in HDFS?

If the one of the previous machines is gone, will it try to get resource
from new labeled machine?

Thanks


On Wed, Nov 5, 2014 at 12:46 PM, Steve Loughran 
wrote:

> On 5 November 2014 20:21, hsy...@gmail.com  wrote:
>
> > Hi guys,
> >
> > I noticed in the code when a container fails it will try to relaunch from
> > the same node. My question is if I restart whole application(Ex. AM got
> > killed, or manually restart the app). Does slider try to launch all
> > containers from the nodes where it was running?
> >
> >
> 1. If the AM crashes then YARN will restart it. The containers will keep
> working. When the AM comes back up it will work out its state and all
> running containers will stay live. Any containers that were part way
> through starting will be released and new ones requested (there's no record
> of what state they were in, so a clean destroy is simpler)
>
>
> If you stop/start the app then it asks for the nodes back on the same
> machines they were on. It saves the locations (look in the history subdir)
> to see the history files.
>
> Slider tries to read the last entry, going back to previous ones if the
> last one doesn't load. It then asks YARN for containers on those machines.
> There's no guarantee you get them though.
>
> Looking at the history code last week I noticed one little quirk: it
> doesn't reload the histories if the number of component types has
> increased. It just indexes the entries; more entries means it doesn't know
> how to handle them.
>
> To avoid this problem define all your components from the outset, setting
> the instances count 0 for ones you don't currently want
>
> Thanks!
> >
> > Best,
> > Siyuan
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Re: slider report

[Billie Rinaldi]

Sure, you could add a couple of sentences to show what we've done.  No need
to go into much detail, as the goal is to illustrate project health rather
than any particular technical milestone.

On Wed, Nov 5, 2014 at 1:04 PM, Sumit Mohanty 
wrote:

> Do we need more details on what we did in terms features developed etc? If
> yes, I can edit and add some details.
>
> On Wed, Nov 5, 2014 at 12:25 PM, Ted Yu  wrote:
>
> > bq. We have added one committer / PMC member.
> >
> > I think we should mention Gour's name in the above sentence.
> >
> > lgtm
> >
> > On Wed, Nov 5, 2014 at 12:12 PM, Billie Rinaldi <
> billie.rina...@gmail.com>
> > wrote:
> >
> > > I drafted the following Slider report.  If anyone wants to edit or add
> > > anything, please do so on the wiki:
> > > https://wiki.apache.org/incubator/November2014
> > >
> > > 
> > > Slider
> > >
> > > Slider is a collection of tools and technologies to package, deploy,
> and
> > > manage long running applications on Apache Hadoop YARN clusters.
> > >
> > > Slider has been incubating since 2014-04-29.
> > >
> > > Three most important issues to address in the move towards graduation:
> > >
> > >   1. Increasing community diversity
> > >   2. Completing podling name search
> > >   3. Building a user community
> > >
> > > Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
> > > aware of?
> > >
> > > No
> > >
> > > How has the community developed since the last report?
> > >
> > > We have added one committer / PMC member.
> > >
> > > How has the project developed since the last report?
> > >
> > > Development and mailing lists have been active, working towards a
> > > major upcoming release.  A service registry has been redesigned and
> > > contributed to Yarn (see YARN-913), and Slider continues to help
> > > drive improvements needed for long-lived services in Yarn.
> > >
> > > Date of last release:
> > >
> > >   2014-07-21
> > >
> > > When were the last committers or PMC members elected?
> > >
> > >   2014-09-22
> > >
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Re: slider report

[Sumit Mohanty]

Do we need more details on what we did in terms features developed etc? If
yes, I can edit and add some details.

On Wed, Nov 5, 2014 at 12:25 PM, Ted Yu  wrote:

> bq. We have added one committer / PMC member.
>
> I think we should mention Gour's name in the above sentence.
>
> lgtm
>
> On Wed, Nov 5, 2014 at 12:12 PM, Billie Rinaldi 
> wrote:
>
> > I drafted the following Slider report.  If anyone wants to edit or add
> > anything, please do so on the wiki:
> > https://wiki.apache.org/incubator/November2014
> >
> > 
> > Slider
> >
> > Slider is a collection of tools and technologies to package, deploy, and
> > manage long running applications on Apache Hadoop YARN clusters.
> >
> > Slider has been incubating since 2014-04-29.
> >
> > Three most important issues to address in the move towards graduation:
> >
> >   1. Increasing community diversity
> >   2. Completing podling name search
> >   3. Building a user community
> >
> > Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
> > aware of?
> >
> > No
> >
> > How has the community developed since the last report?
> >
> > We have added one committer / PMC member.
> >
> > How has the project developed since the last report?
> >
> > Development and mailing lists have been active, working towards a
> > major upcoming release.  A service registry has been redesigned and
> > contributed to Yarn (see YARN-913), and Slider continues to help
> > drive improvements needed for long-lived services in Yarn.
> >
> > Date of last release:
> >
> >   2014-07-21
> >
> > When were the last committers or PMC members elected?
> >
> >   2014-09-22
> >
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

forking off releases/branch-0.60 to work with hadoop-2.6

[Steve Loughran]

FYI, there's going to be an RC of Hadoop 2.6 this weekend. To celebrate
this I'm creating a slider-0.60 release which will be in sync.

1. The branch already exists: releases/branch-0.60  please try and
stabilize this. new features into develop/

2. I did one last-minute feature addition to slider today, before this fork
: SLIDER-619 

the registry --list and --listconf commands support the --out argument to
take a file; if set it saves the output to a text file. This is for testing.

Gour: I know you want to get your work on server-side logging in: once you
are happy with it commit to develop/ and then cherry pick over the 0.60
branch

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Is it able to make AM to restart from previous state?

[Steve Loughran]

On 5 November 2014 20:21, hsy...@gmail.com  wrote:

> Hi guys,
>
> I noticed in the code when a container fails it will try to relaunch from
> the same node. My question is if I restart whole application(Ex. AM got
> killed, or manually restart the app). Does slider try to launch all
> containers from the nodes where it was running?
>
>
1. If the AM crashes then YARN will restart it. The containers will keep
working. When the AM comes back up it will work out its state and all
running containers will stay live. Any containers that were part way
through starting will be released and new ones requested (there's no record
of what state they were in, so a clean destroy is simpler)


If you stop/start the app then it asks for the nodes back on the same
machines they were on. It saves the locations (look in the history subdir)
to see the history files.

Slider tries to read the last entry, going back to previous ones if the
last one doesn't load. It then asks YARN for containers on those machines.
There's no guarantee you get them though.

Looking at the history code last week I noticed one little quirk: it
doesn't reload the histories if the number of component types has
increased. It just indexes the entries; more entries means it doesn't know
how to handle them.

To avoid this problem define all your components from the outset, setting
the instances count 0 for ones you don't currently want

Thanks!
>
> Best,
> Siyuan
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

[jira] [Commented] (SLIDER-544) Release Slider 0.6.0

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SLIDER-544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199050#comment-14199050
 ] 

ASF subversion and git services commented on SLIDER-544:


Commit 4e4652e4bb25c5e472973739f58a1b42176238f0 in incubator-slider's branch 
refs/heads/develop from [~ste...@apache.org]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=4e4652e ]

SLIDER-544 updating release POMs for 0.60-incubating


> Release Slider 0.6.0
> 
>
> Key: SLIDER-544
> URL: https://issues.apache.org/jira/browse/SLIDER-544
> Project: Slider
>  Issue Type: Task
>  Components: build
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Fix For: Slider 0.60
>
>   Original Estimate: 4h
>  Remaining Estimate: 4h
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (SLIDER-602) AgentRegistryIT-testAgentRegistry failing: ~ passdown

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SLIDER-602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199044#comment-14199044
 ] 

ASF subversion and git services commented on SLIDER-602:


Commit 7fb01c36a16cf3ae20278dda2ef02b93b8342d46 in incubator-slider's branch 
refs/heads/releases/slider-0.52 from [~ste...@apache.org]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=7fb01c3 ]

SLIDER-602 AgentRegistryIT-testAgentRegistry failing


> AgentRegistryIT-testAgentRegistry failing: ~ passdown
> -
>
> Key: SLIDER-602
> URL: https://issues.apache.org/jira/browse/SLIDER-602
> Project: Slider
>  Issue Type: Sub-task
>  Components: test
>Affects Versions: Slider 0.60
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Fix For: Slider 0.60
>
>
> the test {{AgentRegistryIT-testAgentRegistry}} is faling with a not found 
> (44) trying to list ~. Looking at the output, the escaping of the ~ appears 
> to be failing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (SLIDER-604) AppsThroughAgentIT brittle against slow startup

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SLIDER-604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199046#comment-14199046
 ] 

ASF subversion and git services commented on SLIDER-604:


Commit b6d91f047738c98097d794dc6927d30279ed5d9a in incubator-slider's branch 
refs/heads/releases/slider-0.52 from [~ste...@apache.org]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=b6d91f0 ]

SLIDER-604 AppsThroughAgentIT to spin waiting for live instance count


> AppsThroughAgentIT brittle against slow startup
> ---
>
> Key: SLIDER-604
> URL: https://issues.apache.org/jira/browse/SLIDER-604
> Project: Slider
>  Issue Type: Sub-task
>  Components: test
>Affects Versions: Slider 0.60
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Fix For: Slider 0.60
>
>
> looking at SLIDER-603 and the root cause, its clear that 
> {{AppsThroughAgentIT}} was failing the assertion as it was checking the size 
> of the cluster way too early. It should be spinning for the expect container 
> count rather than just checking immediately after AM launch



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (SLIDER-544) Release Slider 0.6.0

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SLIDER-544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199049#comment-14199049
 ] 

ASF subversion and git services commented on SLIDER-544:


Commit 5034161db043f99d325a6654081728d42e8d190a in incubator-slider's branch 
refs/heads/develop from [~ste...@apache.org]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=5034161 ]

SLIDER-544 updating release POMs for 0.60-incubating


> Release Slider 0.6.0
> 
>
> Key: SLIDER-544
> URL: https://issues.apache.org/jira/browse/SLIDER-544
> Project: Slider
>  Issue Type: Task
>  Components: build
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Fix For: Slider 0.60
>
>   Original Estimate: 4h
>  Remaining Estimate: 4h
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (SLIDER-544) Release Slider 0.6.0

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SLIDER-544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199048#comment-14199048
 ] 

ASF subversion and git services commented on SLIDER-544:


Commit 5034161db043f99d325a6654081728d42e8d190a in incubator-slider's branch 
refs/heads/releases/slider-0.60 from [~ste...@apache.org]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=5034161 ]

SLIDER-544 updating release POMs for 0.60-incubating


> Release Slider 0.6.0
> 
>
> Key: SLIDER-544
> URL: https://issues.apache.org/jira/browse/SLIDER-544
> Project: Slider
>  Issue Type: Task
>  Components: build
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Fix For: Slider 0.60
>
>   Original Estimate: 4h
>  Remaining Estimate: 4h
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (SLIDER-614) Release Slider 0.52 against old registry code

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SLIDER-614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199047#comment-14199047
 ] 

ASF subversion and git services commented on SLIDER-614:


Commit 57130b5d65b0b5e4e78fe5e394394acb6bca6665 in incubator-slider's branch 
refs/heads/releases/slider-0.52 from [~ste...@apache.org]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=57130b5 ]

SLIDER-614 updating release POMs for 0.52


> Release Slider 0.52 against old registry code
> -
>
> Key: SLIDER-614
> URL: https://issues.apache.org/jira/browse/SLIDER-614
> Project: Slider
>  Issue Type: Bug
>  Components: build
>Affects Versions: Slider 0.50
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> Build a 0.52 release against the "classic" (pre-gour-enhanced) registry.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (SLIDER-603) NPE in AppsThroughAgentIT

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SLIDER-603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199045#comment-14199045
 ] 

ASF subversion and git services commented on SLIDER-603:


Commit ecca0aad72b7b4901bde60a2a24532b3ad4ca5b6 in incubator-slider's branch 
refs/heads/releases/slider-0.52 from [~ste...@apache.org]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=ecca0aa ]

SLIDER-603 : NPE in AppsThroughAgentIT


> NPE in AppsThroughAgentIT
> -
>
> Key: SLIDER-603
> URL: https://issues.apache.org/jira/browse/SLIDER-603
> Project: Slider
>  Issue Type: Bug
>  Components: test
>Affects Versions: Slider 0.60
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Fix For: Slider 0.60
>
>
> NPE in one of the assertions in AppsThroughAgentIT



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (SLIDER-619) registry list and listconf commands to save output to --out file

[Steve Loughran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SLIDER-619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved SLIDER-619.
---
Resolution: Fixed

> registry list and listconf commands to save output to --out file
> 
>
> Key: SLIDER-619
> URL: https://issues.apache.org/jira/browse/SLIDER-619
> Project: Slider
>  Issue Type: Test
>  Components: client
>Affects Versions: Slider 0.50
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Fix For: Slider 0.60
>
>
> To aid testing, have the registry {{--list}} and {{-listconf}} commands 
> support the {{--out}} file entry. If set the output is just saved  as text, 
> one entry per line.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (SLIDER-619) registry list and listconf commands to save output to --out file

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SLIDER-619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199033#comment-14199033
 ] 

ASF subversion and git services commented on SLIDER-619:


Commit 1636951 from [~ste...@apache.org] in branch 'site/trunk'
[ https://svn.apache.org/r1636951 ]

SLIDER-619
registry list and listconf commands to save output to --out file

> registry list and listconf commands to save output to --out file
> 
>
> Key: SLIDER-619
> URL: https://issues.apache.org/jira/browse/SLIDER-619
> Project: Slider
>  Issue Type: Test
>  Components: client
>Affects Versions: Slider 0.50
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Fix For: Slider 0.60
>
>
> To aid testing, have the registry {{--list}} and {{-listconf}} commands 
> support the {{--out}} file entry. If set the output is just saved  as text, 
> one entry per line.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (SLIDER-616) re-enable AMFailuresIT

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SLIDER-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199025#comment-14199025
 ] 

ASF subversion and git services commented on SLIDER-616:


Commit 8943a14f0a1c45f5a83b0b3ccb9cc29bbe2a7cf8 in incubator-slider's branch 
refs/heads/feature/SLIDER-616_re-enable_AMFailuresIT from [~ste...@apache.org]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=8943a14 ]

SLIDER-616 AMFailuresIT


> re-enable AMFailuresIT
> --
>
> Key: SLIDER-616
> URL: https://issues.apache.org/jira/browse/SLIDER-616
> Project: Slider
>  Issue Type: Test
>  Components: test
>Affects Versions: Slider 0.50
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Fix For: Slider 0.60
>
>
> Re-enable the test {{AMFailuresIT}} ; make it robust against timing 
> differences in AM restart delays &C



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (SLIDER-544) Release Slider 0.6.0

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SLIDER-544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199030#comment-14199030
 ] 

ASF subversion and git services commented on SLIDER-544:


Commit 844e4b6776c627f5eddeb22b233eff0da9cea55b in incubator-slider's branch 
refs/heads/develop from [~ste...@apache.org]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=844e4b6 ]

SLIDER-544 fix up dependencies before releasing


> Release Slider 0.6.0
> 
>
> Key: SLIDER-544
> URL: https://issues.apache.org/jira/browse/SLIDER-544
> Project: Slider
>  Issue Type: Task
>  Components: build
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Fix For: Slider 0.60
>
>   Original Estimate: 4h
>  Remaining Estimate: 4h
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (SLIDER-619) registry list and listconf commands to save output to --out file

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SLIDER-619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199024#comment-14199024
 ] 

ASF subversion and git services commented on SLIDER-619:


Commit 70c1b1a7b21e02abff825009d51ca40ee9f2aca6 in incubator-slider's branch 
refs/heads/feature/SLIDER-616_re-enable_AMFailuresIT from [~ste...@apache.org]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=70c1b1a ]

SLIDER-619 registry list and listconf commands to save output to --out file


> registry list and listconf commands to save output to --out file
> 
>
> Key: SLIDER-619
> URL: https://issues.apache.org/jira/browse/SLIDER-619
> Project: Slider
>  Issue Type: Test
>  Components: client
>Affects Versions: Slider 0.50
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Fix For: Slider 0.60
>
>
> To aid testing, have the registry {{--list}} and {{-listconf}} commands 
> support the {{--out}} file entry. If set the output is just saved  as text, 
> one entry per line.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (SLIDER-525) Storm app package for windows does not report component status correctly

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SLIDER-525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199022#comment-14199022
 ] 

ASF subversion and git services commented on SLIDER-525:


Commit 66b1aa4a403bfe60c026b63f201f8cdc539f2dfe in incubator-slider's branch 
refs/heads/feature/SLIDER-616_re-enable_AMFailuresIT from [~sumitmohanty]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=66b1aa4 ]

SLIDER-525. Storm app package for windows does not report component status 
correctly
HBase Win package has the ability to provide custom ports for Thrift and Thrift2


> Storm app package for windows does not report component status correctly
> 
>
> Key: SLIDER-525
> URL: https://issues.apache.org/jira/browse/SLIDER-525
> Project: Slider
>  Issue Type: Bug
>  Components: app-package
>Affects Versions: Slider 0.50
> Environment: Windows
>Reporter: Sumit Mohanty
>Assignee: Sumit Mohanty
> Fix For: Slider 0.60
>
>
> Storm app package for windows does not report component status correctly



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (SLIDER-544) Release Slider 0.6.0

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SLIDER-544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199023#comment-14199023
 ] 

ASF subversion and git services commented on SLIDER-544:


Commit 844e4b6776c627f5eddeb22b233eff0da9cea55b in incubator-slider's branch 
refs/heads/feature/SLIDER-616_re-enable_AMFailuresIT from [~ste...@apache.org]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=844e4b6 ]

SLIDER-544 fix up dependencies before releasing


> Release Slider 0.6.0
> 
>
> Key: SLIDER-544
> URL: https://issues.apache.org/jira/browse/SLIDER-544
> Project: Slider
>  Issue Type: Task
>  Components: build
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Fix For: Slider 0.60
>
>   Original Estimate: 4h
>  Remaining Estimate: 4h
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (SLIDER-616) re-enable AMFailuresIT

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SLIDER-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199026#comment-14199026
 ] 

ASF subversion and git services commented on SLIDER-616:


Commit adb98ee86c5ebb0c9867ca83c20bbaec548e5b48 in incubator-slider's branch 
refs/heads/feature/SLIDER-616_re-enable_AMFailuresIT from [~ste...@apache.org]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=adb98ee ]

SLIDER-616 re-enable AMFailuresIT


> re-enable AMFailuresIT
> --
>
> Key: SLIDER-616
> URL: https://issues.apache.org/jira/browse/SLIDER-616
> Project: Slider
>  Issue Type: Test
>  Components: test
>Affects Versions: Slider 0.50
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Fix For: Slider 0.60
>
>
> Re-enable the test {{AMFailuresIT}} ; make it robust against timing 
> differences in AM restart delays &C



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (SLIDER-619) registry list and listconf commands to save output to --out file

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SLIDER-619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199031#comment-14199031
 ] 

ASF subversion and git services commented on SLIDER-619:


Commit 70c1b1a7b21e02abff825009d51ca40ee9f2aca6 in incubator-slider's branch 
refs/heads/develop from [~ste...@apache.org]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=70c1b1a ]

SLIDER-619 registry list and listconf commands to save output to --out file


> registry list and listconf commands to save output to --out file
> 
>
> Key: SLIDER-619
> URL: https://issues.apache.org/jira/browse/SLIDER-619
> Project: Slider
>  Issue Type: Test
>  Components: client
>Affects Versions: Slider 0.50
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Fix For: Slider 0.60
>
>
> To aid testing, have the registry {{--list}} and {{-listconf}} commands 
> support the {{--out}} file entry. If set the output is just saved  as text, 
> one entry per line.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (SLIDER-616) re-enable AMFailuresIT

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SLIDER-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199021#comment-14199021
 ] 

ASF subversion and git services commented on SLIDER-616:


Commit bb28c5146c1e30087aa945af77a1d5533fa16185 in incubator-slider's branch 
refs/heads/feature/SLIDER-616_re-enable_AMFailuresIT from [~ste...@apache.org]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=bb28c51 ]

SLIDER-616 re-enable AMFailuresIT


> re-enable AMFailuresIT
> --
>
> Key: SLIDER-616
> URL: https://issues.apache.org/jira/browse/SLIDER-616
> Project: Slider
>  Issue Type: Test
>  Components: test
>Affects Versions: Slider 0.50
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Fix For: Slider 0.60
>
>
> Re-enable the test {{AMFailuresIT}} ; make it robust against timing 
> differences in AM restart delays &C



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (SLIDER-616) re-enable AMFailuresIT

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SLIDER-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14199020#comment-14199020
 ] 

ASF subversion and git services commented on SLIDER-616:


Commit da73298d717692f357a4e8e4a247de902d1f7de6 in incubator-slider's branch 
refs/heads/feature/SLIDER-616_re-enable_AMFailuresIT from [~ste...@apache.org]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=da73298 ]

SLIDER-616 AMFailuresIT


> re-enable AMFailuresIT
> --
>
> Key: SLIDER-616
> URL: https://issues.apache.org/jira/browse/SLIDER-616
> Project: Slider
>  Issue Type: Test
>  Components: test
>Affects Versions: Slider 0.50
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Fix For: Slider 0.60
>
>
> Re-enable the test {{AMFailuresIT}} ; make it robust against timing 
> differences in AM restart delays &C



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (SLIDER-619) registry list and listconf commands to save output to --out file

[Steve Loughran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SLIDER-619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated SLIDER-619:
--
Sprint: Slider November #1

> registry list and listconf commands to save output to --out file
> 
>
> Key: SLIDER-619
> URL: https://issues.apache.org/jira/browse/SLIDER-619
> Project: Slider
>  Issue Type: Test
>  Components: client
>Affects Versions: Slider 0.50
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Fix For: Slider 0.60
>
>
> To aid testing, have the registry {{--list}} and {{-listconf}} commands 
> support the {{--out}} file entry. If set the output is just saved  as text, 
> one entry per line.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (SLIDER-619) registry list and listconf commands to save output to --out file

[Steve Loughran (JIRA)]

Steve Loughran created SLIDER-619:
-

 Summary: registry list and listconf commands to save output to 
--out file
 Key: SLIDER-619
 URL: https://issues.apache.org/jira/browse/SLIDER-619
 Project: Slider
  Issue Type: Test
  Components: client
Affects Versions: Slider 0.50
Reporter: Steve Loughran
Assignee: Steve Loughran
 Fix For: Slider 0.60


To aid testing, have the registry {{--list}} and {{-listconf}} commands support 
the {{--out}} file entry. If set the output is just saved  as text, one entry 
per line.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: slider report

[Ted Yu]

bq. We have added one committer / PMC member.

I think we should mention Gour's name in the above sentence.

lgtm

On Wed, Nov 5, 2014 at 12:12 PM, Billie Rinaldi 
wrote:

> I drafted the following Slider report.  If anyone wants to edit or add
> anything, please do so on the wiki:
> https://wiki.apache.org/incubator/November2014
>
> 
> Slider
>
> Slider is a collection of tools and technologies to package, deploy, and
> manage long running applications on Apache Hadoop YARN clusters.
>
> Slider has been incubating since 2014-04-29.
>
> Three most important issues to address in the move towards graduation:
>
>   1. Increasing community diversity
>   2. Completing podling name search
>   3. Building a user community
>
> Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
> aware of?
>
> No
>
> How has the community developed since the last report?
>
> We have added one committer / PMC member.
>
> How has the project developed since the last report?
>
> Development and mailing lists have been active, working towards a
> major upcoming release.  A service registry has been redesigned and
> contributed to Yarn (see YARN-913), and Slider continues to help
> drive improvements needed for long-lived services in Yarn.
>
> Date of last release:
>
>   2014-07-21
>
> When were the last committers or PMC members elected?
>
>   2014-09-22
>

Is it able to make AM to restart from previous state?

[hsy...@gmail.com]

Hi guys,

I noticed in the code when a container fails it will try to relaunch from
the same node. My question is if I restart whole application(Ex. AM got
killed, or manually restart the app). Does slider try to launch all
containers from the nodes where it was running?

Thanks!

Best,
Siyuan

slider report

[Billie Rinaldi]

I drafted the following Slider report.  If anyone wants to edit or add
anything, please do so on the wiki:
https://wiki.apache.org/incubator/November2014


Slider

Slider is a collection of tools and technologies to package, deploy, and
manage long running applications on Apache Hadoop YARN clusters.

Slider has been incubating since 2014-04-29.

Three most important issues to address in the move towards graduation:

  1. Increasing community diversity
  2. Completing podling name search
  3. Building a user community

Any issues that the Incubator PMC (IPMC) or ASF Board wish/need to be
aware of?

No

How has the community developed since the last report?

We have added one committer / PMC member.

How has the project developed since the last report?

Development and mailing lists have been active, working towards a
major upcoming release.  A service registry has been redesigned and
contributed to Yarn (see YARN-913), and Slider continues to help
drive improvements needed for long-lived services in Yarn.

Date of last release:

  2014-07-21

When were the last committers or PMC members elected?

  2014-09-22

Re: Memcahed doesn't start with latest slider code

[Jon Maron]

As Sumit indicated, the "yarn logs --applicationId ” command 
should dump those logs.

— Jon

On Nov 5, 2014, at 2:13 PM, Pushkar Raste  wrote:

> I am new to both slider and hadoop. Where can I find agent logs?
> 
> On Wed, Nov 5, 2014 at 2:02 PM, Jon Maron  wrote:
> 
>> It may help to provide any of the agent logs or memcached logs from the
>> node managers.  This could occur for any number of reasons including wrong
>> java_home value.
>> 
>> — Jon
>> 
>> On Nov 5, 2014, at 1:55 PM, Pushkar Raste  wrote:
>> 
>>> May I should provide entire log
>>> 
>>> 2014-11-05 18:27:24,345 [main] INFO  Configuration.deprecation -
>>> slider.registry.path is deprecated. Instead, use hadoop.registry.zk.root
>>> 2014-11-05 18:27:24,350 [main] INFO  appmaster.SliderAppMaster - AM
>>> configuration:
>>> fs.defaultFS=hdfs://localhost:9000
>>> hadoop.registry.zk.quorum=localhost:2181
>>> hadoop.registry.zk.root=/registry
>>> slider.registry.path=/registry
>>> slider.yarn.queue=default
>>> 
>> yarn.application.classpath=/usr/local/hadoop/etc/hadoop,/usr/local/hadoop/etc/hadoop/*,/usr/local/hadoop/share/hadoop/common/*,/usr/local/hadoop/share/hadoop/common/lib/*,/usr/local/hadoop/share/hadoop/hdfs/*,/usr/local/hadoop/share/hadoop/hdfs/lib/*,/usr/local/hadoop/share/hadoop/yarn/*,/usr/local/hadoop/share/hadoop/yarn/lib/*,/usr/local/hadoop/share/hadoop/mapreduce/*,/usr/local/hadoop/share/hadoop/mapreduce/lib/*
>>> yarn.log-aggregation-enable=true
>>> yarn.resourcemanager.address=localhost:8032
>>> yarn.resourcemanager.scheduler.address=localhost:8030
>>> 
>>> 2014-11-05 18:27:24,507 [main] INFO  appmaster.SliderAppMaster - Cluster
>> is
>>> insecure
>>> 2014-11-05 18:27:25,000 [main] INFO  appmaster.SliderAppMaster - Login
>> user
>>> is root (auth:SIMPLE)
>>> 2014-11-05 18:27:25,013 [openssl-001] INFO  appmaster.SliderAppMaster -
>>> OpenSSL 1.0.1 14 Mar 2012
>>> 2014-11-05 18:27:25,213 [openssl-001] WARN  appmaster.SliderAppMaster -
>>> 2014-11-05 18:27:25,214 [openssl-001] INFO  appmaster.SliderAppMaster -
>>> 2014-11-05 18:27:25,231 [python-003] WARN  appmaster.SliderAppMaster -
>>> Python 2.7.3
>>> 2014-11-05 18:27:25,431 [python-003] WARN  appmaster.SliderAppMaster -
>>> 2014-11-05 18:27:25,431 [python-003] INFO  appmaster.SliderAppMaster -
>>> 2014-11-05 18:27:25,434 [main] INFO  appmaster.SliderAppMaster - Slider
>>> Core-0.51.0-incubating-SNAPSHOT Built against commit# bbde42bdf9 on Java
>>> 1.7.0_67 by praste
>>> 2014-11-05 18:27:25,434 [main] INFO  appmaster.SliderAppMaster - Compiled
>>> against Hadoop 2.6.0-SNAPSHOT
>>> 2014-11-05 18:27:25,436 [main] INFO  appmaster.SliderAppMaster - Hadoop
>>> runtime version (detached from b4446cb) with source checksum
>>> d2d3ea14a0fdbf31a0273fc4f2ad594b and build date 2014-10-29T18:31Z
>>> 2014-11-05 18:27:25,437 [main] INFO  appmaster.SliderAppMaster -
>>> Application defined at
>> hdfs://localhost:9000/user/root/.slider/cluster/cl2
>>> 2014-11-05 18:27:27,195 [main] INFO  appmaster.SliderAppMaster -
>> Deploying
>>> cluster {,
>>> "internal": {
>>> "schema" : "http://example.org/specification/v2.0.0";,
>>> "metadata" : {
>>>   "create.hadoop.deployed.info" : "(detached from b4446cb)
>>> @d2d3ea14a0fdbf31a0273fc4f2ad594b",
>>>   "create.application.build.info" : "Slider
>>> Core-0.51.0-incubating-SNAPSHOT Built against commit# bbde42bdf9 on Java
>>> 1.7.0_67 by praste",
>>>   "create.hadoop.build.info" : "2.6.0-SNAPSHOT",
>>>   "create.time.millis" : "1415212024534",
>>>   "create.time" : "5 Nov 2014 18:27:04 GMT"
>>> },
>>> "global" : {
>>>   "internal.tmp.dir" :
>>> "hdfs://localhost:9000/user/root/.slider/cluster/cl2/tmp",
>>>   "internal.generated.conf.path" :
>>> "hdfs://localhost:9000/user/root/.slider/cluster/cl2/generated",
>>>   "internal.snapshot.conf.path" :
>>> "hdfs://localhost:9000/user/root/.slider/cluster/cl2/snapshot",
>>>   "internal.container.failure.shortlife" : "6",
>>>   "slider.data.directory.permissions" : "0770",
>>>   "application.name" : "cl2",
>>>   "slider.cluster.directory.permissions" : "0770",
>>>   "internal.provider.name" : "agent",
>>>   "internal.am.tmp.dir" :
>>> "hdfs://localhost:9000/user/root/.slider/cluster/cl2/tmp/appmaster",
>>>   "internal.container.failure.threshold" : "5",
>>>   "internal.data.dir.path" :
>>> "hdfs://localhost:9000/user/root/.slider/cluster/cl2/database"
>>> },
>>> "credentials" : { },
>>> "components" : { }
>>> },
>>> "resources": {
>>> "schema" : "http://example.org/specification/v2.0.0";,
>>> "metadata" : { },
>>> "global" : { },
>>> "credentials" : { },
>>> "components" : {
>>>   "slider-appmaster" : {
>>> "yarn.memory" : "256",
>>> "yarn.vcores" : "1",
>>> "yarn.component.instances" : "1"
>>>   },
>>>   "MEMCACHED" : {
>>> "yarn.memory" : "256",
>>> "yarn.role.priority" : "1",
>>> "yarn.component.instances" : "1"
>>>   }
>>> }
>>> },
>>> "appConf" :{
>>> "schema" : "http://example.org/specification/v2.0.0";,
>>> "metadata" : { },
>>> "global" : {
>>>   "site.fs.default.name" : "hdfs:

Re: Memcahed doesn't start with latest slider code

[Pushkar Raste]

I am new to both slider and hadoop. Where can I find agent logs?

On Wed, Nov 5, 2014 at 2:02 PM, Jon Maron  wrote:

> It may help to provide any of the agent logs or memcached logs from the
> node managers.  This could occur for any number of reasons including wrong
> java_home value.
>
> — Jon
>
> On Nov 5, 2014, at 1:55 PM, Pushkar Raste  wrote:
>
> > May I should provide entire log
> >
> > 2014-11-05 18:27:24,345 [main] INFO  Configuration.deprecation -
> > slider.registry.path is deprecated. Instead, use hadoop.registry.zk.root
> > 2014-11-05 18:27:24,350 [main] INFO  appmaster.SliderAppMaster - AM
> > configuration:
> > fs.defaultFS=hdfs://localhost:9000
> > hadoop.registry.zk.quorum=localhost:2181
> > hadoop.registry.zk.root=/registry
> > slider.registry.path=/registry
> > slider.yarn.queue=default
> >
> yarn.application.classpath=/usr/local/hadoop/etc/hadoop,/usr/local/hadoop/etc/hadoop/*,/usr/local/hadoop/share/hadoop/common/*,/usr/local/hadoop/share/hadoop/common/lib/*,/usr/local/hadoop/share/hadoop/hdfs/*,/usr/local/hadoop/share/hadoop/hdfs/lib/*,/usr/local/hadoop/share/hadoop/yarn/*,/usr/local/hadoop/share/hadoop/yarn/lib/*,/usr/local/hadoop/share/hadoop/mapreduce/*,/usr/local/hadoop/share/hadoop/mapreduce/lib/*
> > yarn.log-aggregation-enable=true
> > yarn.resourcemanager.address=localhost:8032
> > yarn.resourcemanager.scheduler.address=localhost:8030
> >
> > 2014-11-05 18:27:24,507 [main] INFO  appmaster.SliderAppMaster - Cluster
> is
> > insecure
> > 2014-11-05 18:27:25,000 [main] INFO  appmaster.SliderAppMaster - Login
> user
> > is root (auth:SIMPLE)
> > 2014-11-05 18:27:25,013 [openssl-001] INFO  appmaster.SliderAppMaster -
> > OpenSSL 1.0.1 14 Mar 2012
> > 2014-11-05 18:27:25,213 [openssl-001] WARN  appmaster.SliderAppMaster -
> > 2014-11-05 18:27:25,214 [openssl-001] INFO  appmaster.SliderAppMaster -
> > 2014-11-05 18:27:25,231 [python-003] WARN  appmaster.SliderAppMaster -
> > Python 2.7.3
> > 2014-11-05 18:27:25,431 [python-003] WARN  appmaster.SliderAppMaster -
> > 2014-11-05 18:27:25,431 [python-003] INFO  appmaster.SliderAppMaster -
> > 2014-11-05 18:27:25,434 [main] INFO  appmaster.SliderAppMaster - Slider
> > Core-0.51.0-incubating-SNAPSHOT Built against commit# bbde42bdf9 on Java
> > 1.7.0_67 by praste
> > 2014-11-05 18:27:25,434 [main] INFO  appmaster.SliderAppMaster - Compiled
> > against Hadoop 2.6.0-SNAPSHOT
> > 2014-11-05 18:27:25,436 [main] INFO  appmaster.SliderAppMaster - Hadoop
> > runtime version (detached from b4446cb) with source checksum
> > d2d3ea14a0fdbf31a0273fc4f2ad594b and build date 2014-10-29T18:31Z
> > 2014-11-05 18:27:25,437 [main] INFO  appmaster.SliderAppMaster -
> > Application defined at
> hdfs://localhost:9000/user/root/.slider/cluster/cl2
> > 2014-11-05 18:27:27,195 [main] INFO  appmaster.SliderAppMaster -
> Deploying
> > cluster {,
> > "internal": {
> >  "schema" : "http://example.org/specification/v2.0.0";,
> >  "metadata" : {
> >"create.hadoop.deployed.info" : "(detached from b4446cb)
> > @d2d3ea14a0fdbf31a0273fc4f2ad594b",
> >"create.application.build.info" : "Slider
> > Core-0.51.0-incubating-SNAPSHOT Built against commit# bbde42bdf9 on Java
> > 1.7.0_67 by praste",
> >"create.hadoop.build.info" : "2.6.0-SNAPSHOT",
> >"create.time.millis" : "1415212024534",
> >"create.time" : "5 Nov 2014 18:27:04 GMT"
> >  },
> >  "global" : {
> >"internal.tmp.dir" :
> > "hdfs://localhost:9000/user/root/.slider/cluster/cl2/tmp",
> >"internal.generated.conf.path" :
> > "hdfs://localhost:9000/user/root/.slider/cluster/cl2/generated",
> >"internal.snapshot.conf.path" :
> > "hdfs://localhost:9000/user/root/.slider/cluster/cl2/snapshot",
> >"internal.container.failure.shortlife" : "6",
> >"slider.data.directory.permissions" : "0770",
> >"application.name" : "cl2",
> >"slider.cluster.directory.permissions" : "0770",
> >"internal.provider.name" : "agent",
> >"internal.am.tmp.dir" :
> > "hdfs://localhost:9000/user/root/.slider/cluster/cl2/tmp/appmaster",
> >"internal.container.failure.threshold" : "5",
> >"internal.data.dir.path" :
> > "hdfs://localhost:9000/user/root/.slider/cluster/cl2/database"
> >  },
> >  "credentials" : { },
> >  "components" : { }
> > },
> > "resources": {
> >  "schema" : "http://example.org/specification/v2.0.0";,
> >  "metadata" : { },
> >  "global" : { },
> >  "credentials" : { },
> >  "components" : {
> >"slider-appmaster" : {
> >  "yarn.memory" : "256",
> >  "yarn.vcores" : "1",
> >  "yarn.component.instances" : "1"
> >},
> >"MEMCACHED" : {
> >  "yarn.memory" : "256",
> >  "yarn.role.priority" : "1",
> >  "yarn.component.instances" : "1"
> >}
> >  }
> > },
> > "appConf" :{
> >  "schema" : "http://example.org/specification/v2.0.0";,
> >  "metadata" : { },
> >  "global" : {
> >"site.fs.default.name" : "hdfs://localhost:9000",
> >"site.global.app_user" : "yarn",
> >"site.global.additional_cp" : "/usr/lib/hadoop/lib/*",
> >"

Re: Memcahed doesn't start with latest slider code

[Sumit Mohanty]

These three properties in appConfig.json will be my initial suspect (I am
using the default values from the appConfig-default.json in the git repo):

"application.def": ".slider/package/MEMCACHED/jmemcached-1.0.0.zip",
Is the app pkg in the correct location in HDFS? The above path is relative
to user's home directory in HDFS.

"java_home": "/usr/jdk64/jdk1.7.0_67",
Is java home valid?

"site.global.additional_cp": "/usr/lib/hadoop/lib/*",
There are a few more jars needed by memcached and they are typically in
this path of a regular hadoop installation.

"yarn logs --applicationId  will provide you all the
gathered logs from the app run.

On Wed, Nov 5, 2014 at 11:02 AM, Jon Maron  wrote:

> It may help to provide any of the agent logs or memcached logs from the
> node managers.  This could occur for any number of reasons including wrong
> java_home value.
>
> — Jon
>
> On Nov 5, 2014, at 1:55 PM, Pushkar Raste  wrote:
>
> > May I should provide entire log
> >
> > 2014-11-05 18:27:24,345 [main] INFO  Configuration.deprecation -
> > slider.registry.path is deprecated. Instead, use hadoop.registry.zk.root
> > 2014-11-05 18:27:24,350 [main] INFO  appmaster.SliderAppMaster - AM
> > configuration:
> > fs.defaultFS=hdfs://localhost:9000
> > hadoop.registry.zk.quorum=localhost:2181
> > hadoop.registry.zk.root=/registry
> > slider.registry.path=/registry
> > slider.yarn.queue=default
> >
> yarn.application.classpath=/usr/local/hadoop/etc/hadoop,/usr/local/hadoop/etc/hadoop/*,/usr/local/hadoop/share/hadoop/common/*,/usr/local/hadoop/share/hadoop/common/lib/*,/usr/local/hadoop/share/hadoop/hdfs/*,/usr/local/hadoop/share/hadoop/hdfs/lib/*,/usr/local/hadoop/share/hadoop/yarn/*,/usr/local/hadoop/share/hadoop/yarn/lib/*,/usr/local/hadoop/share/hadoop/mapreduce/*,/usr/local/hadoop/share/hadoop/mapreduce/lib/*
> > yarn.log-aggregation-enable=true
> > yarn.resourcemanager.address=localhost:8032
> > yarn.resourcemanager.scheduler.address=localhost:8030
> >
> > 2014-11-05 18:27:24,507 [main] INFO  appmaster.SliderAppMaster - Cluster
> is
> > insecure
> > 2014-11-05 18:27:25,000 [main] INFO  appmaster.SliderAppMaster - Login
> user
> > is root (auth:SIMPLE)
> > 2014-11-05 18:27:25,013 [openssl-001] INFO  appmaster.SliderAppMaster -
> > OpenSSL 1.0.1 14 Mar 2012
> > 2014-11-05 18:27:25,213 [openssl-001] WARN  appmaster.SliderAppMaster -
> > 2014-11-05 18:27:25,214 [openssl-001] INFO  appmaster.SliderAppMaster -
> > 2014-11-05 18:27:25,231 [python-003] WARN  appmaster.SliderAppMaster -
> > Python 2.7.3
> > 2014-11-05 18:27:25,431 [python-003] WARN  appmaster.SliderAppMaster -
> > 2014-11-05 18:27:25,431 [python-003] INFO  appmaster.SliderAppMaster -
> > 2014-11-05 18:27:25,434 [main] INFO  appmaster.SliderAppMaster - Slider
> > Core-0.51.0-incubating-SNAPSHOT Built against commit# bbde42bdf9 on Java
> > 1.7.0_67 by praste
> > 2014-11-05 18:27:25,434 [main] INFO  appmaster.SliderAppMaster - Compiled
> > against Hadoop 2.6.0-SNAPSHOT
> > 2014-11-05 18:27:25,436 [main] INFO  appmaster.SliderAppMaster - Hadoop
> > runtime version (detached from b4446cb) with source checksum
> > d2d3ea14a0fdbf31a0273fc4f2ad594b and build date 2014-10-29T18:31Z
> > 2014-11-05 18:27:25,437 [main] INFO  appmaster.SliderAppMaster -
> > Application defined at
> hdfs://localhost:9000/user/root/.slider/cluster/cl2
> > 2014-11-05 18:27:27,195 [main] INFO  appmaster.SliderAppMaster -
> Deploying
> > cluster {,
> > "internal": {
> >  "schema" : "http://example.org/specification/v2.0.0";,
> >  "metadata" : {
> >"create.hadoop.deployed.info" : "(detached from b4446cb)
> > @d2d3ea14a0fdbf31a0273fc4f2ad594b",
> >"create.application.build.info" : "Slider
> > Core-0.51.0-incubating-SNAPSHOT Built against commit# bbde42bdf9 on Java
> > 1.7.0_67 by praste",
> >"create.hadoop.build.info" : "2.6.0-SNAPSHOT",
> >"create.time.millis" : "1415212024534",
> >"create.time" : "5 Nov 2014 18:27:04 GMT"
> >  },
> >  "global" : {
> >"internal.tmp.dir" :
> > "hdfs://localhost:9000/user/root/.slider/cluster/cl2/tmp",
> >"internal.generated.conf.path" :
> > "hdfs://localhost:9000/user/root/.slider/cluster/cl2/generated",
> >"internal.snapshot.conf.path" :
> > "hdfs://localhost:9000/user/root/.slider/cluster/cl2/snapshot",
> >"internal.container.failure.shortlife" : "6",
> >"slider.data.directory.permissions" : "0770",
> >"application.name" : "cl2",
> >"slider.cluster.directory.permissions" : "0770",
> >"internal.provider.name" : "agent",
> >"internal.am.tmp.dir" :
> > "hdfs://localhost:9000/user/root/.slider/cluster/cl2/tmp/appmaster",
> >"internal.container.failure.threshold" : "5",
> >"internal.data.dir.path" :
> > "hdfs://localhost:9000/user/root/.slider/cluster/cl2/database"
> >  },
> >  "credentials" : { },
> >  "components" : { }
> > },
> > "resources": {
> >  "schema" : "http://example.org/specification/v2.0.0";,
> >  "metadata" : { },
> >  "global" : { },
> >  "credentials" : { },
> >  "component

Re: Memcahed doesn't start with latest slider code

[Jon Maron]

It may help to provide any of the agent logs or memcached logs from the node 
managers.  This could occur for any number of reasons including wrong java_home 
value.

— Jon

On Nov 5, 2014, at 1:55 PM, Pushkar Raste  wrote:

> May I should provide entire log
> 
> 2014-11-05 18:27:24,345 [main] INFO  Configuration.deprecation -
> slider.registry.path is deprecated. Instead, use hadoop.registry.zk.root
> 2014-11-05 18:27:24,350 [main] INFO  appmaster.SliderAppMaster - AM
> configuration:
> fs.defaultFS=hdfs://localhost:9000
> hadoop.registry.zk.quorum=localhost:2181
> hadoop.registry.zk.root=/registry
> slider.registry.path=/registry
> slider.yarn.queue=default
> yarn.application.classpath=/usr/local/hadoop/etc/hadoop,/usr/local/hadoop/etc/hadoop/*,/usr/local/hadoop/share/hadoop/common/*,/usr/local/hadoop/share/hadoop/common/lib/*,/usr/local/hadoop/share/hadoop/hdfs/*,/usr/local/hadoop/share/hadoop/hdfs/lib/*,/usr/local/hadoop/share/hadoop/yarn/*,/usr/local/hadoop/share/hadoop/yarn/lib/*,/usr/local/hadoop/share/hadoop/mapreduce/*,/usr/local/hadoop/share/hadoop/mapreduce/lib/*
> yarn.log-aggregation-enable=true
> yarn.resourcemanager.address=localhost:8032
> yarn.resourcemanager.scheduler.address=localhost:8030
> 
> 2014-11-05 18:27:24,507 [main] INFO  appmaster.SliderAppMaster - Cluster is
> insecure
> 2014-11-05 18:27:25,000 [main] INFO  appmaster.SliderAppMaster - Login user
> is root (auth:SIMPLE)
> 2014-11-05 18:27:25,013 [openssl-001] INFO  appmaster.SliderAppMaster -
> OpenSSL 1.0.1 14 Mar 2012
> 2014-11-05 18:27:25,213 [openssl-001] WARN  appmaster.SliderAppMaster -
> 2014-11-05 18:27:25,214 [openssl-001] INFO  appmaster.SliderAppMaster -
> 2014-11-05 18:27:25,231 [python-003] WARN  appmaster.SliderAppMaster -
> Python 2.7.3
> 2014-11-05 18:27:25,431 [python-003] WARN  appmaster.SliderAppMaster -
> 2014-11-05 18:27:25,431 [python-003] INFO  appmaster.SliderAppMaster -
> 2014-11-05 18:27:25,434 [main] INFO  appmaster.SliderAppMaster - Slider
> Core-0.51.0-incubating-SNAPSHOT Built against commit# bbde42bdf9 on Java
> 1.7.0_67 by praste
> 2014-11-05 18:27:25,434 [main] INFO  appmaster.SliderAppMaster - Compiled
> against Hadoop 2.6.0-SNAPSHOT
> 2014-11-05 18:27:25,436 [main] INFO  appmaster.SliderAppMaster - Hadoop
> runtime version (detached from b4446cb) with source checksum
> d2d3ea14a0fdbf31a0273fc4f2ad594b and build date 2014-10-29T18:31Z
> 2014-11-05 18:27:25,437 [main] INFO  appmaster.SliderAppMaster -
> Application defined at hdfs://localhost:9000/user/root/.slider/cluster/cl2
> 2014-11-05 18:27:27,195 [main] INFO  appmaster.SliderAppMaster - Deploying
> cluster {,
> "internal": {
>  "schema" : "http://example.org/specification/v2.0.0";,
>  "metadata" : {
>"create.hadoop.deployed.info" : "(detached from b4446cb)
> @d2d3ea14a0fdbf31a0273fc4f2ad594b",
>"create.application.build.info" : "Slider
> Core-0.51.0-incubating-SNAPSHOT Built against commit# bbde42bdf9 on Java
> 1.7.0_67 by praste",
>"create.hadoop.build.info" : "2.6.0-SNAPSHOT",
>"create.time.millis" : "1415212024534",
>"create.time" : "5 Nov 2014 18:27:04 GMT"
>  },
>  "global" : {
>"internal.tmp.dir" :
> "hdfs://localhost:9000/user/root/.slider/cluster/cl2/tmp",
>"internal.generated.conf.path" :
> "hdfs://localhost:9000/user/root/.slider/cluster/cl2/generated",
>"internal.snapshot.conf.path" :
> "hdfs://localhost:9000/user/root/.slider/cluster/cl2/snapshot",
>"internal.container.failure.shortlife" : "6",
>"slider.data.directory.permissions" : "0770",
>"application.name" : "cl2",
>"slider.cluster.directory.permissions" : "0770",
>"internal.provider.name" : "agent",
>"internal.am.tmp.dir" :
> "hdfs://localhost:9000/user/root/.slider/cluster/cl2/tmp/appmaster",
>"internal.container.failure.threshold" : "5",
>"internal.data.dir.path" :
> "hdfs://localhost:9000/user/root/.slider/cluster/cl2/database"
>  },
>  "credentials" : { },
>  "components" : { }
> },
> "resources": {
>  "schema" : "http://example.org/specification/v2.0.0";,
>  "metadata" : { },
>  "global" : { },
>  "credentials" : { },
>  "components" : {
>"slider-appmaster" : {
>  "yarn.memory" : "256",
>  "yarn.vcores" : "1",
>  "yarn.component.instances" : "1"
>},
>"MEMCACHED" : {
>  "yarn.memory" : "256",
>  "yarn.role.priority" : "1",
>  "yarn.component.instances" : "1"
>}
>  }
> },
> "appConf" :{
>  "schema" : "http://example.org/specification/v2.0.0";,
>  "metadata" : { },
>  "global" : {
>"site.fs.default.name" : "hdfs://localhost:9000",
>"site.global.app_user" : "yarn",
>"site.global.additional_cp" : "/usr/lib/hadoop/lib/*",
>"zookeeper.hosts" : "localhost",
>"site.global.pid_file" : "${AGENT_WORK_ROOT}/app/run/component.pid",
>"java_home" : "/usr/lib/jvm/java-7-openjdk-amd64",
>"site.fs.defaultFS" : "hdfs://localhost:9000",
>"env.MALLOC_ARENA_MAX" : "4",
>"zookeeper.path" : "/services/slider/users/root/cl2",
>"site.global.memory_v

Re: Memcahed doesn't start with latest slider code

[Pushkar Raste]

May I should provide entire log

2014-11-05 18:27:24,345 [main] INFO  Configuration.deprecation -
slider.registry.path is deprecated. Instead, use hadoop.registry.zk.root
2014-11-05 18:27:24,350 [main] INFO  appmaster.SliderAppMaster - AM
configuration:
fs.defaultFS=hdfs://localhost:9000
hadoop.registry.zk.quorum=localhost:2181
hadoop.registry.zk.root=/registry
slider.registry.path=/registry
slider.yarn.queue=default
yarn.application.classpath=/usr/local/hadoop/etc/hadoop,/usr/local/hadoop/etc/hadoop/*,/usr/local/hadoop/share/hadoop/common/*,/usr/local/hadoop/share/hadoop/common/lib/*,/usr/local/hadoop/share/hadoop/hdfs/*,/usr/local/hadoop/share/hadoop/hdfs/lib/*,/usr/local/hadoop/share/hadoop/yarn/*,/usr/local/hadoop/share/hadoop/yarn/lib/*,/usr/local/hadoop/share/hadoop/mapreduce/*,/usr/local/hadoop/share/hadoop/mapreduce/lib/*
yarn.log-aggregation-enable=true
yarn.resourcemanager.address=localhost:8032
yarn.resourcemanager.scheduler.address=localhost:8030

2014-11-05 18:27:24,507 [main] INFO  appmaster.SliderAppMaster - Cluster is
insecure
2014-11-05 18:27:25,000 [main] INFO  appmaster.SliderAppMaster - Login user
is root (auth:SIMPLE)
2014-11-05 18:27:25,013 [openssl-001] INFO  appmaster.SliderAppMaster -
OpenSSL 1.0.1 14 Mar 2012
2014-11-05 18:27:25,213 [openssl-001] WARN  appmaster.SliderAppMaster -
2014-11-05 18:27:25,214 [openssl-001] INFO  appmaster.SliderAppMaster -
2014-11-05 18:27:25,231 [python-003] WARN  appmaster.SliderAppMaster -
Python 2.7.3
2014-11-05 18:27:25,431 [python-003] WARN  appmaster.SliderAppMaster -
2014-11-05 18:27:25,431 [python-003] INFO  appmaster.SliderAppMaster -
2014-11-05 18:27:25,434 [main] INFO  appmaster.SliderAppMaster - Slider
Core-0.51.0-incubating-SNAPSHOT Built against commit# bbde42bdf9 on Java
1.7.0_67 by praste
2014-11-05 18:27:25,434 [main] INFO  appmaster.SliderAppMaster - Compiled
against Hadoop 2.6.0-SNAPSHOT
2014-11-05 18:27:25,436 [main] INFO  appmaster.SliderAppMaster - Hadoop
runtime version (detached from b4446cb) with source checksum
d2d3ea14a0fdbf31a0273fc4f2ad594b and build date 2014-10-29T18:31Z
2014-11-05 18:27:25,437 [main] INFO  appmaster.SliderAppMaster -
Application defined at hdfs://localhost:9000/user/root/.slider/cluster/cl2
2014-11-05 18:27:27,195 [main] INFO  appmaster.SliderAppMaster - Deploying
cluster {,
"internal": {
  "schema" : "http://example.org/specification/v2.0.0";,
  "metadata" : {
"create.hadoop.deployed.info" : "(detached from b4446cb)
@d2d3ea14a0fdbf31a0273fc4f2ad594b",
"create.application.build.info" : "Slider
Core-0.51.0-incubating-SNAPSHOT Built against commit# bbde42bdf9 on Java
1.7.0_67 by praste",
"create.hadoop.build.info" : "2.6.0-SNAPSHOT",
"create.time.millis" : "1415212024534",
"create.time" : "5 Nov 2014 18:27:04 GMT"
  },
  "global" : {
"internal.tmp.dir" :
"hdfs://localhost:9000/user/root/.slider/cluster/cl2/tmp",
"internal.generated.conf.path" :
"hdfs://localhost:9000/user/root/.slider/cluster/cl2/generated",
"internal.snapshot.conf.path" :
"hdfs://localhost:9000/user/root/.slider/cluster/cl2/snapshot",
"internal.container.failure.shortlife" : "6",
"slider.data.directory.permissions" : "0770",
"application.name" : "cl2",
"slider.cluster.directory.permissions" : "0770",
"internal.provider.name" : "agent",
"internal.am.tmp.dir" :
"hdfs://localhost:9000/user/root/.slider/cluster/cl2/tmp/appmaster",
"internal.container.failure.threshold" : "5",
"internal.data.dir.path" :
"hdfs://localhost:9000/user/root/.slider/cluster/cl2/database"
  },
  "credentials" : { },
  "components" : { }
},
"resources": {
  "schema" : "http://example.org/specification/v2.0.0";,
  "metadata" : { },
  "global" : { },
  "credentials" : { },
  "components" : {
"slider-appmaster" : {
  "yarn.memory" : "256",
  "yarn.vcores" : "1",
  "yarn.component.instances" : "1"
},
"MEMCACHED" : {
  "yarn.memory" : "256",
  "yarn.role.priority" : "1",
  "yarn.component.instances" : "1"
}
  }
},
"appConf" :{
  "schema" : "http://example.org/specification/v2.0.0";,
  "metadata" : { },
  "global" : {
"site.fs.default.name" : "hdfs://localhost:9000",
"site.global.app_user" : "yarn",
"site.global.additional_cp" : "/usr/lib/hadoop/lib/*",
"zookeeper.hosts" : "localhost",
"site.global.pid_file" : "${AGENT_WORK_ROOT}/app/run/component.pid",
"java_home" : "/usr/lib/jvm/java-7-openjdk-amd64",
"site.fs.defaultFS" : "hdfs://localhost:9000",
"env.MALLOC_ARENA_MAX" : "4",
"zookeeper.path" : "/services/slider/users/root/cl2",
"site.global.memory_val" : "200M",
"site.global.listen_port" :
"${MEMCACHED.ALLOCATED_PORT}{DO_NOT_PROPAGATE}",
"zookeeper.quorum" : "localhost:2181",
"site.global.xmx_val" : "256m",
"site.global.app_root" :
"${AGENT_WORK_ROOT}/app/install/jmemcached-1.0.0",
"application.def" : ".slider/package/memcached/jmemcached-1.0.0.zip",
"site.global.xms_val" : "128m"
  },
  "credentials"

Memcahed doesn't start with latest slider code

[Pushkar Raste]

I tried deploy jmemcached using latest slider built from dev branch. I see
following error

2014-11-05 18:28:22,804 [AMRM Callback Handler Thread] ERROR
appmaster.SliderAppMaster - Role instance
RoleInstance{role='MEMCACHED',
id='container_1415211406300_0001_01_02',
container=ContainerID=container_1415211406300_0001_01_02
nodeID=hdfs03:59013 http=hdfs03:8042 priority=1073741825,
createTime=1415212054511, startTime=1415212054606, released=false,
roleId=1, host=hdfs03, hostURL=http://hdfs03:8042, state=5,
exitCode=0, command='python ./infra/agent/slider-agent/agent/main.py
--label container_1415211406300_0001_01_02___MEMCACHED --zk-quorum
localhost:2181 --zk-reg-path
/registry/users/root/services/org-apache-slider/cl2 >
/slider-agent.out 2>&1 ; ', diagnostics='', output=null,
environment=[AGENT_WORK_ROOT="$PWD", HADOOP_USER_NAME="root",
AGENT_LOG_ROOT="", PYTHONPATH="./infra/agent/slider-agent/",
SLIDER_PASSPHRASE="qGnrsDpoLqO9TrXEdpIPQwmTgiUfPlEMj5VAaMmaxAZiS8rS9L",
MALLOC_ARENA_MAX="4"]} failed

[jira] [Created] (SLIDER-618) Keep only one folder for conf

[Sumit Mohanty (JIRA)]

Sumit Mohanty created SLIDER-618:


 Summary: Keep only one folder for conf
 Key: SLIDER-618
 URL: https://issues.apache.org/jira/browse/SLIDER-618
 Project: Slider
  Issue Type: Bug
  Components: build
Affects Versions: Slider 0.50
Reporter: Sumit Mohanty
 Fix For: Slider 2.0.0


There are two folders "conf" and "conf-hdp" that contain default configs for 
Slider. One gets used during rpm build and another during the build of the 
binary tarball.

Both should be merged into one folder.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (SLIDER-617) review conf/ directories in slider-assembly

[Steve Loughran (JIRA)]

Steve Loughran created SLIDER-617:
-

 Summary: review conf/ directories in slider-assembly
 Key: SLIDER-617
 URL: https://issues.apache.org/jira/browse/SLIDER-617
 Project: Slider
  Issue Type: Test
Reporter: Steve Loughran
 Fix For: Slider 0.60


have a look at the assembly/src/conf dirs and make sure we are happy with them



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (SLIDER-585) Localize SSL certs for apps

[Jonathan Maron (JIRA)]

[ 
https://issues.apache.org/jira/browse/SLIDER-585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14198706#comment-14198706
 ] 

Jonathan Maron commented on SLIDER-585:
---

If the application is configured for two way SSL then the trust store may 
potentially require more than once cert (it would need the certs of all 
possibly communication partners)

> Localize SSL certs for apps
> ---
>
> Key: SLIDER-585
> URL: https://issues.apache.org/jira/browse/SLIDER-585
> Project: Slider
>  Issue Type: Improvement
>  Components: security
>Reporter: Billie Rinaldi
>Assignee: Jonathan Maron
> Fix For: Slider 2.0.0
>
>
> See discussion on SLIDER-580.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (SLIDER-525) Storm app package for windows does not report component status correctly

[Sumit Mohanty (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SLIDER-525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sumit Mohanty resolved SLIDER-525.
--
Resolution: Fixed

> Storm app package for windows does not report component status correctly
> 
>
> Key: SLIDER-525
> URL: https://issues.apache.org/jira/browse/SLIDER-525
> Project: Slider
>  Issue Type: Bug
>  Components: app-package
>Affects Versions: Slider 0.50
> Environment: Windows
>Reporter: Sumit Mohanty
>Assignee: Sumit Mohanty
> Fix For: Slider 0.60
>
>
> Storm app package for windows does not report component status correctly



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (SLIDER-525) Storm app package for windows does not report component status correctly

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SLIDER-525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14198646#comment-14198646
 ] 

ASF subversion and git services commented on SLIDER-525:


Commit 66b1aa4a403bfe60c026b63f201f8cdc539f2dfe in incubator-slider's branch 
refs/heads/develop from [~sumitmohanty]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=66b1aa4 ]

SLIDER-525. Storm app package for windows does not report component status 
correctly
HBase Win package has the ability to provide custom ports for Thrift and Thrift2


> Storm app package for windows does not report component status correctly
> 
>
> Key: SLIDER-525
> URL: https://issues.apache.org/jira/browse/SLIDER-525
> Project: Slider
>  Issue Type: Bug
>  Components: app-package
>Affects Versions: Slider 0.50
> Environment: Windows
>Reporter: Sumit Mohanty
>Assignee: Sumit Mohanty
> Fix For: Slider 0.60
>
>
> Storm app package for windows does not report component status correctly



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (SLIDER-614) Release Slider 0.52 against old registry code

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SLIDER-614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14198525#comment-14198525
 ] 

ASF subversion and git services commented on SLIDER-614:


Commit 0fe9f57111bb8c85bf15f6f1960ce0067c1db1af in incubator-slider's branch 
refs/heads/develop from [~ste...@apache.org]
[ https://git-wip-us.apache.org/repos/asf?p=incubator-slider.git;h=0fe9f57 ]

SLIDER-614 updating release POMs for 0.53-incubating-SNAPSHOT


> Release Slider 0.52 against old registry code
> -
>
> Key: SLIDER-614
> URL: https://issues.apache.org/jira/browse/SLIDER-614
> Project: Slider
>  Issue Type: Bug
>  Components: build
>Affects Versions: Slider 0.50
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> Build a 0.52 release against the "classic" (pre-gour-enhanced) registry.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (SLIDER-616) re-enable AMFailuresIT

[Steve Loughran (JIRA)]

Steve Loughran created SLIDER-616:
-

 Summary: re-enable AMFailuresIT
 Key: SLIDER-616
 URL: https://issues.apache.org/jira/browse/SLIDER-616
 Project: Slider
  Issue Type: Test
  Components: test
Affects Versions: Slider 0.50
Reporter: Steve Loughran
Assignee: Steve Loughran
 Fix For: Slider 0.60


Re-enable the test {{AMFailuresIT}} ; make it robust against timing differences 
in AM restart delays &C



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (SLIDER-616) re-enable AMFailuresIT

[Steve Loughran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SLIDER-616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated SLIDER-616:
--
Sprint: Slider November #1

> re-enable AMFailuresIT
> --
>
> Key: SLIDER-616
> URL: https://issues.apache.org/jira/browse/SLIDER-616
> Project: Slider
>  Issue Type: Test
>  Components: test
>Affects Versions: Slider 0.50
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Fix For: Slider 0.60
>
>
> Re-enable the test {{AMFailuresIT}} ; make it robust against timing 
> differences in AM restart delays &C



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (SLIDER-585) Localize SSL certs for apps

[Billie Rinaldi (JIRA)]

[ 
https://issues.apache.org/jira/browse/SLIDER-585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14198507#comment-14198507
 ] 

Billie Rinaldi commented on SLIDER-585:
---

Sounds pretty comprehensive.  It might be better not to allow the files to have 
different names, because then we don't have to handle what to do if the name is 
outside the container dir.

Would a container ever need more than one cert?  (I don't think Accumulo does.)

> Localize SSL certs for apps
> ---
>
> Key: SLIDER-585
> URL: https://issues.apache.org/jira/browse/SLIDER-585
> Project: Slider
>  Issue Type: Improvement
>  Components: security
>Reporter: Billie Rinaldi
>Assignee: Jonathan Maron
> Fix For: Slider 2.0.0
>
>
> See discussion on SLIDER-580.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (SLIDER-614) Release Slider 0.52 against old registry code

[Steve Loughran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SLIDER-614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran resolved SLIDER-614.
---
Resolution: Fixed

done; no major problems, except noticed that there were conflicting versions of 
netty in the lib/ dir; and >1 servlet API JAR.

> Release Slider 0.52 against old registry code
> -
>
> Key: SLIDER-614
> URL: https://issues.apache.org/jira/browse/SLIDER-614
> Project: Slider
>  Issue Type: Bug
>  Components: build
>Affects Versions: Slider 0.50
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> Build a 0.52 release against the "classic" (pre-gour-enhanced) registry.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (SLIDER-585) Localize SSL certs for apps

[Jonathan Maron (JIRA)]

[ 
https://issues.apache.org/jira/browse/SLIDER-585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14198438#comment-14198438
 ] 

Jonathan Maron commented on SLIDER-585:
---

So I imagine the requirements here:

  - indicate the need for a certificate for a given role/component
  - an algorithm for the generation of the CN (could be "CN=, 
OU=")
  - a standard location/keystore name for seeding certificates (e.g. 
${AGENT_WORK_DIR}/certs/appkeystore.jks), or do we need to allow the naming of 
this file?
  - if the certs are leveraged by Java/JSSE processes, the generated keystores 
will need to be designated as truststores (client or master) or keystores 
(master), they'll need passwords (standard or randomly generated and shared via 
CredentialProvider API), etc
 - Others?

Some assumptions:
  - Application components have pre-existing mechanisms for leveraging 
keystores/truststore (e.g. javax.net.ssl system properties, SSL socket factory 
and hostname verifier creation and utilization by HTTP connections, etc).
 - others?


> Localize SSL certs for apps
> ---
>
> Key: SLIDER-585
> URL: https://issues.apache.org/jira/browse/SLIDER-585
> Project: Slider
>  Issue Type: Improvement
>  Components: security
>Reporter: Billie Rinaldi
>Assignee: Jonathan Maron
> Fix For: Slider 2.0.0
>
>
> See discussion on SLIDER-580.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (SLIDER-614) Release Slider 0.52 against old registry code

[Steve Loughran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SLIDER-614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated SLIDER-614:
--
Summary: Release Slider 0.52 against old registry code  (was: Release 
Slider 0.50.2 against old registry code)

> Release Slider 0.52 against old registry code
> -
>
> Key: SLIDER-614
> URL: https://issues.apache.org/jira/browse/SLIDER-614
> Project: Slider
>  Issue Type: Bug
>  Components: build
>Affects Versions: Slider 0.50
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>   Original Estimate: 2h
>  Remaining Estimate: 2h
>
> Build a 0.52 release against the "classic" (pre-gour-enhanced) registry.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (SLIDER-613) client registry retriever can be configured by ssl-client.xml for https connections

[Gour Saha (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SLIDER-613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gour Saha resolved SLIDER-613.
--
Resolution: Fixed

> client registry retriever can be configured by ssl-client.xml for https 
> connections
> ---
>
> Key: SLIDER-613
> URL: https://issues.apache.org/jira/browse/SLIDER-613
> Project: Slider
>  Issue Type: Bug
>Reporter: Jonathan Maron
>Assignee: Jonathan Maron
> Fix For: Slider 0.60
>
>
> To comply with standard hadoop client practices, allow the registry retriever 
> to be configured for HTTPS interactions by having an ssl-client.xml 
> configuration file that can be read from its configuration directory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)