Thanks Nate - that does the trick. With the command-line upgrade + a curl call
to add my new service now I can see it among
http://vmhost05-hbase3.bdva.wandisco.com:8080/api/v1/clusters/DC-2/services
The issues I still see are:
- the service doesn't appear in "Add Service" wizard
- even after getting added by POST'ing
{"ServiceInfo":{"service_name":"NSNN"}} info, I don't see the service in
the dashboard. It might be caused by some inconsistencies in my service
definition as I can use the same procedure to add Tez service, and being
able to see it in the dashboard.
Will dig a bit more into this tomorrow and report back if I hit anything new.
Thank you all guys for your help - quite appreciated!
Cos
On Mon, Dec 08, 2014 at 05:11PM, Nate Cole wrote:
> Cos,
>
> There is an ambari server command that should allow you to switch the
> version. Here are the steps I took to change the cluster version:
>
> 1. Install a cluster as normal (I used cluster HDP-2.1 as the stack).
> 2. In /var/lib/ambari-server/resources/stacks/HDP, copy 2.1 to 2.1.TestStack
> 3. Shutdown ambari.
> 4. ran: ambari-server upgradestack HDP-2.1.TestStack
> 5. Start ambari.
> 6. On the cluster admin page, verified at the top of the page that the
> cluster says "HDP-2.1.TestStack"
>
> Hope that helps,
> Nate
>
> On Dec 8, 2014, at 4:24 PM, Konstantin Boudnik <[email protected]> wrote:
>
> > On Mon, Dec 08, 2014 at 04:14PM, John Speidel wrote:
> >> Cos,
> >>
> >> I don't know why this is occurring. I have not actually ever changed the
> >> stack of a running cluster, my response was a result of my understanding
> >> and talking to others that have done this.
> >>
> >> Hopefully somebody that has actually done this can comment.
> >> In the meantime, I will see if I can find somebody that may may know why
> >> this isn't working for you.
> >
> > Thank you very much - really appreciate your help! In the interest of full
> > disclosure: I am using 1.6.1 server, not sure if this is of any
> > significance.
> >
> > Cos
> >
> >> On Fri, Dec 5, 2014 at 8:38 PM, Konstantin Boudnik <[email protected]> wrote:
> >>
> >>> release Thanks for the info John. I think I am getting there, but not
> >>> completely....
> >>>
> >>> So, here's what I've done:
> >>> - added service, bounced Ambari server
> >>>
> >>> - curl to http://vmhost05-hbase3.bdva.wandisco.com:8080/api/v1/stacks/HDP/
> >>> says that I have new stack
> >>> {
> >>> "href" : "
> >>> http://vmhost05-hbase3.bdva.wandisco.com:8080/api/v1/stacks/HDP/versions/2.1.WANdisco
> >>> ",
> >>> "Versions" : {
> >>> "stack_name" : "HDP",
> >>> "stack_version" : "2.1.WANdisco"
> >>> }
> >>> }
> >>>
> >>> - ran update
> >>> curl -u admin:admin -i -X PUT -H 'X-Requested-By: ambari' -d
> >>> '{"Clusters":{"version":"HDP-2.1.WANdisco"}}'
> >>> http://vmhost05-hbase3.bdva.wandisco.com:8080/api/v1/clusters/DC-2
> >>>
> >>> The result was:
> >>> HTTP/1.1 200 OK
> >>> Set-Cookie: AMBARISESSIONID=vueucwdh8p6mkmcygj803vi9;Path=/
> >>> Expires: Thu, 01 Jan 1970 00:00:00 GMT
> >>> Content-Type: text/plain
> >>> Content-Length: 0
> >>> Server: Jetty(7.6.7.v20120910)
> >>>
> >>> However, running
> >>> curl -u admin:admin
> >>> http://vmhost05-hbase3.bdva.wandisco.com:8080/api/v1/clusters/
> >>>
> >>> Still gives me this....
> >>> {
> >>> "href" : "http://vmhost05-hbase3.bdva.wandisco.com:8080/api/v1/clusters/
> >>> ",
> >>> "items" : [
> >>> {
> >>> "href" : "
> >>> http://vmhost05-hbase3.bdva.wandisco.com:8080/api/v1/clusters/DC-2";,
> >>> "Clusters" : {
> >>> "cluster_name" : "DC-2",
> >>> "version" : "HDP-2.1"
> >>> }
> >>> }
> >>> ]
> >>>
> >>> Any ideas/hints? Thank you very much for your help!
> >>> Cos
> >>>
> >>> On Thu, Dec 04, 2014 at 08:08PM, John Speidel wrote:
> >>>> Cos,
> >>>>
> >>>> Yes, if you are able to push a new stack to the ambari server hosts
> >>>> filesystem, you can use a REST api call to update the cluster to be
> >>>> associated with the new stack.
> >>>>
> >>>> So, the steps would would look like this:
> >>>> - install cluster with existing stack
> >>>> - create new stack with additional services/components
> >>>> - write the new stack to the filesystem of the ambari server host
> >>>> - bounce ambari server
> >>>> -- after the server restart, the hash of the stacks directory is
> >>>> recalculated which will result in the new updated stack to be pushed to
> >>> the
> >>>> agents (not immediate, the check of the stack dir hash occurs every n(not
> >>>> sure of exact value) heartbeats from the agents)
> >>>> - do an update (PUT) on the cluster resource changing the value of the
> >>>> Clusters/version property to point to the new stack
> >>>>
> >>>> When these steps are completed the new service/components should be
> >>>> recognized by ambari.
> >>>>
> >>>> -John
> >>>>
> >>>>
> >>>> On Thu, Dec 4, 2014 at 6:07 PM, Konstantin Boudnik <[email protected]>
> >>> wrote:
> >>>>
> >>>>> Hi John.
> >>>>>
> >>>>> Thanks for more info. I'd like to pry more for a little bit of
> >>> information
> >>>>> (in-lined):
> >>>>>
> >>>>> On Thu, Dec 04, 2014 at 04:19PM, John Speidel wrote:
> >>>>>> Hi Cos,
> >>>>>> As mentioned above, it isn't currently possible to augment an
> >>> existing
> >>>>>> stack at runtime with new services/components. You could possibly
> >>> change
> >>>>>> the stack associated with a running cluster to a new stack which
> >>> contains
> >>>>>> the required services/components, but this new stack definition would
> >>>>> need
> >>>>>> to be available to ambari so it really isn't very dynamic. We
> >>> understand
> >>>>>
> >>>>> Let me describe this scenario as I understood this. Let's say I have
> >>> two
> >>>>> stacks:
> >>>>> - StackA (with usual set of components), currently installed on a
> >>> cluster
> >>>>> - StackB == StackA + ComponentX
> >>>>> So, if I add StackB definition to the Ambari server and restart it,
> >>> will I
> >>>>> be
> >>>>> able to switch from StackA to StackB _without_ resintalling the
> >>> cluster,
> >>>>> and
> >>>>> then add/configure ComponentX's service?
> >>>>>
> >>>>> If my depiction is correct then it might be an acceptable workaround
> >>> in the
> >>>>> absence of dynamic extensibility.
> >>>>>
> >>>>>> that going forward that the stack framework will need to be more
> >>>>> extensible
> >>>>>> and dynamic as additional stack definitions are created. Some of the
> >>>>>> underlying work to make stack processing more flexible has already
> >>> been
> >>>>>> done in anticipation of new requirements in this area.
> >>>>>
> >>>>> Shivaji Dutta, in an offline email exchange, pointed me to AMBARI-7175
> >>> and
> >>>>> AMBARI-7201, which is seemingly going into the right direction, but
> >>> still
> >>>>> would require a stack re-definition, if I read it right.
> >>>>>
> >>>>>> This is actually the second time that this scenario has came up for
> >>> me
> >>>>>> today so it would be nice to get a Jira filed for an enhancement
> >>> where
> >>>>> the
> >>>>>> community can start to discuss the associated use cases.
> >>>>>
> >>>>> I will start a JIRA and put together the set of requirements.
> >>>>>
> >>>>> Regards,
> >>>>> Cos
> >>>>>
> >>>>>> -John
> >>>>>>
> >>>>>> On Tue, Dec 2, 2014 at 2:00 PM, Konstantin Boudnik <[email protected]>
> >>>>> wrote:
> >>>>>>
> >>>>>>> Thanks for chiming in, Erin! Do you know if there's any plan to
> >>> have
> >>>>>>> dynamic
> >>>>>>> extensibility for services? For instance, Cloudera Manager has this
> >>>>>>> concept of
> >>>>>>> Custom Services. And while it was done as an added late fix for
> >>>>> parcels, it
> >>>>>>> provides a way of adding new software components into an existing
> >>>>> Hadoop
> >>>>>>> cluster. Which seems to be a pretty handy concept, considering the
> >>>>> liquid
> >>>>>>> state of the whole Hadoop ecosystem.
> >>>>>>>
> >>>>>>> It seems that this functionality, if desired, is going to be brand
> >>> new
> >>>>> and
> >>>>>>> as
> >>>>>>> such require not just development but also re-consideration at the
> >>>>> design
> >>>>>>> level. Am I right? Appreciate the thoughts!
> >>>>>>>
> >>>>>>> The best part of active-active masters paradigm is that a client
> >>> can
> >>>>> work
> >>>>>>> with
> >>>>>>> either of them without a concern of which one is ahead or behind
> >>> of the
> >>>>>>> rest
> >>>>>>> as they are equal. Essentially this solves some major issues that
> >>>>> pester
> >>>>>>> active-standby solutions.
> >>>>>>>
> >>>>>>> And yes, we do install the services to the cluster's nodes as you
> >>>>>>> suggested in
> >>>>>>> the later part of your email - after all they are just Linux
> >>> daemons
> >>>>>>> (although
> >>>>>>> Ambari tries to hide the fact for whatever reason that is). However
> >>>>> this
> >>>>>>> presents the core of the very issue: Ambari isn't aware about
> >>> services
> >>>>>>> installed outside of its realm, and it doesn't provide any
> >>> monitoring
> >>>>> or
> >>>>>>> life-cycle management for those. And because of that I am trying to
> >>>>> figure
> >>>>>>> out
> >>>>>>> what can be done in this regards to have this dynamism.
> >>>>>>>
> >>>>>>> With regards,
> >>>>>>> Cos
> >>>>>>>
> >>>>>>> On Tue, Dec 02, 2014 at 08:41AM, Erin Boyd wrote:
> >>>>>>>> Hi Cos,
> >>>>>>>> Currently, adding services dynamically is not supported in
> >>> Ambari.
> >>>>> It is
> >>>>>>>> generally done, as you mentioned through the creation of a stack
> >>>>> with the
> >>>>>>>> services definition defined within in. I also don't believe we
> >>>>> currently
> >>>>>>>> support multiple masters on the same cluster. How would one
> >>>>>>> deferientiate
> >>>>>>>> which master it should be using in such an environment?
> >>>>>>>>
> >>>>>>>> Of course services can be installed independent of Ambari on the
> >>>>> hosts
> >>>>>>>> inside the cluster...but I don't think that is what you are
> >>> getting
> >>>>> at.
> >>>>>>> Are
> >>>>>>>> you expecting the services to have a presence on the UI though
> >>>>> installed
> >>>>>>>> outside of Ambari?
> >>>>>>>>
> >>>>>>>> Erin
> >>>>>>>>
> >>>>>>>> ----- Original Message -----
> >>>>>>>> From: "Konstantin Boudnik" <[email protected]>
> >>>>>>>> To: [email protected]
> >>>>>>>> Sent: Monday, December 1, 2014 6:42:30 PM
> >>>>>>>> Subject: About extensibility of Ambari stacks (not inheritance)
> >>>>>>>>
> >>>>>>>> Guys,
> >>>>>>>>
> >>>>>>>> I am looking into possible stack extensibility properties of
> >>> Ambari
> >>>>> (not
> >>>>>>> to be
> >>>>>>>> confused with inheritance), but haven't been able to derive any
> >>> final
> >>>>>>>> conclusions just yet. Hence, I'd appreciate the input from the
> >>> people
> >>>>>>> behind
> >>>>>>>> the system.
> >>>>>>>>
> >>>>>>>> I have a few questions about current state of the Ambari (version
> >>>>> 1.6.1
> >>>>>>> and
> >>>>>>>> coming 1.7, and possibly later?) with regards to ability to
> >>> expand an
> >>>>>>> existing
> >>>>>>>> stack definitions with a 3rd party services and do it in the
> >>> runtime,
> >>>>>>> rather
> >>>>>>>> than only during the installation. We need to be able to run a
> >>>>> multiple
> >>>>>>>> instances of our master service in the cluster, which isn't
> >>> typical
> >>>>> for
> >>>>>>>> "normal" Hadoop concept where only one master can exist for any
> >>>>> giving
> >>>>>>>> service.
> >>>>>>>>
> >>>>>>>> Our use case is to be able to amend an exiting cluster setup
> >>> (HDP,
> >>>>>>> Bigtop,
> >>>>>>>> etc.) with a new service running on top of HDFS; but not to
> >>>>> reinstall the
> >>>>>>>> whole stack. The reason we have the use case is that oftentimes
> >>> our
> >>>>>>> software
> >>>>>>>> is being added to an existing 3rd party environment, as an added
> >>>>> bonus,
> >>>>>>> not
> >>>>>>>> available during the initial planning and setup of the cluster.
> >>>>>>>>
> >>>>>>>> The way I understand Ambari's stack inheritance is that a scion
> >>> stack
> >>>>>>> will
> >>>>>>>> be a brand new entity, e.g. I won't be able to cherry-pick
> >>> services
> >>>>> from
> >>>>>>> it
> >>>>>>>> and add them to the already installed parent stack's cluster,
> >>> right?
> >>>>>>>>
> >>>>>>>> So far from I see it doesn't seem possible without introducing a
> >>>>>>> brand-new
> >>>>>>>> version of a stack e.g. 'stack inheritance'. Which,
> >>> unfortunately,
> >>>>> won't
> >>>>>>> work
> >>>>>>>> as per my explanation of the use above case. I guess another way
> >>> to
> >>>>> look
> >>>>>>> at it
> >>>>>>>> is this: would it be possible to add a component (or a service)
> >>> that
> >>>>> will
> >>>>>>>> override an existing component or a service, but without a need
> >>> to
> >>>>>>> reinstall
> >>>>>>>> the rest of the stack?
> >>>>>>>>
> >>>>>>>> Thanks in advance for any info/ideas.
> >>>>>>>> Cos
> >>>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> CONFIDENTIALITY NOTICE
> >>>>>> NOTICE: This message is intended for the use of the individual or
> >>> entity
> >>>>> to
> >>>>>> which it is addressed and may contain information that is
> >>> confidential,
> >>>>>> privileged and exempt from disclosure under applicable law. If the
> >>> reader
> >>>>>> of this message is not the intended recipient, you are hereby
> >>> notified
> >>>>> that
> >>>>>> any printing, copying, dissemination, distribution, disclosure or
> >>>>>> forwarding of this communication is strictly prohibited. If you have
> >>>>>> received this communication in error, please contact the sender
> >>>>> immediately
> >>>>>> and delete it from your system. Thank You.
> >>>>>
> >>>>>
> >>>>
> >>>> --
> >>>> CONFIDENTIALITY NOTICE
> >>>> NOTICE: This message is intended for the use of the individual or entity
> >>> to
> >>>> which it is addressed and may contain information that is confidential,
> >>>> privileged and exempt from disclosure under applicable law. If the reader
> >>>> of this message is not the intended recipient, you are hereby notified
> >>> that
> >>>> any printing, copying, dissemination, distribution, disclosure or
> >>>> forwarding of this communication is strictly prohibited. If you have
> >>>> received this communication in error, please contact the sender
> >>> immediately
> >>>> and delete it from your system. Thank You.
> >>>
> >>
> >> --
> >> CONFIDENTIALITY NOTICE
> >> NOTICE: This message is intended for the use of the individual or entity
> >> to
> >> which it is addressed and may contain information that is confidential,
> >> privileged and exempt from disclosure under applicable law. If the reader
> >> of this message is not the intended recipient, you are hereby notified
> >> that
> >> any printing, copying, dissemination, distribution, disclosure or
> >> forwarding of this communication is strictly prohibited. If you have
> >> received this communication in error, please contact the sender
> >> immediately
> >> and delete it from your system. Thank You.
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
