Re: [DISCUSS] Ambari Metron Configuration Management consequences and call to action

[JJ Meyer]

Quite late to the party, but with all this great back and forth I felt like
I had to join in :)

I believe SolrCloud uses ZooKeeper to manage most of its configuration
files. When searching, I was only able to find this (
https://cwiki.apache.org/confluence/display/solr/Using+ZooKeeper+to+Manage+Configuration+Files).
I wasn't able to find any initial discussion on their architecture. If we
can find more we still may be able to learn from them.

Also, on the idea of passing a username/password to a Stellar function or
to some shell script. We may want to do it a bit differently or at least
give the option to do it differently. I know supplying the
username/password directly is easy when testing and playing around, but it
probably isn't going to be allowed for a user in production. Maybe we can
also support a credentials file and eventually support encrypting sensitive
values in configs?

Thanks,
JJ

On Sun, Jan 15, 2017 at 1:26 PM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Ha, I was betrayed by copy/paste in Chrome.
>
> On Thu, Jan 12, 2017 at 7:24 PM, Matt Foley  wrote:
>
>> Mike, could you try again on the image, please, making sure it is a
>> simple format (gif, png, or jpeg)?  It got munched, at least in my viewer.
>> Thanks.
>>
>> Casey, responding to some of the questions you raised:
>>
>> I’m going to make a rather strong statement:  We already have a service
>> “to intermediate and handle config update/retrieval”.
>> Furthermore, it:
>> - Correctly handles the problems of distributed services running on
>> multi-node clusters.  (That’s a HARD problem, people, and we shouldn’t try
>> to reinvent the wheel.)
>> - Correctly handles Kerberos security. (That’s kinda hard too, or at
>> least a lot of work.)
>> - It does automatic versioning of configurations, and allows viewing,
>> comparing, and reverting historical configs
>> - It has a capable REST API for all those things.
>> It doesn’t natively integrate Zookeeper storage of configs, but there is
>> a natural place to specify copy to/from Zookeeper for the files desired.
>>
>> It is Ambari.  And we should commit to it, rather than try to re-create
>> such features.
>> Because it has a good REST API, it is perfectly feasible to implement
>> Stellar functions that call it.
>> GUI configuration tools can also use the Ambari APIs, or better yet be
>> integrated in an “Ambari View”. (Eg, see the “Yarn Capacity Scheduler
>> Configuration Tool” example in the Ambari documentation, under “Using
>> Ambari Views”.)
>>
>> Arguments are: Parsimony, Sufficiency, Not reinventing the wheel, and Not
>> spending weeks and weeks of developer time over the next year reinventing
>> the wheel while getting details wrong multiple times…
>>
>> Okay, off soapbox.
>>
>> Casey asked what the config update behavior of Ambari is, and how it will
>> interact with changes made from outside Ambari.
>> The following is from my experience working with the Ambari Mpack for
>> Metron.  I am not otherwise an Ambari expert, so tomorrow I’ll get it
>> reviewed by an Ambari development engineer.
>>
>> Ambari-server runs on one node, and Ambari-agent runs on each of all the
>> nodes.
>> Ambari-server has a private set of py, xml, and template files, which
>> together are used both to generate the Ambari configuration GUI, with
>> defaults, and to generate configuration files (of any needed filetype) for
>> the various Stack components.
>> Ambari-server also has a database where it stores the schema related to
>> these files, so even if you reach in and edit Ambari’s files, it will Error
>> out if the set of parameters or parameter names changes.  The historical
>> information about configuration changes is also stored in the db.
>> For each component (and in the case of Metron, for each topology), there
>> is a python file which controls the logic for these actions, among others:
>> - Install
>> - Start / stop / restart / status
>> - Configure
>>
>> It is actually up to this python code (which we wrote for the Metron
>> Mpack) what happens in each of these API calls.  But the current code, and
>> I believe this is typical of Ambari-managed components, performs a
>> “Configure” action whenever you press the “Save” button after changing a
>> component config in Ambari, and also on each Install and Start or Restart.
>>
>> The Configure action consists of approximately the following sequence
>> (see disclaimer above :-)
>> - Recreate the generated config files, using the template files and the
>> actual configuration most recently set in Ambari
>> o Note this is also under the control of python code that we wrote, and
>> this is the appropriate place to push to ZK if desired.
>> - Propagate those config files to each Ambari-agent, with a command to
>> set them locally
>> - The ambari-agents on each node receive the files and write them to the
>> specified locations on local storage
>>
>> Ambari-server then whines that the updated services should be 

Re: [DISCUSS] Ambari Metron Configuration Management consequences and call to action

[Michael Miklavcic]

Ha, I was betrayed by copy/paste in Chrome.

On Thu, Jan 12, 2017 at 7:24 PM, Matt Foley  wrote:

> Mike, could you try again on the image, please, making sure it is a simple
> format (gif, png, or jpeg)?  It got munched, at least in my viewer.  Thanks.
>
> Casey, responding to some of the questions you raised:
>
> I’m going to make a rather strong statement:  We already have a service
> “to intermediate and handle config update/retrieval”.
> Furthermore, it:
> - Correctly handles the problems of distributed services running on
> multi-node clusters.  (That’s a HARD problem, people, and we shouldn’t try
> to reinvent the wheel.)
> - Correctly handles Kerberos security. (That’s kinda hard too, or at least
> a lot of work.)
> - It does automatic versioning of configurations, and allows viewing,
> comparing, and reverting historical configs
> - It has a capable REST API for all those things.
> It doesn’t natively integrate Zookeeper storage of configs, but there is a
> natural place to specify copy to/from Zookeeper for the files desired.
>
> It is Ambari.  And we should commit to it, rather than try to re-create
> such features.
> Because it has a good REST API, it is perfectly feasible to implement
> Stellar functions that call it.
> GUI configuration tools can also use the Ambari APIs, or better yet be
> integrated in an “Ambari View”. (Eg, see the “Yarn Capacity Scheduler
> Configuration Tool” example in the Ambari documentation, under “Using
> Ambari Views”.)
>
> Arguments are: Parsimony, Sufficiency, Not reinventing the wheel, and Not
> spending weeks and weeks of developer time over the next year reinventing
> the wheel while getting details wrong multiple times…
>
> Okay, off soapbox.
>
> Casey asked what the config update behavior of Ambari is, and how it will
> interact with changes made from outside Ambari.
> The following is from my experience working with the Ambari Mpack for
> Metron.  I am not otherwise an Ambari expert, so tomorrow I’ll get it
> reviewed by an Ambari development engineer.
>
> Ambari-server runs on one node, and Ambari-agent runs on each of all the
> nodes.
> Ambari-server has a private set of py, xml, and template files, which
> together are used both to generate the Ambari configuration GUI, with
> defaults, and to generate configuration files (of any needed filetype) for
> the various Stack components.
> Ambari-server also has a database where it stores the schema related to
> these files, so even if you reach in and edit Ambari’s files, it will Error
> out if the set of parameters or parameter names changes.  The historical
> information about configuration changes is also stored in the db.
> For each component (and in the case of Metron, for each topology), there
> is a python file which controls the logic for these actions, among others:
> - Install
> - Start / stop / restart / status
> - Configure
>
> It is actually up to this python code (which we wrote for the Metron
> Mpack) what happens in each of these API calls.  But the current code, and
> I believe this is typical of Ambari-managed components, performs a
> “Configure” action whenever you press the “Save” button after changing a
> component config in Ambari, and also on each Install and Start or Restart.
>
> The Configure action consists of approximately the following sequence (see
> disclaimer above :-)
> - Recreate the generated config files, using the template files and the
> actual configuration most recently set in Ambari
> o Note this is also under the control of python code that we wrote, and
> this is the appropriate place to push to ZK if desired.
> - Propagate those config files to each Ambari-agent, with a command to set
> them locally
> - The ambari-agents on each node receive the files and write them to the
> specified locations on local storage
>
> Ambari-server then whines that the updated services should be restarted,
> but does not initiate that action itself (unless of course the initiating
> action was a Start command from the administrator).
>
> Make sense?  It’s all quite straightforward in concept, there’s just an
> awful lot of stuff wrapped around that to make it all go smoothly and
> handle the problems when it doesn’t.
>
> There’s additional complexity in that the Ambari-agent also caches (on
> each node) both the template files and COMPILED forms of the python files
> (.pyc) involved in transforming them.  The pyc files incorporate some
> amount of additional info regarding parameter values, but I’m not sure of
> the form.  I don’t think that changes the above in any practical way unless
> you’re trying to cheat Ambari by reaching in and editing its files
> directly.  In that case, you also need to whack the pyc files (on each
> node) to force the data to be reloaded from Ambari-server.  Best solution
> is don’t cheat.
>
> Also, there may be circumstances under which the Ambari-agent will detect
> changes and re-write the latest version it knows of the config files, 

Re: [DISCUSS] Ambari Metron Configuration Management consequences and call to action

[Dima Kovalyov]

Great discussion you have here!

I have a newbie question, what is the case for installing Metron
manually (without Ambari)? The only one I can think of right now is to
integrate Metron with other non-ambari toolset/framework. What are the
others?

- Dima

On 01/13/2017 10:03 PM, Matt Foley wrote:
> Thanks Casey.  And I’m happy to see so much agreement on directions in this 
> thread.
>
> I looked up HBase and its use of ZK.  Basically, it uses ZK for coordination, 
> but not configuration.
> https://blog.cloudera.com/blog/2013/10/what-are-hbase-znodes/ says “HBase has 
> a design policy of using ZooKeeper only for transient data (that is, for 
> coordination and state communication). Thus if the HBase’s ZooKeeper data is 
> removed, only the transient operations are affected — data can continue to be 
> written and read to/from HBase.”
>
> However, config management for distributed apps is specifically one of the 
> uses contemplated for Zookeeper, and its persistent store guarantees 
> recoverability.  So the only constraint in using ZK is that ZK be up and 
> running, which is already required for other aspects of Metron.
>
> Casey brings up a good point below that Metron’s idea of “configuration” 
> needs to go beyond the simple text or list values of typical Hadoop 
> configuration parameters.  Json provides rich data structures that are 
> nevertheless easily editable, and that’s good for Metron.  However, this in 
> itself doesn’t require using ZK for parameter storage, since the Hadoop 
> ecosystem and Ambari do not impose constraints on the type of files used to 
> store configs.
>
> So the main reasons for using ZK in addition to Ambari are:
> - Supports asynchronous change notification for subscribed watchers, which is 
> (fairly) easy to use in Metron via Curator framework, thus enabling config 
> update without restarting processes, for selected config parameters.
> - Easy propagation of changes to distributed services, when Ambari is not in 
> use (manual install case)
>
>
> On 1/13/17, 11:16 AM, "Casey Stella"  wrote:
>
> Ok, I'll try to give the historical context for the parser and enrichment
> and indexing topologies since I was the one who made a lot of the 
> mistakes.
> ;)
> 
> In the beginning there was only configuration changes at topology start 
> and
> this was done via those flux properties.  As the configs got more
> complicated and more multi-dimensional, it became evident that properties
> files weren't sufficient to contain them, so we started contemplating
> externalizing the configs in JSON files.  During this investigation, we
> contemplated using zookeeper because it was one of the usecases for that
> tool, apache curator is awesome and it would enable us, later on, to
> support runtime changes without topology restart.  So, pieces of the 
> config
> that were the most important to change ended up getting implemented into
> zookeeper.  We did not go back and consider deeply the remaining configs 
> in
> the flux properties for potential retrofit into zookeeper.  We should
> probably do eventually for consistency sake.  That being said, I don't 
> know
> of any addition to the flux properties for enrichment for a looong time, 
> so
> new configs end up in zookeeper.
> 
> I can't speak about the rationale for some of the profiler's flux
> properties; I'll defer to Nick to cover that if there are questions.
> 
> On Fri, Jan 13, 2017 at 11:58 AM, Matt Foley  wrote:
> 
> > Gosh I hate being two hours behind you guys! :-)
> >
> > I’ll go back through the thread and collect open questions, but wanted 
> to
> > put a word in about Zookeeper integration.
> > I had been about to ask what the benefits of using ZK are, and I’ve now
> > heard two features:
> > - A logical “single place” that is visible and efficiently accessible by
> > all processes on all nodes.
> > - Supports async notification, and therefore updates without restarts.
> >
> > If only the first point was there, it could be replaced by Ambari, 
> because
> > Ambari manages propagating configs so they look local to all nodes.  And
> > btw, there’s no bottlenecking, because clients don’t call the Ambari 
> REST
> > APIs to *read* configs, just to *change* them.  For reading established
> > configs, clients just go to those local files, which are managed by the
> > ambari-agents.
> >
> > But the ability to use ZK and Curator to support async config updates, 
> as
> > we do, is really important.  In fact, I think if we make our use of it
> > consistent we might offer an example to the Ambari team for a general
> > feature they could adopt, precisely to support config changes without
> > restart.
> >
> > Three additional considerations:
> >
> > 1. HBase has always used ZK for various