@jeremy montgomery <alericmcke...@gmail.com> thanks that would be amazing if you could add what you know. The docs are a little scarce.. I'll add what I learn as well as I flush out BigTop mpack so Ambari can still have a future.
On Thu, May 14, 2020 at 9:22 AM jeremy montgomery <alericmcke...@gmail.com> wrote: > @aaron - i don't think that the server/agent piece needs to be messed > with. It is more puppet than ansible, but it works pretty well. It has a > few issues that need to be worked out, but nothing insurmountable. > > Agent/Server (pieces that need work) > > - Ability to invalidate the agent cache. > - Multithreading - Currently alerts/restarts work on a concept of > threads and timeouts. You can end up with too many concurrent tasks/alerts > and things get backlogged waiting on the timeouts. However, if you up the > threads you run into a lot of race conditions with auto-restart and alerts > firing concurrently. ----> main piece here is it needs a prioritized > queuing system locally. > - Order of restarts and hierarchy - Right now this is hard coded at > the server level within a component and somewhat for full restarts based on > the order on the left hand bar. > - Need the ability to modify the full restart sequence > - Need the ability to state whether a restart is blocking, blocking > only when component is on the same server, and non-blocking. > - Need the ability to chain 2 different components together. > Example, if i restart a kerberized hiveserver2 i can have it restart > hue as > well after it is restarted. Otherwise hue will error out since the > spnego > hashes are different. > > I'll make some edits to the confluence page explaining how the > stacks/mpacks/extensions work. > > On Tue, May 12, 2020 at 10:51 AM Matt Andruff <matt.andr...@gmail.com> > wrote: > >> @Aaron Bossert - For that type of change (Ansible) you probably need to >> write up a design paper. (I have seen this done, and I'm not sure it's >> required but seems to be common practice) Sounds like ansible would be a >> good fit. >> >> @Jeremy - thanks for the insights, I might ping you if I get stuck trying >> to get into it. I certainly wish there was a clean standard for >> installation locations. >> >> >> >> On Tue, May 12, 2020 at 10:36 AM Aaron Bossert <aa...@punchcyber.com> >> wrote: >> >>> Jeremy, >>> >>> >>> >>> I would be happy to tackle this with you and whoever else is >>> willing…that being said, I have always had a love/hate relationship with >>> Ambari…the stacks have always been intimidating to me….you mention >>> revisiting the architecture….do you have any ideas for what might be >>> better? In my opinion, I have always wanted to replace the guts of ambari >>> with a system that is underpinned by Ansible…seems much more user-friendly >>> and MUCH easier to configure and extend for those who are not comfortable >>> with doing their own Java to Python and creating RPM’s and the like….What >>> are your thoughts? >>> >>> >>> >>> >>> >>> -- M. Aaron Bossert >>> >>> >>> >>> *From: *jeremy montgomery <alericmcke...@gmail.com> >>> *Reply-To: *"user@ambari.apache.org" <user@ambari.apache.org> >>> *Date: *Tuesday, May 12, 2020 at 9:14 AM >>> *To: *"user@ambari.apache.org" <user@ambari.apache.org> >>> *Subject: *Re: life after Hortonworks >>> >>> >>> >>> I make mpacks all the time, but tend to push them as extensions because >>> the stack architecture is byzantine. >>> >>> - It is java that deploys python code. As such, the structure >>> behaves like java, which makes it a pain in the rear to track down simple >>> python functions. Sometimes the code ends up in common-services, >>> sometimes >>> it stays in the current version of stacks. Some stack components only >>> use >>> code in their latest version, some have code files strung through the >>> folders of 8 previous versions and 3 stacks (looking at you LLAP). >>> - The install code doesn't know where it wants to be. sometimes its >>> in the stack, sometimes it is in ambari. This means that a version of >>> ambari tends to be hard coded to 1-2 versions of a stack. >>> - Changing javascript for a stack isn't possible. This means that >>> stack components are hard coded into the ember.js with a bunch of if >>> statements. >>> - Making additions to a stack feature is a major process (like >>> adding hbase thrift to an existing hbase installation) >>> - Upgrades require a yum file regardless of installation method. so >>> learning how to create dummy rpms is necessary. >>> >>> I'd be willing to chip in and maintain the stack python code but the >>> stack architecture really needs to be revisited. >>> >>> >>> >>> On Mon, May 11, 2020 at 1:50 PM Aaron Bossert <aa...@punchcyber.com> >>> wrote: >>> >>> Matt, >>> >>> >>> >>> Yeah, my thought was to start with whatever the most recent HDP/HDF >>> stack definition as a starting point. It just so happens that I have a >>> backburner project to do this already. I have been using Hortonworks for a >>> long time, but have found recently that I needed to install newer versions >>> of Apache Druid and Apache Storm, which would require me to do a new >>> stack…Full disclosure: I have NEVER mucked with stacks and am not a Python >>> guy…I write in Scala/Java predominantly….that being said, I would be happy >>> to collaborate on this if anyone feels that this would be worthwhile and >>> useful to the broader community. >>> >>> >>> >>> -- M. Aaron Bossert >>> >>> >>> >>> *From: *Matt Andruff <matt.andr...@gmail.com> >>> *Reply-To: *"user@ambari.apache.org" <user@ambari.apache.org> >>> *Date: *Monday, May 11, 2020 at 1:46 PM >>> *To: *"user@ambari.apache.org" <user@ambari.apache.org> >>> *Subject: *Re: life after Hortonworks >>> >>> >>> >>> Cloudera no longer uses ambari. They stuck with Cloudera manager in >>> their release of CDP.(CDH+HDP=CDP) >>> >>> >>> >>> https://docs.cloudera.com/cdpdc/7.0/overview/topics/cdpdc-overview.html >>> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.cloudera.com%2Fcdpdc%2F7.0%2Foverview%2Ftopics%2Fcdpdc-overview.html&data=01%7C01%7Caaron%40punchcyber.com%7C89b759ffe21a47ee874a08d7f6766280%7Cf7ca96216ba34d52a241e0ed3ea6a78a%7C0&sdata=7%2BMaRSRcCIjjKVT777pp18IYYsXEAGk3VPdNuOexV2w%3D&reserved=0> >>> >>> >>> >>> I don't think this means that Ambari is dead. I do think it means as >>> stated that the community will need to take on packaging a stack and >>> building rpms.(or at least packaging the stack) >>> >>> >>> >>> The legacy code for stack of HDP 2.6 stream is out there already in the >>> repo so it's just some work to create rpms of whatever Ambari wants to >>> release. >>> >>> >>> >>> I assume the stacks section is so poorly documented is because >>> Hortonworks was doing the work of packaging. I'm not sure the level of >>> effort needed to make a stack work but it seems like we could start with >>> the last HDP build (3.1.4) and keep moving forward. >>> >>> >>> >>> >>> >>> On Mon., May 11, 2020, 13:08 Stephen Boesch, <java...@gmail.com> wrote: >>> >>> I am reading between the lines that ambari is no longer a strategic >>> platform. Would someone please provide a link/reference to a Cloudera press >>> release or blog describing this and maybe related decisions/roadmaps? thx! >>> >>> >>> >>> Am Mo., 11. Mai 2020 um 10:05 Uhr schrieb Aaron Bossert < >>> aa...@punchcyber.com>: >>> >>> For what it is worth, I have written blueprints before, but never >>> stacks. The documentation and tutorials for ambari stacks and blueprints >>> are horribly out of date, incomplete, or flat out missing. Perhaps that >>> could be an initial task for the community to undertake so that those of us >>> who were using the Hortonworks suite of tools and were comfortable with >>> Ambari can sever the cord, as it were....relying on commercial companies to >>> support open source tools once their objectives have changed is rarely a >>> good thing. >>> >>> >>> >>> Get Outlook for iOS >>> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2Fo0ukef&data=01%7C01%7Caaron%40punchcyber.com%7C89b759ffe21a47ee874a08d7f6766280%7Cf7ca96216ba34d52a241e0ed3ea6a78a%7C0&sdata=AqkexLKW6pLpkhG0auiGnGpIy5aZpcc%2B%2BG53O2qCNek%3D&reserved=0> >>> ------------------------------ >>> >>> *From:* Ganesh Raju <ganesh.r...@linaro.org> >>> *Sent:* Monday, May 11, 2020 12:57:50 PM >>> *To:* user@ambari.apache.org <user@ambari.apache.org> >>> *Subject:* Re: life after Hortonworks >>> >>> >>> >>> Here is more Apache Bigtop info >>> >>> >>> >>> releases >>> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FBIGTOP%2FReleases&data=01%7C01%7Caaron%40punchcyber.com%7C89b759ffe21a47ee874a08d7f6766280%7Cf7ca96216ba34d52a241e0ed3ea6a78a%7C0&sdata=8xGs5j8Akzib7cRVlEgWq8ORcz8dWduDXqu8ev69mTE%3D&reserved=0> >>> >>> mailinglist >>> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbigtop.apache.org%2Fmail-lists.html&data=01%7C01%7Caaron%40punchcyber.com%7C89b759ffe21a47ee874a08d7f6766280%7Cf7ca96216ba34d52a241e0ed3ea6a78a%7C0&sdata=NgLr8gAPF2BXSr%2FXnUuaR2%2FdDU0c40RlyCoxiATR%2FWw%3D&reserved=0> >>> >>> list of components >>> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FBIGTOP%2FBigtop%2B1.4.0%2BRelease&data=01%7C01%7Caaron%40punchcyber.com%7C89b759ffe21a47ee874a08d7f6766280%7Cf7ca96216ba34d52a241e0ed3ea6a78a%7C0&sdata=vXMOynon6RZFabIVAo5PQ4D%2B6cB9rq%2BHSBbYBnH9uPI%3D&reserved=0> >>> >>> >>> >>> Thanks >>> >>> Ganesh Raju >>> >>> >>> >>> >>> >>> On Mon, May 11, 2020 at 10:00 AM Ganesh Raju <ganesh.r...@linaro.org> >>> wrote: >>> >>> Yes that's what Bigtop did. BTW, Apache Bigtop already has hadoop and 30 >>> other components. >>> >>> >>> >>> Ganesh >>> >>> >>> >>> On Mon, May 11, 2020 at 9:56 AM Matt Andruff <matt.andr...@gmail.com> >>> wrote: >>> >>> I kept digging and I found that ambari supports stacks (that we have all >>> seen before). You can write your own custom stack >>> <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FAMBARI%2FOverview%23Overview-Structure&data=01%7C01%7Caaron%40punchcyber.com%7C89b759ffe21a47ee874a08d7f6766280%7Cf7ca96216ba34d52a241e0ed3ea6a78a%7C0&sdata=3itBtwJy1mH3Eu2FbsVx0WurYZFlTlVY7VtV5D%2FbDjg%3D&reserved=0> >>> meaning you could write the "glue" what apache projects hortonworks had >>> written to roll out a set of services. I guess this is how they got around >>> being open source. I guess I'll start looking into making my own stack >>> from scratch. If anyone already has one or know of one, please let me know. >>> >>> >>> >>> On Mon, May 11, 2020 at 10:41 AM Ganesh Raju <ganesh.r...@linaro.org> >>> wrote: >>> >>> Apache Bigtop now has mpack and works with Ambari. You could try that. >>> >>> >>> >>> Thanks, >>> >>> Ganesh >>> >>> >>> >>> On Mon, May 11, 2020 at 9:15 AM Matt Andruff <matt.andr...@gmail.com> >>> wrote: >>> >>> Hey I was just poking around today trying to figure out if ambari still >>> works to install a cluster. >>> >>> >>> >>> I recently was able to install a Cloudera Manager installation. >>> >>> >>> >>> I was about to start an Ambari installation but when I poke the repos >>> they all seem to need a user/password. >>> >>> >>> >>> Is there a non-hortonworks(cloudera) hadoop that can be still be >>> installed with Amabari? >>> >>> >>> >>> If there is documentation somewhere just let me know, I couldn't find >>> the answer googling as Hortonworks(cloudera) docs are all that come up. >>> >>> >>> >>> -- >>> >>> Live life >>> Laugh Often >>> >>> >>> >>> -- >>> >>> IRC: ganeshraju@#linaro on irc.freenode.ne >>> <https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Firc.freenode.net%2F&data=01%7C01%7Caaron%40punchcyber.com%7C89b759ffe21a47ee874a08d7f6766280%7Cf7ca96216ba34d52a241e0ed3ea6a78a%7C0&sdata=uOTbOwThQhNWnkTGmf3HZ6lIxPG0jjmxMUFOdANlN8I%3D&reserved=0> >>> t >>> >>> >>> >>> >>> -- >>> >>> Live life >>> Laugh Often >>> >>> >>> >>> -- >>> >>> IRC: ganeshraju@#linaro on irc.freenode.ne >>> <https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Firc.freenode.net%2F&data=01%7C01%7Caaron%40punchcyber.com%7C89b759ffe21a47ee874a08d7f6766280%7Cf7ca96216ba34d52a241e0ed3ea6a78a%7C0&sdata=uOTbOwThQhNWnkTGmf3HZ6lIxPG0jjmxMUFOdANlN8I%3D&reserved=0> >>> t >>> >>> >>> >>> -- >>> >>> IRC: ganeshraju@#linaro on irc.freenode.ne >>> <https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Firc.freenode.net%2F&data=01%7C01%7Caaron%40punchcyber.com%7C89b759ffe21a47ee874a08d7f6766280%7Cf7ca96216ba34d52a241e0ed3ea6a78a%7C0&sdata=uOTbOwThQhNWnkTGmf3HZ6lIxPG0jjmxMUFOdANlN8I%3D&reserved=0> >>> t >>> >>> >> >> -- >> Live life >> Laugh Often >> > -- Live life Laugh Often
