Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

2019-09-03 Thread James Sirota
Given what Simon pointed out, where in your view are the current feature gaps with respect to that? We can back up the zookeeper config, any files we need on HDFS, and any custom parsers/enrichers that were produced. The rest of it is pretty much HDP-pecific upgrade. In my view, backing up

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

2019-08-27 Thread zeo...@gmail.com
I agree that having a scripted approach for backup and restore of Metron configs should be necessary for such a large change/upgrade process. Having been through this many times in the past I can tell you that the difficulty of upgrading (whether perceived or actual) holds back adoption of the

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

2019-08-27 Thread Simon Elliston Ball
Not sure it’s in the scope of the project to handle the HDP upgrade as well, I would scope it to metron config only, and point at the extensive upgrade capability of Ambari, rather than us trying to recreate the way the distribution works. Simon On Tue, 27 Aug 2019 at 22:23, Otto Fowler wrote:

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

2019-08-27 Thread Otto Fowler
If anyone can think of the things that need to be backed up, please comment the jira. On August 27, 2019 at 17:07:20, Otto Fowler (ottobackwa...@gmail.com) wrote: Good idea METRON–2239 [blocker]. On August 27, 2019 at 16:30:13, Simon Elliston Ball ( si...@simonellistonball.com) wrote: You

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

2019-08-27 Thread Otto Fowler
Good idea METRON–2239 [blocker]. On August 27, 2019 at 16:30:13, Simon Elliston Ball ( si...@simonellistonball.com) wrote: You could always submit a Jira :) On Tue, 27 Aug 2019 at 21:27, Otto Fowler wrote: > You are right, that is much better than backup_metron_configs.sh. > > > > On

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

2019-08-27 Thread Simon Elliston Ball
You could always submit a Jira :) On Tue, 27 Aug 2019 at 21:27, Otto Fowler wrote: > You are right, that is much better than backup_metron_configs.sh. > > > > > On August 27, 2019 at 16:05:38, Simon Elliston Ball ( > si...@simonellistonball.com) wrote: > > You can do this with zk_load_configs

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

2019-08-27 Thread Otto Fowler
You are right, that is much better than backup_metron_configs.sh. On August 27, 2019 at 16:05:38, Simon Elliston Ball ( si...@simonellistonball.com) wrote: You can do this with zk_load_configs and Ambari’s blueprint api, so we kinda already do. Simon On Tue, 27 Aug 2019 at 20:19, Otto

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

2019-08-27 Thread Simon Elliston Ball
You can do this with zk_load_configs and Ambari’s blueprint api, so we kinda already do. Simon On Tue, 27 Aug 2019 at 20:19, Otto Fowler wrote: > Maybe we need some automated method to backup configurations and restore > them. > > > > > On August 27, 2019 at 14:46:58, Michael Miklavcic ( >

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

2019-08-27 Thread Otto Fowler
Maybe we need some automated method to backup configurations and restore them. On August 27, 2019 at 14:46:58, Michael Miklavcic ( michael.miklav...@gmail.com) wrote: > Once you back up your metron configs, the same configs that worked on the previous version will continue to work on the

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

2019-08-27 Thread Michael Miklavcic
> Once you back up your metron configs, the same configs that worked on the previous version will continue to work on the version running on HDP 3.x. If there is any discrepancy between the two or additional settings will be required, those will be documented in the release notes. From the Metron

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

2019-08-27 Thread Michael Miklavcic
Adding the dev list back into the thread (a reply-all was missed). On Tue, Aug 27, 2019 at 10:49 AM James Sirota wrote: > I agree with Simon. HDP 2.x platform is rapidly approaching EOL and > everyone will likely need to migrate by end of year. Doing this platform > upgrade sooner will give

Re: [DISCUSS] HDP 3.1 Upgrade and release strategy

2019-08-27 Thread Otto Fowler
Although we had the discussion, and some great ideas where passed around, I do not believe we came to some kind of consensus on what 1.0 should look like. So that discussion would have to be picked up again so that we could know where we are at, and make it an actual thing if we were going to make

[DISCUSS] HDP 3.1 Upgrade and release strategy

2019-08-26 Thread Michael Miklavcic
Hi devs and users, Some questions were asked in the Slack channel about our ongoing HDP/Hadoop upgrade and I'd like to get a discussion rolling. The original Hadoop upgrade discuss thread can be found here