Good idea METRON–2239 [blocker].
On August 27, 2019 at 16:30:13, Simon Elliston Ball ( si...@simonellistonball.com) wrote: You could always submit a Jira :) On Tue, 27 Aug 2019 at 21:27, Otto Fowler <ottobackwa...@gmail.com> wrote: > You are right, that is much better than backup_metron_configs.sh. > > > > On August 27, 2019 at 16:05:38, Simon Elliston Ball ( > si...@simonellistonball.com) wrote: > > You can do this with zk_load_configs and Ambari’s blueprint api, so we > kinda already do. > > Simon > > On Tue, 27 Aug 2019 at 20:19, Otto Fowler <ottobackwa...@gmail.com> wrote: > >> Maybe we need some automated method to backup configurations and restore >> them. >> >> >> >> On August 27, 2019 at 14:46:58, Michael Miklavcic ( >> michael.miklav...@gmail.com) wrote: >> >> > Once you back up your metron configs, the same configs that worked on >> the previous version will continue to work on the version running on HDP >> 3.x. If there is any discrepancy between the two or additional settings >> will be required, those will be documented in the release notes. From the >> Metron perspective, this upgrade would be no different than simply >> upgrading to the new Metron version. >> >> This upgrade cannot be performed the same way we've done it in the past. >> A number of platform upgrades, including OS, are required: >> >> 1. Requires the OS to be updated on all nodes because there are no >> Centos6 RPMs provided in HDP 3.1. Must bump to Centos7. >> 2. The final new HBase code will not run on HDP 2.6 >> 3. The MPack changes for new Ambari are not backwards compatible >> 4. YARN and HDFS/MR are also at risk to be backwards incompatible >> >> >> On Tue, Aug 27, 2019 at 12:39 PM Michael Miklavcic < >> michael.miklav...@gmail.com> wrote: >> >>> Adding the dev list back into the thread (a reply-all was missed). >>> >>> On Tue, Aug 27, 2019 at 10:49 AM James Sirota <jsir...@apache.org> >>> wrote: >>> >>>> I agree with Simon. HDP 2.x platform is rapidly approaching EOL and >>>> everyone will likely need to migrate by end of year. Doing this platform >>>> upgrade sooner will give everyone visibility into what Metron on HDP 3.x >>>> looks like so they have time to plan and upgrade at their own pace. >>>> Feature-wise, the Metron application itself will be unchanged. It is >>>> merely the platform underneath that is changing. HDP itself can be >>>> upgraded per instructions here: >>>> https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/upgrading_parent.html >>>> >>>> Once you back up your metron configs, the same configs that worked on >>>> the previous version will continue to work on the version running on HDP >>>> 3.x. If there is any discrepancy between the two or additional settings >>>> will be required, those will be documented in the release notes. From the >>>> Metron perspective, this upgrade would be no different than simply >>>> upgrading to the new Metron version. >>>> >>>> James >>>> >>>> >>>> 27.08.2019, 07:09, "Simon Elliston Ball" <si...@simonellistonball.com>: >>>> >>>> Something worth noting here is that HDP 2.6.5 is quite old and >>>> approaching EoL rapidly, so the issue of upgrade is urgent. I am aware of a >>>> large number of users who require this upgrade ASAP, and in fact an aware >>>> of zero users who wish to remain on HDP 2. >>>> >>>> Perhaps those users who want to stay on the old platform can stick >>>> their hands up and raise concerns, but this move will likely have to happen >>>> very soon. >>>> >>>> Simon >>>> >>>> On Tue, 27 Aug 2019 at 15:04, Otto Fowler <ottobackwa...@gmail.com> >>>> wrote: >>>> >>>> Although we had the discussion, and some great ideas where passed >>>> around, I do not believe we came to some kind of consensus on what 1.0 >>>> should look like. So that discussion would have to be picked up again so >>>> that we could know where we are at, and make it an actual thing if we were >>>> going to make this a 1.0 release. >>>> >>>> I believe that the issues raised in that discussion gating 1.0 are >>>> still largely applicable, including upgrade. >>>> >>>> Right now we have *ZERO* HDP 3.1 users. We will go from that to *only* >>>> supporting 3.1 work and releases. So every user and deployment we currently >>>> have will feel real pain, have to slay real dragons to move forward with >>>> metron. >>>> >>>> With regards to support for older versions, the “backward capability” >>>> that has been mentioned, I would not say that I have any specific plan for >>>> that in mind. What I would say rather, is that I believe that we must be >>>> explicit, setting expectations correctly and clearly with regards to our >>>> intent while demonstrating that we have thought through the situation. That >>>> discussion has not happened, at least I do not believe that the prior dev >>>> thread really handles it in context. >>>> >>>> Depending on the upgrade situation for going to 3.1, it may be that a >>>> dual stream of releases or fixes or new features to the extent that we can >>>> do it will greatly reduce the pain for folks, or make it viable to stick >>>> with metron until they can upgrade. >>>> >>>> The issue of what metron *is* features wise may be another one we want >>>> to take up at some point. The idea being can we separate the metron >>>> _integration parts from the metron core functionality such that we can work >>>> on them separately and thus support multiple platforms through >>>> integrations/applications. Of course definition of metron’s value beyond >>>> integration, and what those features and application boundaries are would >>>> be necessary. >>>> >>>> >>>> >>>> On August 26, 2019 at 18:52:57, Michael Miklavcic ( >>>> michael.miklav...@gmail.com) wrote: >>>> >>>> Hi devs and users, >>>> >>>> Some questions were asked in the Slack channel about our ongoing >>>> HDP/Hadoop upgrade and I'd like to get a discussion rolling. The original >>>> Hadoop upgrade discuss thread can be found here >>>> https://lists.apache.org/thread.html/37cc29648f0592cc39d3c78a0d07fce38521bdbbc4cf40e022a7a8ea@%3Cdev.metron.apache.org%3E >>>> >>>> The major issue and impact with upgrading the Hadoop platform is that >>>> there are breaking changes. Code that runs on HDP 3.1 will not run on 2.x. >>>> Here is a sampling of core components we depend on that we know of so >>>> far that are not backwards compatible: >>>> >>>> 1. The Core OS - we currently base our builds and test deployment >>>> off of artifacts pulled from HDP. The latest rev of HDP no longer ships >>>> RPMs for Centos 6, which means we need to upgrade to Centos 7 >>>> 2. Ambari >>>> 3. HBase >>>> >>>> This differs from individual components we've upgraded in the past in >>>> that our code could still be deployed on the old as well as new version of >>>> the component in a backwards compatible way. Based on semantic versioning, >>>> I don't know if we can introduce this level of change in a point release, >>>> which is the reason for kicking off this discussion. In the past, users and >>>> developers in the community have suggested that they are -1 on a 1.x >>>> release that does not provide an upgrade >>>> https://lists.apache.org/thread.html/eb1a8df2d0a6a79c5d50540d1fdbf215ec83d831ff15d3117c2592cc@%3Cdev.metron.apache.org%3E >>>> . >>>> >>>> Is there a way we can avoid a 1.x release? If we do need 1.x, do we >>>> still see upgrades as a gating function? The main issue is that this has >>>> the potential to drag out the upgrade and further couple it with other >>>> features. And with Storm 1.x being eol'ed, I'm not sure this is something >>>> we can wait much longer for. I'll think on this and send out my own >>>> thoughts once folks have had a chance to review. >>>> >>>> Best, >>>> Mike Miklavcic >>>> Apache Metron, PMC, committer >>>> >>>> >>>> -- >>>> -- >>>> simon elliston ball >>>> @sireb >>>> >>>> >>>> >>>> ------------------- >>>> Thank you, >>>> >>>> James Sirota >>>> PMC- Apache Metron >>>> jsirota AT apache DOT org >>>> >>>> -- > -- > simon elliston ball > @sireb > > -- -- simon elliston ball @sireb
