Thanks a lot for the feedback, I really appreciated it :) Luca
On Mon, Nov 23, 2020 at 6:18 PM Evans Ye <evan...@apache.org> wrote: > > When I was at TrendMicro, we did not backup the data as well. > Since the upgrade itself is somehow duplicating the data in two different > versions. > I'd say FSImage backup is super important to make sure you can at least bring > namenode back online. And it's cheap. > Since I was at the role 4 years ago. Take the following with your own > judgement... > > For scenario 1: Upgrade goes fine, but the hdfs finalize step fails > AFAIK finalize is to cleanup previous blocks and yield the space onward. So I > think this is ok. > > For scenario 2: Upgrade fails, rollback fails as well, and restoring a > previous namenode fsimage is not enough > Restoring the FSImage should be ok but you'll probably get several missing > blocks due to inconsistent. You can only accept the data loss but the main > portion of the Data Lake should be fine. > > For additional backup, I think it's your decision to weight the risk and your > business. Let's say when upgrading, at a very specific moment that you just > hit the upgrade button, suddenly the power/network is down and making entire > cluster entering a wired state. The chance is rare, but no one can say. > > Additional suggestions: > 1. classify your data into more fine-grained levels and only backup those > super important ones. > 2. If you have staging, dev clusters, conscript them to participate the > upgrade. Use distcp to backup data between prod/stage/dev clusters to > minimize the data recency gap. > > > > Luca Toscano <toscano.l...@gmail.com> 於 2020年11月23日 週一 下午5:35寫道： >> >> Hi everybody, >> >> I am currently struggling with a precautionary step before moving to >> Bigtop, namely finding enough space on a temporary backup cluster >> (separated from the one to upgrade) to save important data that my >> team wouldn't be able to recover in case of HDFS failure (historical >> data etc..). I have tested the upgrade and rollback several times >> (from CDH to Bigtop), but the nightmare scenarios that I have in mind >> are: >> >> - Upgrade goes fine, but the hdfs finalize step fails and leaves HDFS >> inconsistent for some reason (datanodes' previous directory not >> present anymore, etc..) >> - Upgrade fails, rollback fails as well, and restoring a previous >> namenode fsimage is not enough (inconsistent datanode state across the >> cluster etc..) >> >> Having the absolute important data set aside on a separate cluster >> (not involved in the upgrade) seems the right and more conservative >> choice, but it is of course challenging when the data to backup spans >> hundreds of terabytes. Are the above right concerns? Or are those too >> much considering the risks of the upgrade? I've researched this a bit >> and didn't find anybody backing up data (except for the Namenode's >> metadata like fsimages etc..) before an HDFS upgrade. >> >> If anybody went through the same doubts and could give me some >> feedback it would be really appreciated :) >> >> Thanks in advance, >> >> Luca >> >> >> >> On Mon, Sep 28, 2020 at 8:48 AM Evans Ye <evan...@apache.org> wrote: >> > >> > Oh ok. That sounds great! >> > >> > Luca Toscano <toscano.l...@gmail.com> 於 2020年9月28日 週一 14:31 寫道： >> >> >> >> Hi Evans, >> >> >> >> what I meant with a blog post shared would be something that goes in >> >> http://techblog.wikimedia.org/ and on >> >> https://blogs.apache.org/bigtop/, stating that we collaborated and how >> >> :) >> >> >> >> Luca >> >> >> >> On Mon, Sep 21, 2020 at 5:44 PM Evans Ye <evan...@apache.org> wrote: >> >> > >> >> > Yes. Overall it sounds great to me! >> >> > >> >> > I think the "summary of known pitfalls/bugs/etc.." section is worth to >> >> > add and might be a super valuable part of the whole thing. >> >> > >> >> > | "The Blog post would be a good idea, maybe something that we can >> >> > share between Wikimedia and Apache" >> >> > What do you mean by this one, specifically? Currently 3 things we can >> >> > in below. Do they match what you think or it's something else? >> >> > >> >> > 1. Bigtop wiki/blogs: >> >> > https://cwiki.apache.org/confluence/display/BIGTOP/Index >> >> > https://blogs.apache.org/bigtop/ >> >> > >> >> > 2. Success At Apache: >> >> > https://blogs.apache.org/foundation/category/SuccessAtApache >> >> > >> >> > 3. ApacheCon Talk (this year CFP is over, we can do it next year as a >> >> > post production expereince sharing) >> >> > https://apachecon.com/index.html >> >> > >> >> > - Evans >> >> > >> >> > >> >> > Luca Toscano <toscano.l...@gmail.com> 於 2020年9月20日 週日 下午4:55寫道： >> >> >> >> >> >> Hi Evans, >> >> >> >> >> >> I am late in answering as well :) >> >> >> >> >> >> I thought about it and I think that with the right premises (example: >> >> >> this is tailored for Wikimedia's environment, it assumes that a >> >> >> cluster downtime is acceptable, etc..) the storytelling style might be >> >> >> more easy to digest than a list of steps to follow. I think that in >> >> >> all use cases different from Wikimedia there will be adjustments to >> >> >> make, and things that work/don't-work/etc.. One thing that it might be >> >> >> good to add at the end is a "summary of known pitfalls/bugs/etc.." >> >> >> found during the procedure, that in my case were the most >> >> >> time-consuming ones. I'll add it during the next few days and people >> >> >> can comment :) >> >> >> >> >> >> The Blog post would be a good idea, maybe something that we can share >> >> >> between Wikimedia and Apache? I am planning to move to BigTop during >> >> >> the upcoming quarter (October -> December), that will also show if my >> >> >> procedure works on a cluster of 60+ nodes (rather than on a small one >> >> >> of 8 nodes) :D. As soon as I have done it I'll follow up with this >> >> >> list so organize a blog post, does it sound ok? >> >> >> >> >> >> Thanks a lot for all the support! >> >> >> >> >> >> Luca >> >> >> >> >> >> On Tue, Sep 15, 2020 at 6:06 PM Evans Ye <evan...@apache.org> wrote: >> >> >> > >> >> >> > Hey Luca, >> >> >> > >> >> >> > Sorry for the late reply. I was busy for a conference. It's just >> >> >> > over now. >> >> >> > Anyway, I think the writing is pretty informative. But it's more >> >> >> > like a storytelling style. Also several contents are WikiMedia >> >> >> > specific things. That's why I think it's more suitable for a >> >> >> > blogpost. >> >> >> > >> >> >> > Anyhow, I think either way it's great content. If we keep it as is, >> >> >> > I think we can make it available on Bigtop's WIKI & Blog, or even >> >> >> > Success at Apache with the title like "WikiMedia's story to migrate >> >> >> > from CDH to Bigtop". If you want to make it more like an official >> >> >> > guide, the title will be "CDH to Bigtop Migration Guide". We can >> >> >> > state the limitation and environment so that people can take it w/ >> >> >> > a caution that it might not suit their own environment. >> >> >> > >> >> >> > Which way to go depends on how much effort you'd like to take. Let >> >> >> > me know what you think so that we can move forward. >> >> >> > >> >> >> > - Evans >> >> >> > >> >> >> > Luca Toscano <toscano.l...@gmail.com> 於 2020年9月7日 週一 下午3:39寫道： >> >> >> >> >> >> >> >> Hi Evans, >> >> >> >> >> >> >> >> thanks for the review! What are the things that you'd like to see to >> >> >> >> make them more consumable for users? I can re-shape the writing, I >> >> >> >> tried to come up with something to kick off a conversation with the >> >> >> >> community, it would be interesting to know if anybody else has a >> >> >> >> similar use case and how/if they are working on a solution. >> >> >> >> >> >> >> >> For the blogpost, maybe we can coordinate something shared between >> >> >> >> Apache and Wikimedia when the migration is done, I am sure it would >> >> >> >> be >> >> >> >> a nice example of the two Foundations collaborating :) >> >> >> >> >> >> >> >> Luca >> >> >> >> >> >> >> >> On Wed, Sep 2, 2020 at 8:21 PM Evans Ye <evan...@apache.org> wrote: >> >> >> >> > >> >> >> >> > Hi Luca, >> >> >> >> > >> >> >> >> > I read through the doc briefly. I think the doc works very well >> >> >> >> > as a blogpost of a successful story for Wikimedia migrating from >> >> >> >> > CDH to Bigtop. However, the current writing doesn't seem to be >> >> >> >> > easily consumable for users' who are just looking into the >> >> >> >> > solutions/steps for doing similar migrations. May I know what >> >> >> >> > title you would prefer if we put the doc in Bigtop's wiki? >> >> >> >> > >> >> >> >> > What I was thinking is the cookbook for migration. But we can >> >> >> >> > discuss this. IMHO a Success at Apache[1] blogpost is also >> >> >> >> > possible. But I need to figure out who to talk to. Let me know >> >> >> >> > what you think. >> >> >> >> > >> >> >> >> > [1] https://blogs.apache.org/foundation/category/SuccessAtApache >> >> >> >> > >> >> >> >> > Evans >> >> >> >> > >> >> >> >> > Evans Ye <evan...@apache.org> 於 2020年8月30日 週日 上午3:18寫道： >> >> >> >> >> >> >> >> >> >> Hi Luca, >> >> >> >> >> >> >> >> >> >> I'm on vacation hence do not have time for review right now. >> >> >> >> >> I'll get back to you next week. >> >> >> >> >> >> >> >> >> >> The doc is definitely valuable. Once you have your production >> >> >> >> >> migrated successfully. We can prove to the other users that this >> >> >> >> >> is a battle proven solution. Even more, we can give a talk at >> >> >> >> >> ApacheCon or somewhere else to further amplify the impact of the >> >> >> >> >> work. This is definitely an open source winning case so I think >> >> >> >> >> it deserve a talk. >> >> >> >> >> >> >> >> >> >> Evans >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> Luca Toscano <toscano.l...@gmail.com> 於 2020年8月27日 週四 下午9:11寫道： >> >> >> >> >>> >> >> >> >> >>> Hi Evans, >> >> >> >> >>> >> >> >> >> >>> it took a while I know but I have the first version of the gdoc >> >> >> >> >>> for the upgrade: >> >> >> >> >>> >> >> >> >> >>> https://docs.google.com/document/d/1fI1mvbR1mFLV6ohU5cIEnU5hFvEE7EWnKYWOkF55jtE/edit?usp=sharing >> >> >> >> >>> >> >> >> >> >>> I tried to list all the steps involved in migrating from CDH 5 >> >> >> >> >>> to >> >> >> >> >>> Bigtop 1.4, anybody interested should be able to comment. The >> >> >> >> >>> idea >> >> >> >> >>> that I have is to discuss this for a few days and then possibly >> >> >> >> >>> make >> >> >> >> >>> it permanent somewhere in the Bigtop wiki? (of course if the >> >> >> >> >>> document >> >> >> >> >>> will be considered useful for others etc..) >> >> >> >> >>> >> >> >> >> >>> During these days I tested the procedure multiple times, and I >> >> >> >> >>> have >> >> >> >> >>> also tested the HDFS finalize step, everything works as >> >> >> >> >>> expected. I >> >> >> >> >>> hope to be able to move to Bigtop during the next couple of >> >> >> >> >>> months. >> >> >> >> >>> >> >> >> >> >>> Luca >> >> >> >> >>> >> >> >> >> >>> On Tue, Jul 21, 2020 at 4:04 PM Evans Ye <evan...@apache.org> >> >> >> >> >>> wrote: >> >> >> >> >>> > >> >> >> >> >>> > Yes. I think a shared gdoc is prefered, and you can open up a >> >> >> >> >>> > JIRA ticket to track it. >> >> >> >> >>> > >> >> >> >> >>> > Luca Toscano <toscano.l...@gmail.com> 於 2020年7月20日 週一 21:10 >> >> >> >> >>> > 寫道： >> >> >> >> >>> >> >> >> >> >> >>> >> Hi Evans! >> >> >> >> >>> >> >> >> >> >> >>> >> What is the best medium to use for the >> >> >> >> >>> >> documentation/comments ? A >> >> >> >> >>> >> shared gdoc or something similar? >> >> >> >> >>> >> >> >> >> >> >>> >> Luca >> >> >> >> >>> >> >> >> >> >> >>> >> On Thu, Jul 16, 2020 at 5:11 PM Evans Ye >> >> >> >> >>> >> <evan...@apache.org> wrote: >> >> >> >> >>> >> > >> >> >> >> >>> >> > One thing I think would be great to have is a doc version >> >> >> >> >>> >> > of the steps for upgrade and rollback. The benefits: >> >> >> >> >>> >> > 1. Anything unexpected happened during automation, you do >> >> >> >> >>> >> > have folks can quickly understand what's going on and get >> >> >> >> >>> >> > into the investigation. >> >> >> >> >>> >> > 2. Share the doc with us to help the others OSS users for >> >> >> >> >>> >> > doing the migration. For the env specific things I think >> >> >> >> >>> >> > that's fine. We can left comment on it. At least all the >> >> >> >> >>> >> > other users can get a high level view of a proven >> >> >> >> >>> >> > solution. And then they can go and find out the rest of >> >> >> >> >>> >> > the pieces by themselves. >> >> >> >> >>> >> > >> >> >> >> >>> >> > For automations, I suggest to split up the automation into >> >> >> >> >>> >> > several stages, and apply some validation steps(manually >> >> >> >> >>> >> > is ok) before kicking of the next stage. >> >> >> >> >>> >> > >> >> >> >> >>> >> > Best, >> >> >> >> >>> >> > Evans >> >> >> >> >>> >> > >> >> >> >> >>> >> > >> >> >> >> >>> >> > >> >> >> >> >>> >> > >> >> >> >> >>> >> > Luca Toscano <toscano.l...@gmail.com> 於 2020年7月15日 週三 >> >> >> >> >>> >> > 下午9:07寫道： >> >> >> >> >>> >> >> >> >> >> >> >>> >> >> Hi everybody, >> >> >> >> >>> >> >> >> >> >> >> >>> >> >> I didn't get the time to work on this until recently, but >> >> >> >> >>> >> >> I finally >> >> >> >> >>> >> >> managed to have a reliable procedure to upgrade from CDH >> >> >> >> >>> >> >> to Bigtop 1.4 >> >> >> >> >>> >> >> and rollback if needed. The assumptions are: >> >> >> >> >>> >> >> >> >> >> >> >>> >> >> 1) It is ok to have (limited) cluster downtime. >> >> >> >> >>> >> >> 2) Rolling upgrade is not needed. >> >> >> >> >>> >> >> 3) QJM is used. >> >> >> >> >>> >> >> >> >> >> >> >>> >> >> The procedure is listed in these two scripts: >> >> >> >> >>> >> >> >> >> >> >> >>> >> >> https://github.com/wikimedia/operations-cookbooks/blob/master/cookbooks/sre/hadoop/stop-cluster.py >> >> >> >> >>> >> >> https://github.com/wikimedia/operations-cookbooks/blob/master/cookbooks/sre/hadoop/change-distro-from-cdh.py >> >> >> >> >>> >> >> >> >> >> >> >>> >> >> The code is highly dependent on my working environment, >> >> >> >> >>> >> >> but it should >> >> >> >> >>> >> >> be clear to follow when writing a tutorial about how to >> >> >> >> >>> >> >> migrate from >> >> >> >> >>> >> >> CDH to Bigtop. All the suggestions given by this mailing >> >> >> >> >>> >> >> list were >> >> >> >> >>> >> >> really useful to reach a solution! >> >> >> >> >>> >> >> >> >> >> >> >>> >> >> My next steps will be: >> >> >> >> >>> >> >> >> >> >> >> >>> >> >> 1) Keep testing Bigtop 1.4 (finalize HDFS upgrade, run >> >> >> >> >>> >> >> more hadoop >> >> >> >> >>> >> >> jobs, test Hive 2, etc..). >> >> >> >> >>> >> >> 2) Upgrade the production Hadoop cluster to Bigtop 1.4 on >> >> >> >> >>> >> >> Debian 9 >> >> >> >> >>> >> >> (HDFS 2.6.0-cdh -> 2.8.5). >> >> >> >> >>> >> >> 3) Upgrade to Bigtop 1.5 on Debian 9 (HDFS 2.8.5 -> 2.10). >> >> >> >> >>> >> >> 4) Upgrade to Debian 10. >> >> >> >> >>> >> >> >> >> >> >> >>> >> >> With automation it shouldn't be very difficult, I'll >> >> >> >> >>> >> >> report progress once made. >> >> >> >> >>> >> >> >> >> >> >> >>> >> >> Thanks a lot! >> >> >> >> >>> >> >> >> >> >> >> >>> >> >> Luca >> >> >> >> >>> >> >> >> >> >> >> >>> >> >> On Mon, Apr 13, 2020 at 9:25 AM Luca Toscano >> >> >> >> >>> >> >> <toscano.l...@gmail.com> wrote: >> >> >> >> >>> >> >> > >> >> >> >> >>> >> >> > Hi Evans, >> >> >> >> >>> >> >> > >> >> >> >> >>> >> >> > thanks a lot for the feedback, it was exactly what I >> >> >> >> >>> >> >> > needed. The >> >> >> >> >>> >> >> > simpler the better is definitely a good advice in this >> >> >> >> >>> >> >> > use case, I'll >> >> >> >> >>> >> >> > try this week another rollout/rollback and report back >> >> >> >> >>> >> >> > :) >> >> >> >> >>> >> >> > >> >> >> >> >>> >> >> > Luca >> >> >> >> >>> >> >> > >> >> >> >> >>> >> >> > On Thu, Apr 9, 2020 at 8:09 PM Evans Ye >> >> >> >> >>> >> >> > <evan...@apache.org> wrote: >> >> >> >> >>> >> >> > > >> >> >> >> >>> >> >> > > Hi Luca, >> >> >> >> >>> >> >> > > >> >> >> >> >>> >> >> > > Thanks for reporting back and let us know how it goes. >> >> >> >> >>> >> >> > > I don't have the exactly HDFS with QJM HA upgrade >> >> >> >> >>> >> >> > > experience. The experience I had was 0.20 non-HA >> >> >> >> >>> >> >> > > upgrade to 2.0 non-HA and then enable QJM HA, which >> >> >> >> >>> >> >> > > was back in 2014. >> >> >> >> >>> >> >> > > >> >> >> >> >>> >> >> > > Regarding to rollback, I think you're right: >> >> >> >> >>> >> >> > > >> >> >> >> >>> >> >> > > it is possible to rollback to HDFS’ state before the >> >> >> >> >>> >> >> > > upgrade in case of unexpected problems. >> >> >> >> >>> >> >> > > >> >> >> >> >>> >> >> > > My previous experience is the same that the rollback >> >> >> >> >>> >> >> > > is merely a snapshot before the upgrade. If you've >> >> >> >> >>> >> >> > > gone far, then rollback cost more data lost... Our >> >> >> >> >>> >> >> > > runbook is if our sanity check failed during upgrade >> >> >> >> >>> >> >> > > downtime, we perform the rollback immediately. >> >> >> >> >>> >> >> > > >> >> >> >> >>> >> >> > > Regarding to that FSImage hole issue, I've >> >> >> >> >>> >> >> > > experienced it as well. >> >> >> >> >>> >> >> > > I managed to fix it by manually edit the FSImage with >> >> >> >> >>> >> >> > > offline image viewer[1] and delete that missing >> >> >> >> >>> >> >> > > editLog in FSImage. That actually brought my cluster >> >> >> >> >>> >> >> > > back with a little number of missing blocks. >> >> >> >> >>> >> >> > > >> >> >> >> >>> >> >> > > Our experience says that the more the steps, the more >> >> >> >> >>> >> >> > > the chance you failed the upgrade. We did good on >> >> >> >> >>> >> >> > > dozen times of testing, DEV cluster, STAGING cluster, >> >> >> >> >>> >> >> > > but still got missing blocks when upgrading >> >> >> >> >>> >> >> > > Production... >> >> >> >> >>> >> >> > > >> >> >> >> >>> >> >> > > The suggestion is to get your production in good >> >> >> >> >>> >> >> > > shape first(the less decommissioned, offline DNs, >> >> >> >> >>> >> >> > > disk failures, the better). >> >> >> >> >>> >> >> > > Also, maybe you can switch to non-HA mode and do the >> >> >> >> >>> >> >> > > upgrade to simplify the things? >> >> >> >> >>> >> >> > > >> >> >> >> >>> >> >> > > Not many helps but please let us know if any progress. >> >> >> >> >>> >> >> > > Last one, have you reached out to Hadoop community? >> >> >> >> >>> >> >> > > the authors should know the most :) >> >> >> >> >>> >> >> > > >> >> >> >> >>> >> >> > > - Evans >> >> >> >> >>> >> >> > > >> >> >> >> >>> >> >> > > [1] >> >> >> >> >>> >> >> > > https://hadoop.apache.org/docs/r2.8.5/hadoop-project-dist/hadoop-hdfs/HdfsImageViewer.html >> >> >> >> >>> >> >> > > >> >> >> >> >>> >> >> > > Luca Toscano <toscano.l...@gmail.com> 於 2020年4月8日 週三 >> >> >> >> >>> >> >> > > 21:03 寫道： >> >> >> >> >>> >> >> > >> >> >> >> >> >>> >> >> > >> Hi everybody, >> >> >> >> >>> >> >> > >> >> >> >> >> >>> >> >> > >> most of the bugs/issues/etc.. that I found while >> >> >> >> >>> >> >> > >> upgrading from CDH 5 >> >> >> >> >>> >> >> > >> to BigTop 1.4 are fixed, I am now testing (as >> >> >> >> >>> >> >> > >> suggested also in here) >> >> >> >> >>> >> >> > >> upgrade/rollback procedures for HDFS (all written in >> >> >> >> >>> >> >> > >> https://phabricator.wikimedia.org/T244499, will add >> >> >> >> >>> >> >> > >> documentation >> >> >> >> >>> >> >> > >> about this at the end I promise). >> >> >> >> >>> >> >> > >> >> >> >> >> >>> >> >> > >> I initially followed [1][2] in my Test cluster, >> >> >> >> >>> >> >> > >> choosing the Rolling >> >> >> >> >>> >> >> > >> upgrade, but when I tried to rollback (after days >> >> >> >> >>> >> >> > >> since the initial >> >> >> >> >>> >> >> > >> upgrade) I ended up in an inconsistent state and I >> >> >> >> >>> >> >> > >> wasn't able to >> >> >> >> >>> >> >> > >> recover the previous HDFS state. I didn't save the >> >> >> >> >>> >> >> > >> exact error >> >> >> >> >>> >> >> > >> messages but the situation was more or less the >> >> >> >> >>> >> >> > >> following: >> >> >> >> >>> >> >> > >> >> >> >> >> >>> >> >> > >> FS-Image-rollback (created at the time of the >> >> >> >> >>> >> >> > >> upgrade) - up to transaction X >> >> >> >> >>> >> >> > >> FS-Image-current - up to transaction Y, with Y = X + >> >> >> >> >>> >> >> > >> 10000 (number >> >> >> >> >>> >> >> > >> totally made up for the example) >> >> >> >> >>> >> >> > >> QJM cluster: first available transaction Z = X + >> >> >> >> >>> >> >> > >> 10000 + 1 >> >> >> >> >>> >> >> > >> >> >> >> >> >>> >> >> > >> When I tried to rolling rollback, the Namenode >> >> >> >> >>> >> >> > >> complained about a hole >> >> >> >> >>> >> >> > >> in the transaction log, namely at X + 1, so it >> >> >> >> >>> >> >> > >> refused to start. I >> >> >> >> >>> >> >> > >> tried to force a regular rollback, but the Namenode >> >> >> >> >>> >> >> > >> refused again >> >> >> >> >>> >> >> > >> saying that there was no available FS Image to roll >> >> >> >> >>> >> >> > >> back to. I checked >> >> >> >> >>> >> >> > >> in the Hadoop code and indeed the Namenode saves the >> >> >> >> >>> >> >> > >> fs image with >> >> >> >> >>> >> >> > >> different naming/path in case of a rolling upgrade >> >> >> >> >>> >> >> > >> or a regular >> >> >> >> >>> >> >> > >> upgrade. Both cases make sense, especially the first >> >> >> >> >>> >> >> > >> one since there >> >> >> >> >>> >> >> > >> was indeed a hole between the last transaction of the >> >> >> >> >>> >> >> > >> FS-Image-rollback and the first available >> >> >> >> >>> >> >> > >> transaction to reply on the >> >> >> >> >>> >> >> > >> QJM cluster. I chose the rolling upgrade initially >> >> >> >> >>> >> >> > >> since it was >> >> >> >> >>> >> >> > >> appealing: it promises to bring back the Namenodes >> >> >> >> >>> >> >> > >> to their previous >> >> >> >> >>> >> >> > >> versions, but keeping the data modified between >> >> >> >> >>> >> >> > >> upgrade and rollback. >> >> >> >> >>> >> >> > >> >> >> >> >> >>> >> >> > >> I then found [3], in which it is said that with QJM >> >> >> >> >>> >> >> > >> everything is more >> >> >> >> >>> >> >> > >> complicated, and a regular rollback is the only >> >> >> >> >>> >> >> > >> option available. What >> >> >> >> >>> >> >> > >> I think this mean is that due to the Edit log spread >> >> >> >> >>> >> >> > >> among multiple >> >> >> >> >>> >> >> > >> nodes, a rollback that keeps data between upgrade >> >> >> >> >>> >> >> > >> and rollback is not >> >> >> >> >>> >> >> > >> available, so worst case scenario the data modified >> >> >> >> >>> >> >> > >> during that >> >> >> >> >>> >> >> > >> timeframe is lost. Not a big deal in my case, but I >> >> >> >> >>> >> >> > >> want to triple >> >> >> >> >>> >> >> > >> check with you if this is the correct interpretation >> >> >> >> >>> >> >> > >> or if there is >> >> >> >> >>> >> >> > >> another tutorial/guide/etc.. that I haven't read >> >> >> >> >>> >> >> > >> with a different >> >> >> >> >>> >> >> > >> procedure :) >> >> >> >> >>> >> >> > >> >> >> >> >> >>> >> >> > >> Is my interpretation correct? If not, is there >> >> >> >> >>> >> >> > >> anybody with experience >> >> >> >> >>> >> >> > >> in HDFS upgrades that could shed some light on the >> >> >> >> >>> >> >> > >> subject? >> >> >> >> >>> >> >> > >> >> >> >> >> >>> >> >> > >> Thanks in advance! >> >> >> >> >>> >> >> > >> >> >> >> >> >>> >> >> > >> Luca >> >> >> >> >>> >> >> > >> >> >> >> >> >>> >> >> > >> >> >> >> >> >>> >> >> > >> >> >> >> >> >>> >> >> > >> [1] >> >> >> >> >>> >> >> > >> https://hadoop.apache.org/docs/r2.8.5/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Upgrade_and_Rollback >> >> >> >> >>> >> >> > >> [2] >> >> >> >> >>> >> >> > >> https://hadoop.apache.org/docs/r2.8.5/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html >> >> >> >> >>> >> >> > >> [3] >> >> >> >> >>> >> >> > >> https://hadoop.apache.org/docs/r2.8.5/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html#HDFS_UpgradeFinalizationRollback_with_HA_Enabled
