And a bad news is that this patch is not contained in 1.2.x and 1.1.x which means it is not safe to rolling upgrade from 1.2.x or 1.1.x to 2.0 if we change the default wal implementation to AsyncFSWA, or a user set the wal implementation to asyncfs manually...
2016-12-07 18:17 GMT+08:00 张铎 <palomino...@gmail.com>: > See this one > > https://issues.apache.org/jira/browse/HBASE-14949 > > 2016-12-07 18:09 GMT+08:00 Ted Yu <yuzhih...@gmail.com>: > >> Which other features ? >> Can you name them ? >> >> Thanks >> >> On Wed, Dec 7, 2016 at 2:06 AM, 张铎 <palomino...@gmail.com> wrote: >> >> > I believe there are other features that require upgrading to a specific >> > version in each 1.x release before rolling upgrading to 2.0? I think >> this >> > is a typical trick to add incompatible changes... >> > >> > 2016-12-07 18:02 GMT+08:00 ramkrishna vasudevan < >> > ramkrishna.s.vasude...@gmail.com>: >> > >> > > If its going to be a pain then we need to support writing the older >> class >> > > name in the FFT (trailer) and then use our internal mapping. >> > > >> > > On Wed, Dec 7, 2016 at 3:22 PM, Anoop John <anoop.hb...@gmail.com> >> > wrote: >> > > >> > > > I read the comments in the issue 16189.. This concern was raised >> > > > there already.. This is what I replied there. >> > > > "Ya we may need to make such a need. User on 1.x has to first >> upgrade >> > > > (rolling upgrade supported) to latest version in 1.x and then to >> 2.0. >> > > > This was discussed some other place also. I believe in the mail >> thread >> > > > where we discussed abt rolling upgrade support in 2.0" >> > > > >> > > > Not able to find the original mail thread which discussed abt >> rolling >> > > > upgrade support in 2.0 Pls correct if took some thing wrong way.. >> > > > Ya I agree that this way of indirection might be inconvenience for >> > > > users. >> > > > >> > > > >> > > > -Anoop- >> > > > >> > > > On Wed, Dec 7, 2016 at 3:13 PM, Ted Yu <yuzhih...@gmail.com> wrote: >> > > > > bq. write old name only and within code we will have to map it >> with >> > new >> > > > > name. >> > > > > >> > > > > This is a viable approach. >> > > > > Even if we document upgrading to 1.2.3+, some users may not >> perform >> > > this >> > > > > step before upgrading to 2.0 >> > > > > >> > > > > On Tue, Dec 6, 2016 at 10:04 PM, Anoop John < >> anoop.hb...@gmail.com> >> > > > wrote: >> > > > > >> > > > >> Ya that bug raised by Enis was very valid.. So this means when >> > > > >> rolling upgrade happens, if some of the other RSs with version >> > <1.2.3 >> > > > >> , which is not having this fix, this issue might come up! >> > > > >> How to address then? Do we need to enforce a 1.2.3+ upgrade 1st >> > and >> > > > >> then only 2.0 rolling upgrade? >> > > > >> >> > > > >> Or else we will need to fix in 2.0.. We write the new Comparator >> > > > >> class name in FFT (trunk code) To fix, we might have to write >> old >> > > > >> name only and within code we will have to map it with new name. >> It >> > > > >> will be ugly! It can be fixed only in 3.0 may be.. But that can >> > make >> > > > >> the rolling upgrade story easier for users.. Just saying the >> > > > >> possibilities.. >> > > > >> >> > > > >> Thanks Ted to bring it up again. >> > > > >> >> > > > >> -Anoop- >> > > > >> >> > > > >> On Wed, Dec 7, 2016 at 10:04 AM, ramkrishna vasudevan >> > > > >> <ramkrishna.s.vasude...@gmail.com> wrote: >> > > > >> > I think when Enis reported the issue ( >> > > > >> > https://issues.apache.org/jira/browse/HBASE-16189), in rolling >> > > > upgrade >> > > > >> > regions could move around. So a region created in 2.0 moved to >> a >> > > > server >> > > > >> > with 1.x. >> > > > >> > >> > > > >> > Regards >> > > > >> > Ram >> > > > >> > >> > > > >> > >> > > > >> > On Wed, Dec 7, 2016 at 1:27 AM, Stack <st...@duboce.net> >> wrote: >> > > > >> > >> > > > >> >> On Tue, Dec 6, 2016 at 10:19 AM, Ted Yu <yuzhih...@gmail.com> >> > > wrote: >> > > > >> >> >> > > > >> >> > Looking at http://hbase.apache.org/book. >> > html#executing.the.0.96. >> > > > >> upgrade >> > > > >> >> , >> > > > >> >> > there is step of running "bin/hbase upgrade -check" >> > > > >> >> > >> > > > >> >> > How about adding a sample hfile which contains >> > > > >> >> > CellComparator$MetaCellComparator >> > > > >> >> > so that the upgrade check can read and verify ? >> > > > >> >> > >> > > > >> >> > >> > > > >> >> >> > > > >> >> We don't support downgrade. Never have. >> > > > >> >> St.Ack >> > > > >> >> >> > > > >> >> >> > > > >> >> >> > > > >> >> > On Tue, Dec 6, 2016 at 8:50 AM, Ted Yu <yuzhih...@gmail.com >> > >> > > > wrote: >> > > > >> >> > >> > > > >> >> > > The build I used yesterday didn't include HBASE-16189 >> > > > >> >> > > <https://issues.apache.org/jira/browse/HBASE-16189> >> > > > >> >> > > >> > > > >> >> > > Once it is included, the cluster can be downgraded fine. >> > > > >> >> > > >> > > > >> >> > > I wonder how users would know that their existing >> deployment >> > > has >> > > > >> >> > > HBASE-16189 <https://issues.apache.org/ >> > jira/browse/HBASE-16189 >> > > > >> > > > >> before >> > > > >> >> > > upgrading to 2.0 release. >> > > > >> >> > > >> > > > >> >> > > On Tue, Dec 6, 2016 at 2:29 AM, ramkrishna vasudevan < >> > > > >> >> > > ramkrishna.s.vasude...@gmail.com> wrote: >> > > > >> >> > > >> > > > >> >> > >> @Ted >> > > > >> >> > >> Does your version have this fix >> > > > >> >> > >> https://issues.apache.org/jira/browse/HBASE-16189 >> > > > >> >> > >> >> > > > >> >> > >> Regards >> > > > >> >> > >> Ram >> > > > >> >> > >> >> > > > >> >> > >> On Tue, Dec 6, 2016 at 3:56 PM, Ted Yu < >> yuzhih...@gmail.com >> > > >> > > > >> wrote: >> > > > >> >> > >> >> > > > >> >> > >> > Is the assumption that hbase:meta would not split ? >> > > > >> >> > >> > >> > > > >> >> > >> > In other thread, Francis Liu was proposing supporting >> > > > splittable >> > > > >> >> > >> > hbase:meta in 2.0 release. >> > > > >> >> > >> > >> > > > >> >> > >> > Cheers >> > > > >> >> > >> > >> > > > >> >> > >> > > On Dec 6, 2016, at 2:20 AM, 张铎 < >> palomino...@gmail.com> >> > > > wrote: >> > > > >> >> > >> > > >> > > > >> >> > >> > > Could this be solved by hosting meta only on master? >> > > > >> >> > >> > > >> > > > >> >> > >> > > BTW, MetaCellComparator is introduced in HBASE-10800. >> > > > >> >> > >> > > >> > > > >> >> > >> > > Thanks. >> > > > >> >> > >> > > >> > > > >> >> > >> > > 2016-12-06 17:44 GMT+08:00 Ted Yu < >> yuzhih...@gmail.com >> > >: >> > > > >> >> > >> > > >> > > > >> >> > >> > >> Hi, >> > > > >> >> > >> > >> When I restarted a cluster with 1.1 , I found that >> > > > hbase:meta >> > > > >> >> > region >> > > > >> >> > >> > >> (written to by the previously deployed 2.0) >> couldn't be >> > > > >> opened: >> > > > >> >> > >> > >> >> > > > >> >> > >> > >> Caused by: java.io.IOException: >> > > > >> >> > >> > >> org.apache.hadoop.hbase.io. >> > hfile.CorruptHFileException: >> > > > >> Problem >> > > > >> >> > >> reading >> > > > >> >> > >> > >> HFile Trailer from file hdfs:// >> yz1.xx.com:8020/apps/ >> > > > >> >> > >> hbase/data/data/ >> > > > >> >> > >> > >> hbase/meta/1588230740/info/ >> > > 599fc8a37311414e876803312009a9 >> > > > 86 >> > > > >> >> > >> > >> at >> > > > >> >> > >> > >> org.apache.hadoop.hbase.regionserver.HStore. >> > > > openStoreFiles( >> > > > >> >> > >> HStore.java: >> > > > >> >> > >> > >> 579) >> > > > >> >> > >> > >> at >> > > > >> >> > >> > >> org.apache.hadoop.hbase.regionserver.HStore. >> > > > loadStoreFiles( >> > > > >> >> > >> HStore.java: >> > > > >> >> > >> > >> 534) >> > > > >> >> > >> > >> at >> > > > >> >> > >> > >> org.apache.hadoop.hbase.regionserver.HStore.<init>( >> > > > >> >> > HStore.java:275) >> > > > >> >> > >> > >> at >> > > > >> >> > >> > >> org.apache.hadoop.hbase.regionserver.HRegion. >> > > > instantiateHSto >> > > > >> >> > >> re(HRegion. >> > > > >> >> > >> > >> java:5150) >> > > > >> >> > >> > >> at >> > > > >> >> > >> > >> org.apache.hadoop.hbase.region >> server.HRegion$1.call( >> > > > HRegion. >> > > > >> >> > >> java:912) >> > > > >> >> > >> > >> at >> > > > >> >> > >> > >> org.apache.hadoop.hbase.region >> server.HRegion$1.call( >> > > > HRegion. >> > > > >> >> > >> java:909) >> > > > >> >> > >> > >> at java.util.concurrent. >> > > FutureTask.run(FutureTask. >> > > > >> >> java:266) >> > > > >> >> > >> > >> at >> > > > >> >> > >> > >> java.util.concurrent.Executors >> $RunnableAdapter.call( >> > > > >> >> > >> Executors.java:511) >> > > > >> >> > >> > >> at java.util.concurrent. >> > > FutureTask.run(FutureTask. >> > > > >> >> java:266) >> > > > >> >> > >> > >> ... 3 more >> > > > >> >> > >> > >> Caused by: org.apache.hadoop.hbase.io. >> > > > >> >> hfile.CorruptHFileException: >> > > > >> >> > >> > Problem >> > > > >> >> > >> > >> reading HFile Trailer from file hdfs:// >> > > > >> >> > >> > >> yz1.xx.com:8020/apps/hbase/data/data/hbase/ >> > > > meta/1588230740/ >> > > > >> >> > >> > >> info/599fc8a37311414e876803312009a986 >> > > > >> >> > >> > >> at >> > > > >> >> > >> > >> org.apache.hadoop.hbase.io. >> > > hfile.HFile.pickReaderVersion( >> > > > >> >> > >> > HFile.java:483) >> > > > >> >> > >> > >> at >> > > > >> >> > >> > >> org.apache.hadoop.hbase.io.hfi >> le.HFile.createReader( >> > > > >> >> > HFile.java:511) >> > > > >> >> > >> > >> at >> > > > >> >> > >> > >> org.apache.hadoop.hbase.region >> server.StoreFile$Reader. >> > > > >> >> > >> > >> <init>(StoreFile.java:1128) >> > > > >> >> > >> > >> at >> > > > >> >> > >> > >> org.apache.hadoop.hbase.regionserver.StoreFileInfo. >> > > > >> >> > >> > >> open(StoreFileInfo.java:267) >> > > > >> >> > >> > >> at >> > > > >> >> > >> > >> org.apache.hadoop.hbase.region >> server.StoreFile.open( >> > > > StoreFil >> > > > >> >> > >> e.java:409) >> > > > >> >> > >> > >> at >> > > > >> >> > >> > >> org.apache.hadoop.hbase.regionserver.StoreFile. >> > > > >> >> > >> > >> createReader(StoreFile.java:517) >> > > > >> >> > >> > >> at >> > > > >> >> > >> > >> org.apache.hadoop.hbase.regionserver.HStore. >> > > > createStoreFileA >> > > > >> >> > >> ndReader( >> > > > >> >> > >> > >> HStore.java:687) >> > > > >> >> > >> > >> at >> > > > >> >> > >> > >> org.apache.hadoop.hbase.regionserver.HStore.access$ >> > 000( >> > > > >> >> > >> HStore.java:130) >> > > > >> >> > >> > >> at >> > > > >> >> > >> > >> org.apache.hadoop.hbase.regionserver.HStore$1.call( >> > > > >> >> > HStore.java:554) >> > > > >> >> > >> > >> at >> > > > >> >> > >> > >> org.apache.hadoop.hbase.regionserver.HStore$1.call( >> > > > >> >> > HStore.java:551) >> > > > >> >> > >> > >> ... 6 more >> > > > >> >> > >> > >> Caused by: java.io.IOException: java.lang. >> > > > >> >> ClassNotFoundException: >> > > > >> >> > >> > >> org.apache.hadoop.hbase.CellComparator$ >> > > MetaCellComparator >> > > > >> >> > >> > >> at >> > > > >> >> > >> > >> org.apache.hadoop.hbase.io.hfile.FixedFileTrailer. >> > > > getCompara >> > > > >> >> > >> torClass( >> > > > >> >> > >> > >> FixedFileTrailer.java:581) >> > > > >> >> > >> > >> at >> > > > >> >> > >> > >> org.apache.hadoop.hbase.io.hfile.FixedFileTrailer. >> > > > >> >> > deserializeFromPB( >> > > > >> >> > >> > >> FixedFileTrailer.java:300) >> > > > >> >> > >> > >> at >> > > > >> >> > >> > >> org.apache.hadoop.hbase.io.hfile.FixedFileTrailer. >> > > > >> >> > >> > >> deserialize(FixedFileTrailer.java:242) >> > > > >> >> > >> > >> at >> > > > >> >> > >> > >> org.apache.hadoop.hbase.io.hfile.FixedFileTrailer. >> > > > >> >> readFromStream( >> > > > >> >> > >> > >> FixedFileTrailer.java:407) >> > > > >> >> > >> > >> at >> > > > >> >> > >> > >> org.apache.hadoop.hbase.io. >> > > hfile.HFile.pickReaderVersion( >> > > > >> >> > >> > HFile.java:468) >> > > > >> >> > >> > >> ... 15 more >> > > > >> >> > >> > >> Caused by: java.lang.ClassNotFoundException: >> > > > >> >> > >> > >> org.apache.hadoop.hbase.CellComparator$ >> > > MetaCellComparator >> > > > >> >> > >> > >> at java.net.URLClassLoader. >> > > > >> findClass(URLClassLoader.java: >> > > > >> >> > 381) >> > > > >> >> > >> > >> at java.lang.ClassLoader. >> > > > loadClass(ClassLoader.java: >> > > > >> 424) >> > > > >> >> > >> > >> at sun.misc.Launcher$AppClassLoad >> er.loadClass( >> > > > >> >> > Launcher.java: >> > > > >> >> > >> 331) >> > > > >> >> > >> > >> at java.lang.ClassLoader. >> > > > loadClass(ClassLoader.java: >> > > > >> 357) >> > > > >> >> > >> > >> at java.lang.Class.forName0(Native Method) >> > > > >> >> > >> > >> at java.lang.Class.forName(Class.java:264) >> > > > >> >> > >> > >> at >> > > > >> >> > >> > >> org.apache.hadoop.hbase.io.hfile.FixedFileTrailer. >> > > > getCompara >> > > > >> >> > >> torClass( >> > > > >> >> > >> > >> FixedFileTrailer.java:579) >> > > > >> >> > >> > >> >> > > > >> >> > >> > >> When user does rolling upgrade from 1.1 to 2.0, the >> > above >> > > > may >> > > > >> >> cause >> > > > >> >> > >> > problem >> > > > >> >> > >> > >> if hbase:meta region is updated by server running >> 2.0 >> > but >> > > > >> later >> > > > >> >> > >> > assigned to >> > > > >> >> > >> > >> a region server which still runs 1.1 (due to crash >> of >> > the >> > > > >> server >> > > > >> >> > >> running >> > > > >> >> > >> > >> 2.0, e.g.) >> > > > >> >> > >> > >> >> > > > >> >> > >> > >> I want to get community feedback on the severity of >> > this >> > > > >> issue. >> > > > >> >> > >> > >> >> > > > >> >> > >> > >> Thanks >> > > > >> >> > >> > >> >> > > > >> >> > >> > >> > > > >> >> > >> >> > > > >> >> > > >> > > > >> >> > > >> > > > >> >> > >> > > > >> >> >> > > > >> >> > > > >> > > >> > >> > >
