[jira] [Commented] (HBASE-16567) Upgrade to protobuf3
[ https://issues.apache.org/jira/browse/HBASE-16567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15470479#comment-15470479 ] stack commented on HBASE-16567: --- HBASE-15638 is where I'm trying to find the classloader magic. tl;dr I've not found it yet. [~Apache9] > Upgrade to protobuf3 > > > Key: HBASE-16567 > URL: https://issues.apache.org/jira/browse/HBASE-16567 > Project: HBase > Issue Type: Task > Components: Protobufs >Affects Versions: 2.0.0 >Reporter: stack >Assignee: stack >Priority: Critical > Attachments: HBASE-16567.master.001.patch > > > Move master branch on to protobuf3. See > https://github.com/google/protobuf/releases We'd do it because pb3 saves some > on byte copies can work with offheap buffers -- needed for the off-heap write > path project -- though read-time is still a TODO (this means pb3 is not > enough; we'll have to patch it -- or patch pb2.5). > HBASE-15638 has us first shading protobufs before upgrading. Let us list here > issues just going to pb3 without shading if only for completeness sake; i.e. > do we have to shade? > * pb3 is by default wire compatible with pb2. > * protoc3 run against our .protos works fine except pb3 breaks our > HBaseZeroCopyLiteralByteString hack so this has to be removed (possibly > recast using new pb3 types) > * Starting up a cluster that is all pb3 seems to work fine. > * A pb2 branch-1 can read and write against the pb3 master cluster. > What will break if we just upgrade to pb3? > * We should be able to write HDFS messages on our AsyncWAL using pb3; the > pb2 HDFS should be able to read them (not tested). Or maybe not. See policy > here: https://github.com/google/protobuf/issues/1852 which seems to indicate > pb3s will not be able to write compatible pb2 Messages. TODO. > * Core Coprocessor Endpoints such as AccessControl seem to just work (their > protos will have been protoc3'd). I did simple test with a server from master > branch up on pb3 and then going against it with a branch-1 client on pb2. I > was able to add grants. > * For non-core CPEPs where the protos are pb2 still, it might just work. To > test. It would not be the end-of-the-world if they did not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16567) Upgrade to protobuf3
[
https://issues.apache.org/jira/browse/HBASE-16567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15469861#comment-15469861
]
Duo Zhang commented on HBASE-16567:
---
{quote}
For non-core CPEPs where the protos are pb2 still, it might just work. To test.
It would not be the end-of-the-world if they did not.
{quote}
Do we only accept Endpoints with pb2.5? I think it is really hard to support
both pb2.5 and pb3 for Endpoint? Maybe some classloader magic?
Thanks.
> Upgrade to protobuf3
>
>
> Key: HBASE-16567
> URL: https://issues.apache.org/jira/browse/HBASE-16567
> Project: HBase
> Issue Type: Task
> Components: Protobufs
>Affects Versions: 2.0.0
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: HBASE-16567.master.001.patch
>
>
> Move master branch on to protobuf3. See
> https://github.com/google/protobuf/releases We'd do it because pb3 saves some
> on byte copies can work with offheap buffers -- needed for the off-heap write
> path project -- though read-time is still a TODO (this means pb3 is not
> enough; we'll have to patch it -- or patch pb2.5).
> HBASE-15638 has us first shading protobufs before upgrading. Let us list here
> issues just going to pb3 without shading if only for completeness sake; i.e.
> do we have to shade?
> * pb3 is by default wire compatible with pb2.
> * protoc3 run against our .protos works fine except pb3 breaks our
> HBaseZeroCopyLiteralByteString hack so this has to be removed (possibly
> recast using new pb3 types)
> * Starting up a cluster that is all pb3 seems to work fine.
> * A pb2 branch-1 can read and write against the pb3 master cluster.
> What will break if we just upgrade to pb3?
> * We should be able to write HDFS messages on our AsyncWAL using pb3; the
> pb2 HDFS should be able to read them (not tested). Or maybe not. See policy
> here: https://github.com/google/protobuf/issues/1852 which seems to indicate
> pb3s will not be able to write compatible pb2 Messages. TODO.
> * Core Coprocessor Endpoints such as AccessControl seem to just work (their
> protos will have been protoc3'd). I did simple test with a server from master
> branch up on pb3 and then going against it with a branch-1 client on pb2. I
> was able to add grants.
> * For non-core CPEPs where the protos are pb2 still, it might just work. To
> test. It would not be the end-of-the-world if they did not.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16567) Upgrade to protobuf3
[ https://issues.apache.org/jira/browse/HBASE-16567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15469637#comment-15469637 ] ramkrishna.s.vasudevan commented on HBASE-16567: Ya. I just saw their page after adding that comment that it got released. > Upgrade to protobuf3 > > > Key: HBASE-16567 > URL: https://issues.apache.org/jira/browse/HBASE-16567 > Project: HBase > Issue Type: Task > Components: Protobufs >Affects Versions: 2.0.0 >Reporter: stack >Assignee: stack >Priority: Critical > Attachments: HBASE-16567.master.001.patch > > > Move master branch on to protobuf3. See > https://github.com/google/protobuf/releases We'd do it because pb3 saves some > on byte copies can work with offheap buffers -- needed for the off-heap write > path project -- though read-time is still a TODO (this means pb3 is not > enough; we'll have to patch it -- or patch pb2.5). > HBASE-15638 has us first shading protobufs before upgrading. Let us list here > issues just going to pb3 without shading if only for completeness sake; i.e. > do we have to shade? > * pb3 is by default wire compatible with pb2. > * protoc3 run against our .protos works fine except pb3 breaks our > HBaseZeroCopyLiteralByteString hack so this has to be removed (possibly > recast using new pb3 types) > * Starting up a cluster that is all pb3 seems to work fine. > * A pb2 branch-1 can read and write against the pb3 master cluster. > What will break if we just upgrade to pb3? > * We should be able to write HDFS messages on our AsyncWAL using pb3; the > pb2 HDFS should be able to read them (not tested). Or maybe not. See policy > here: https://github.com/google/protobuf/issues/1852 which seems to indicate > pb3s will not be able to write compatible pb2 Messages. TODO. > * Core Coprocessor Endpoints such as AccessControl seem to just work (their > protos will have been protoc3'd). I did simple test with a server from master > branch up on pb3 and then going against it with a branch-1 client on pb2. I > was able to add grants. > * For non-core CPEPs where the protos are pb2 still, it might just work. To > test. It would not be the end-of-the-world if they did not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16567) Upgrade to protobuf3
[
https://issues.apache.org/jira/browse/HBASE-16567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15469588#comment-15469588
]
ramkrishna.s.vasudevan commented on HBASE-16567:
One question on the pom upgrade.
When I did this some time back I found this,
Just adding the note I had stored when we did this upgrade
{code}
the actual protoc.exe version shows 3.0.0 but the protobuf-java .jars are of
3.0.0-beta2. so things does not work.
Since when we configure 3.0.0-beta 2 in hbase pom the protoc says it needs
3.0.0. When we configure 3.0.0 in hbase pom
it says canoot find 3.0.0 protobuf jar. Hence renmaed 3.0.0-beta2 jar to 3.0.0
protobuf jar.
{code}
Also how were you able to compile the .proto files under src/test folder?
> Upgrade to protobuf3
>
>
> Key: HBASE-16567
> URL: https://issues.apache.org/jira/browse/HBASE-16567
> Project: HBase
> Issue Type: Task
> Components: Protobufs
>Affects Versions: 2.0.0
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: HBASE-16567.master.001.patch
>
>
> Move master branch on to protobuf3. See
> https://github.com/google/protobuf/releases We'd do it because pb3 saves some
> on byte copies can work with offheap buffers -- needed for the off-heap write
> path project -- thought read-time is still a TODO.
> HBASE-15638 has us first shading protobufs before upgrading. Let us list here
> issues just going to pb3 without shading if only for completeness sake; i.e.
> do we have to shade?
> * pb3 is by default wire compatible with pb2.
> * protoc3 run against our .protos works fine except pb3 breaks our
> HBaseZeroCopyLiteralByteString hack.
> * Starting up a cluster that is all pb3'd seems to work fine.
> * A pb2 branch-1 can read and write against the pb3 master cluster.
> What will break if we just upgrade to pb3?
> * We should be able to write HDFS messages on our AsyncWAL using pb3; the
> pb2 HDFS should be able to read them (not tested). Or maybe not. See policy
> here: https://github.com/google/protobuf/issues/1852
> * Core Coprocessor Endpoints such as AccessControl seem to just work (their
> protos will have been protoc3'd). I did simple test with a server from master
> branch up on pb3 and then going against it with a branch-1 client on pb2. I
> was able to add grants.
> * For non-core CPEPs where the protos are pb2 still, it might just work. To
> test. It would not be the end-of-the-world if they did not.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16567) Upgrade to protobuf3
[
https://issues.apache.org/jira/browse/HBASE-16567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15469159#comment-15469159
]
stack commented on HBASE-16567:
---
Just a note to say that upgrading our pb to pb3 w/o running protoc3, we do
pretty good but fail reading trailers on hfiles. We can't find CellComparator.
TODO: check why.
{code}
2016-09-06 18:06:42,856 ERROR [RS_OPEN_META-localhost:55129-0]
handler.OpenRegionHandler: Failed open of region=hbase:meta,,1.1588230740,
starting to roll back the global memstore size.
java.io.IOException: java.io.IOException:
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading HFile
Trailer from file
file:/var/folders/cj/jgfy62h13vz019xgz681df_rgp/T/hbase-stack/hbase/data/hbase/meta/1588230740/info/47a6d1f4adb24f6e92de8dbe9e9c144f
at
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionStores(HRegion.java:831)
at
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:732)
at
org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:705)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4750)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4721)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4693)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4649)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4600)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:276)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:103)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException:
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading HFile
Trailer from file
file:/var/folders/cj/jgfy62h13vz019xgz681df_rgp/T/hbase-stack/hbase/data/hbase/meta/1588230740/info/47a6d1f4adb24f6e92de8dbe9e9c144f
at
org.apache.hadoop.hbase.regionserver.HStore.openStoreFiles(HStore.java:544)
at
org.apache.hadoop.hbase.regionserver.HStore.loadStoreFiles(HStore.java:499)
at org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:267)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:3652)
at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:805)
at org.apache.hadoop.hbase.regionserver.HRegion$1.call(HRegion.java:802)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem
reading HFile Trailer from file
file:/var/folders/cj/jgfy62h13vz019xgz681df_rgp/T/hbase-stack/hbase/data/hbase/meta/1588230740/info/47a6d1f4adb24f6e92de8dbe9e9c144f
at
org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:477)
at org.apache.hadoop.hbase.io.hfile.HFile.createReader(HFile.java:505)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.(StoreFile.java:1033)
at
org.apache.hadoop.hbase.regionserver.StoreFileInfo.open(StoreFileInfo.java:241)
at
org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:365)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:462)
at
org.apache.hadoop.hbase.regionserver.HStore.createStoreFileAndReader(HStore.java:629)
at
org.apache.hadoop.hbase.regionserver.HStore.access$000(HStore.java:123)
at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:519)
at org.apache.hadoop.hbase.regionserver.HStore$1.call(HStore.java:516)
... 6 more
Caused by: java.io.IOException: java.lang.ClassNotFoundException:
org.apache.hadoop.hbase.CellComparator$MetaCellComparator
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.getComparatorClass(FixedFileTrailer.java:581)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.deserializeFromPB(FixedFileTrailer.java:300)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.deserialize(FixedFileTrailer.java:242)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:407)
at
org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:462)
... 15 more
Caused by: java.lang.ClassNotFoundException:
[jira] [Commented] (HBASE-16567) Upgrade to protobuf3
[ https://issues.apache.org/jira/browse/HBASE-16567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15468971#comment-15468971 ] stack commented on HBASE-16567: --- Attached patch is pretty much same as a patch that [~ram_krish] and [~anoop.hbase] have been slinging around for a while only in this case, ByteStringer and HBaseZeroCopyByteString are removed. Posting to see what fails in our test run. > Upgrade to protobuf3 > > > Key: HBASE-16567 > URL: https://issues.apache.org/jira/browse/HBASE-16567 > Project: HBase > Issue Type: Task > Components: Protobufs >Affects Versions: 2.0.0 >Reporter: stack >Assignee: stack >Priority: Critical > Attachments: HBASE-16567.master.001.patch > > > Move master branch on to protobuf3. See > https://github.com/google/protobuf/releases We'd do it because pb3 saves some > on byte copies can work with offheap buffers -- needed for the off-heap write > path project -- thought read-time is still a TODO. > HBASE-15638 has us first shading protobufs before upgrading. Let us list here > issues just going to pb3 without shading if only for completeness sake; i.e. > do we have to shade? > * pb3 is by default wire compatible with pb2. > * protoc3 run against our .protos works fine except pb3 breaks our > HBaseZeroCopyLiteralByteString hack. > * Starting up a cluster that is all pb3'd seems to work fine. > * A pb2 branch-1 can read and write against the pb3 master cluster. > What will break if we just upgrade to pb3? > * We should be able to write HDFS messages on our AsyncWAL using pb3; the > pb2 HDFS should be able to read them (not tested). Or maybe not. See policy > here: https://github.com/google/protobuf/issues/1852 > * Core Coprocessor Endpoints such as AccessControl seem to just work (their > protos will have been protoc3'd). I did simple test with a server from master > branch up on pb3 and then going against it with a branch-1 client on pb2. I > was able to add grants. > * For non-core CPEPs where the protos are pb2 still, it might just work. To > test. It would not be the end-of-the-world if they did not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
