[jira] [Updated] (HBASE-15638) Shade protobuf

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15638:
--
   Resolution: Fixed
Fix Version/s: 2.0.0
 Release Note: 
Shade/relocate and include the protobuf we use internally. See protobuf chapter 
in the refguide for more on how we protobuf in hbase-.2.0.0 and going forward.

See 
https://docs.google.com/document/d/1H4NgLXQ9Y9KejwobddCqaVMEDCGbyDcXtdF5iAfDIEk/edit#
 for how we arrived at this approach.

See 
http://mail-archives.apache.org/mod_mbox/hbase-dev/201610.mbox/%3C07850EDD-7230-431B-9AB0-C5C91B105EEC%40gmail.com%3E
 for discussion around merging this change and of how we might revert if an 
alternative to this awkward patch presents itself; e.g. an hadoop with 
CLASSPATH isolation (and means of dealing with Sparks use of protobuf 2.5.0, 
etc.)
   Status: Resolved  (was: Patch Available)

Above set of patches adds shading of protobufs (and upgrades our internal 
protobuf to be 3.1.x). Resolving. See release note for pointers on how we got 
to this awkward patch and of how we would unto it if circumstances change.

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, HBASE-15638.master.007.patch, 
> HBASE-15638.master.007.patch, HBASE-15638.master.008.patch, 
> HBASE-15638.master.009.patch, as.far.as.server.patch
>
>
> We need to change our protobuf. Currently it is pb2.5.0. As is, protobufs 
> expect all buffers to be on-heap byte arrays. It does not have facility for 
> dealing in ByteBuffers and off-heap ByteBuffers in particular. This fact 
> frustrates the off-heaping-of-the-write-path project as 
> marshalling/unmarshalling of protobufs involves a copy on-heap first.
> So, we need to patch our protobuf so it supports off-heap ByteBuffers. To 
> ensure we pick up the patched protobuf always, we need to relocate/shade our 
> protobuf and adjust all protobuf references accordingly.
> Given as we have protobufs in our public facing API, Coprocessor Endpoints -- 
> which use protobuf Service to describe new API -- a blind relocation/shading 
> of com.google.protobuf.* will break our API for CoProcessor EndPoints (CPEP) 
> in particular. For example, in the Table Interface, to invoke a method on a 
> registered CPEP, we have:
> {code} Map 
> coprocessorService(
> Class service, byte[] startKey, byte[] endKey, 
> org.apache.hadoop.hbase.client.coprocessor.Batch.Call 
> callable)
> throws com.google.protobuf.ServiceException, Throwable{code}
> This issue is how we intend to shade protobuf for hbase-2.0.0 while 
> preserving our API as is so CPEPs continue to work on the new hbase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15638) Shade protobuf

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15638:
--
Description: 
We need to change our protobuf. Currently it is pb2.5.0. As is, protobufs 
expect all buffers to be on-heap byte arrays. It does not have facility for 
dealing in ByteBuffers and off-heap ByteBuffers in particular. This fact 
frustrates the off-heaping-of-the-write-path project as 
marshalling/unmarshalling of protobufs involves a copy on-heap first.

So, we need to patch our protobuf so it supports off-heap ByteBuffers. To 
ensure we pick up the patched protobuf always, we need to relocate/shade our 
protobuf and adjust all protobuf references accordingly.

Given as we have protobufs in our public facing API, Coprocessor Endpoints -- 
which use protobuf Service to describe new API -- a blind relocation/shading of 
com.google.protobuf.* will break our API for CoProcessor EndPoints (CPEP) in 
particular. For example, in the Table Interface, to invoke a method on a 
registered CPEP, we have:

{code} Map 
coprocessorService(
Class service, byte[] startKey, byte[] endKey,   
  org.apache.hadoop.hbase.client.coprocessor.Batch.Call 
callable)
throws com.google.protobuf.ServiceException, Throwable{code}

This issue is how we intend to shade protobuf for hbase-2.0.0 while preserving 
our API as is so CPEPs continue to work on the new hbase.

  was:
We need to change our protobuf. Currently it is pb2.5.0. As is, protobufs 
expect all buffers to be on-heap byte arrays. It does not have facility for 
dealing in ByteBuffers and off-heap ByteBuffers in particular. This fact 
frustrates the off-heaping-of-the-write-path project as 
marshalling/unmarshalling of protobufs involves a copy on-heap first.

So, we need to patch our protobuf so it supports off-heap ByteBuffers. To 
ensure we pick up the patched protobuf always, we need to relocate/shade our 
protobuf and adjust all protobuf references accordingly.

Given as we have protobufs in our public facing API, Coprocessor Endpoints -- 
which use protobuf Service to describe new API -- a blind relocation/shading of 
com.google.protobuf.* will break our API for CoProcessor EndPoints (CPEP) in 
particular. For example, in the Table Interface, to invoke a method on a 
registered CPEP, we have:

{{ Map coprocessorService(
Class service, byte[] startKey, byte[] endKey,   
  org.apache.hadoop.hbase.client.coprocessor.Batch.Call 
callable)
throws com.google.protobuf.ServiceException, Throwable}}

This issue is how we intend to shade protobuf for hbase-2.0.0 while preserving 
our API as is so CPEPs continue to work on the new hbase.


> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, HBASE-15638.master.007.patch, 
> HBASE-15638.master.007.patch, HBASE-15638.master.008.patch, 
> HBASE-15638.master.009.patch, as.far.as.server.patch
>
>
> We need to change our protobuf. Currently it is pb2.5.0. As is, protobufs 
> expect all buffers to be on-heap byte arrays. It does not have facility for 
> dealing in ByteBuffers and off-heap ByteBuffers in particular. This fact 
> frustrates the off-heaping-of-the-write-path project as 
> marshalling/unmarshalling of protobufs involves a copy on-heap first.
> So, we need to patch our protobuf so it supports off-heap ByteBuffers. To 
> ensure we pick up the patched protobuf always, we need to relocate/shade our 
> protobuf and adjust all protobuf references accordingly.
> Given as we have protobufs in our public facing API, Coprocessor Endpoints -- 
> which use protobuf Service to describe new API -- a blind relocation/shading 
> of com.google.protobuf.* will break our API for CoProcessor EndPoints (CPEP) 
> in particular. For example, in the Table Interface, to invoke a method on a 
> registered CPEP, we have:
> {code} Map 
> coprocessorService(
> Class service, byte[] startKey, byte[] endKey, 
> org.apache.hadoop.hbase.client.coprocessor.Batch.Call 
> callable)
> throws com.google.protobuf.ServiceException, Throwable{code}
> This issue is how we intend to shade protobuf for hbase-2.0.0 while 
> preserving our API as is so CPEPs continue to work on the new 

[jira] [Updated] (HBASE-15638) Shade protobuf

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15638:
--
Description: 
We need to change our protobuf. Currently it is pb2.5.0. As is, protobufs 
expect all buffers to be on-heap byte arrays. It does not have facility for 
dealing in ByteBuffers and off-heap ByteBuffers in particular. This fact 
frustrates the off-heaping-of-the-write-path project as 
marshalling/unmarshalling of protobufs involves a copy on-heap first.

So, we need to patch our protobuf so it supports off-heap ByteBuffers. To 
ensure we pick up the patched protobuf always, we need to relocate/shade our 
protobuf and adjust all protobuf references accordingly.

Given as we have protobufs in our public facing API, Coprocessor Endpoints -- 
which use protobuf Service to describe new API -- a blind relocation/shading of 
com.google.protobuf.* will break our API for CoProcessor EndPoints (CPEP) in 
particular. For example, in the Table Interface, to invoke a method on a 
registered CPEP, we have:

{{ Map coprocessorService(
Class service, byte[] startKey, byte[] endKey,   
  org.apache.hadoop.hbase.client.coprocessor.Batch.Call 
callable)
throws com.google.protobuf.ServiceException, Throwable}}

This issue is how we intend to shade protobuf for hbase-2.0.0 while preserving 
our API as is so CPEPs continue to work on the new hbase.

  was:
We need to change our protobuf. Currently it is pb2.5.0. As is, protobufs 
expect all buffers to be on-heap byte arrays. It does not have facility for 
dealing in ByteBuffers and off-heap ByteBuffers in particular. This fact 
frustrates the off-heaping-of-the-write-path project as 
marshalling/unmarshalling of protobufs involves a copy on-heap first.

So, we need to patch our protobuf so it supports off-heap ByteBuffers. To 
ensure we pick up the patched protobuf always, we need to relocate/shade our 
protobuf and adjust all protobuf references accordingly.

Given as we have protobufs in our public facing API, Coprocessor Endpoints -- 
which use protobuf Service to describe new API -- a blind relocation/shading of 
com.google.protobuf.* will break our API for CoProcessor EndPoints (CPEP) in 
particular. For example, in the Table Interface, to invoke a method on a 
registered CPEP, we have:

 Map coprocessorService(
Class service, byte[] startKey, byte[] endKey,   
  org.apache.hadoop.hbase.client.coprocessor.Batch.Call 
callable)
throws com.google.protobuf.ServiceException, Throwable

This issue is how we intend to shade protobuf for hbase-2.0.0 while preserving 
our API as is so CPEPs continue to work on the new hbase.


> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, HBASE-15638.master.007.patch, 
> HBASE-15638.master.007.patch, HBASE-15638.master.008.patch, 
> HBASE-15638.master.009.patch, as.far.as.server.patch
>
>
> We need to change our protobuf. Currently it is pb2.5.0. As is, protobufs 
> expect all buffers to be on-heap byte arrays. It does not have facility for 
> dealing in ByteBuffers and off-heap ByteBuffers in particular. This fact 
> frustrates the off-heaping-of-the-write-path project as 
> marshalling/unmarshalling of protobufs involves a copy on-heap first.
> So, we need to patch our protobuf so it supports off-heap ByteBuffers. To 
> ensure we pick up the patched protobuf always, we need to relocate/shade our 
> protobuf and adjust all protobuf references accordingly.
> Given as we have protobufs in our public facing API, Coprocessor Endpoints -- 
> which use protobuf Service to describe new API -- a blind relocation/shading 
> of com.google.protobuf.* will break our API for CoProcessor EndPoints (CPEP) 
> in particular. For example, in the Table Interface, to invoke a method on a 
> registered CPEP, we have:
> {{ Map coprocessorService(
> Class service, byte[] startKey, byte[] endKey, 
> org.apache.hadoop.hbase.client.coprocessor.Batch.Call 
> callable)
> throws com.google.protobuf.ServiceException, Throwable}}
> This issue is how we intend to shade protobuf for hbase-2.0.0 while 
> preserving our API as is so CPEPs continue to work on the new hbase.



--
This 

[jira] [Updated] (HBASE-15638) Shade protobuf

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15638:
--
Description: 
We need to change our protobuf. Currently it is pb2.5.0. As is, protobufs 
expect all buffers to be on-heap byte arrays. It does not have facility for 
dealing in ByteBuffers and off-heap ByteBuffers in particular. This fact 
frustrates the off-heaping-of-the-write-path project as 
marshalling/unmarshalling of protobufs involves a copy on-heap first.

So, we need to patch our protobuf so it supports off-heap ByteBuffers. To 
ensure we pick up the patched protobuf always, we need to relocate/shade our 
protobuf and adjust all protobuf references accordingly.

Given as we have protobufs in our public facing API, Coprocessor Endpoints -- 
which use protobuf Service to describe new API -- a blind relocation/shading of 
com.google.protobuf.* will break our API for CoProcessor EndPoints (CPEP) in 
particular. For example, in the Table Interface, to invoke a method on a 
registered CPEP, we have:

 Map coprocessorService(
Class service, byte[] startKey, byte[] endKey,   
  org.apache.hadoop.hbase.client.coprocessor.Batch.Call 
callable)
throws com.google.protobuf.ServiceException, Throwable

This issue is how we intend to shade protobuf for hbase-2.0.0 while preserving 
our API as is so CPEPs continue to work on the new hbase.

  was:
Shade protobufs so we can move to a different version without breaking the 
world. We want to get up on pb3 because it has unsafe methods that allow us 
save on copies; it also has some means of dealing with BBs so we can pass it 
offheap DBBs. We'll probably want to change PB3 to open it up some more too so 
we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] and 
[~ram_krish]'s offheaping of the readpath work.

This change is mostly straight-forward but there are some tricky bits:

 # How to interface with HDFS? It wants its ByteStrings. Here in particular in 
FanOutOneBlockAsyncDFSOutputSaslHelper:

{code}
  if (payload != null) {
builder.setPayload(ByteString.copyFrom(payload));
  }
{code}

 # [~busbey] also points out that we need to take care of endpoints done as pb. 
Test at least.

Let me raise this one on the dev list too.



> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, HBASE-15638.master.007.patch, 
> HBASE-15638.master.007.patch, HBASE-15638.master.008.patch, 
> HBASE-15638.master.009.patch, as.far.as.server.patch
>
>
> We need to change our protobuf. Currently it is pb2.5.0. As is, protobufs 
> expect all buffers to be on-heap byte arrays. It does not have facility for 
> dealing in ByteBuffers and off-heap ByteBuffers in particular. This fact 
> frustrates the off-heaping-of-the-write-path project as 
> marshalling/unmarshalling of protobufs involves a copy on-heap first.
> So, we need to patch our protobuf so it supports off-heap ByteBuffers. To 
> ensure we pick up the patched protobuf always, we need to relocate/shade our 
> protobuf and adjust all protobuf references accordingly.
> Given as we have protobufs in our public facing API, Coprocessor Endpoints -- 
> which use protobuf Service to describe new API -- a blind relocation/shading 
> of com.google.protobuf.* will break our API for CoProcessor EndPoints (CPEP) 
> in particular. For example, in the Table Interface, to invoke a method on a 
> registered CPEP, we have:
>  Map coprocessorService(
> Class service, byte[] startKey, byte[] endKey, 
> org.apache.hadoop.hbase.client.coprocessor.Batch.Call 
> callable)
> throws com.google.protobuf.ServiceException, Throwable
> This issue is how we intend to shade protobuf for hbase-2.0.0 while 
> preserving our API as is so CPEPs continue to work on the new hbase.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15638) Shade protobuf

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15638:
--
Attachment: HBASE-15638.master.009.patch

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, HBASE-15638.master.007.patch, 
> HBASE-15638.master.007.patch, HBASE-15638.master.008.patch, 
> HBASE-15638.master.009.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15638) Shade protobuf

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15638:
--
Attachment: HBASE-15638.master.008.patch

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, HBASE-15638.master.007.patch, 
> HBASE-15638.master.007.patch, HBASE-15638.master.008.patch, 
> as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15638) Shade protobuf

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15638:
--
Status: Patch Available  (was: In Progress)

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, HBASE-15638.master.007.patch, 
> HBASE-15638.master.007.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15638) Shade protobuf

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15638:
--
Status: In Progress  (was: Patch Available)

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, HBASE-15638.master.007.patch, 
> HBASE-15638.master.007.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15638) Shade protobuf

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15638:
--
Attachment: HBASE-15638.master.007.patch

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, HBASE-15638.master.007.patch, 
> HBASE-15638.master.007.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15638) Shade protobuf

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15638:
--
Attachment: HBASE-15638.master.007.patch

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, HBASE-15638.master.007.patch, 
> as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15638) Shade protobuf

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15638:
--
Attachment: HBASE-15638.master.006.patch

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> HBASE-15638.master.006.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15638) Shade protobuf

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15638:
--
Attachment: HBASE-15638.master.005.patch

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, HBASE-15638.master.005.patch, 
> as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15638) Shade protobuf

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15638:
--
Attachment: HBASE-15638.master.004.patch

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.004.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15638) Shade protobuf

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15638:
--
Attachment: HBASE-15638.master.003 (1).patch

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15638) Shade protobuf

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15638:
--
Status: Patch Available  (was: Open)

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.003.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15638) Shade protobuf

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15638:
--
Attachment: HBASE-15638.master.003 (1).patch

Trying... again 

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003 (1).patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.003.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15638) Shade protobuf

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15638:
--
Attachment: HBASE-15638.master.003 (1).patch

Retry. Don't know why this patch is getting ignored.

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003 (1).patch, 
> HBASE-15638.master.003.patch, HBASE-15638.master.003.patch, 
> as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15638) Shade protobuf

[Vivek Koppuru (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vivek Koppuru updated HBASE-15638:
--
Attachment: (was: read-write-rpc-timeouts.patch)

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.003.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15638) Shade protobuf

[Vivek Koppuru (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vivek Koppuru updated HBASE-15638:
--
Attachment: read-write-rpc-timeouts.patch

These are all the necessary changes to split "hbase.rpc.timeout" into 
"hbase.rpc.read.timeout" and "hbase.rpc.write.timeout". 

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.003.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15638) Shade protobuf

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15638:
--
Attachment: HBASE-15638.master.003.patch

Retry

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003.patch, 
> HBASE-15638.master.003.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15638) Shade protobuf

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15638:
--
Attachment: HBASE-15638.master.003.patch

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, HBASE-15638.master.003.patch, 
> as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15638) Shade protobuf

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15638:
--
Attachment: HBASE-15638.master.002.patch

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> HBASE-15638.master.002.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15638) Shade protobuf

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15638:
--
Attachment: HBASE-15638.master.001.patch

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch, 
> as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15638) Shade protobuf

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15638:
--
Assignee: stack

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15638) Shade protobuf

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15638:
--
Priority: Critical  (was: Major)
Hadoop Flags: Incompatible change

This is looking to be an incompatible change that breaks endpoint CPs at least. 
The CP rpc channels take pb Descriptors and an RpcController to exec. The 
endpoint would have to be regenerated using our pb version and with the 
appropriate shade offset in the packaging. Any thoughts here [~ghelmling]?

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
>Priority: Critical
> Attachments: 15638v2.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15638) Shade protobuf

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15638:
--
Attachment: 15638v2.patch

Progress. I got over the FanOutOneBLockAsyncDFSOutput... issue where an HDFS 
method expected a com.google.protobuf.ByteString by explicitly referencing 
c.g.p.ByteString; the HDFS pb2.5 will be transitively included so this explicit 
reference will be satisfied (we'll have two pbs on our CLASSPATH; hadoops pb2.5 
transitively included and our own relocated pb3).

This patch excludes hbase-rest and hbase-spark for now.  These both fail 
because they have .proto files local to the modules. I intend to move the 
protos out and back into hbase-protocol module. This breaks 
all-spark-related-code is in hbase-spark but at the expense of keeping our 
pb'ing mess all in one place.

If folks are fine w/ above approach, I'll proceed. Will let this percolate a 
while in case better suggestions or objection.

[~sergey.soldatov] and [~busbey] FYI

Thanks.

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
> Attachments: 15638v2.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15638) Shade protobuf

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-15638:
--
Attachment: as.far.as.server.patch

I got as far as server module. Stuck on how to pass a ByteString to HDFS in 
FanOutOneBlockAsyncDFSOutputSaslHelper and then the test of ipc which builds 
its own classes from protos...  Need to fix work on these.

> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
>  Issue Type: Bug
>  Components: Protobufs
>Reporter: stack
> Attachments: as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the 
> world. We want to get up on pb3 because it has unsafe methods that allow us 
> save on copies; it also has some means of dealing with BBs so we can pass it 
> offheap DBBs. We'll probably want to change PB3 to open it up some more too 
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase] 
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
>  # How to interface with HDFS? It wants its ByteStrings. Here in particular 
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
>   if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
>   }
> {code}
>  # [~busbey] also points out that we need to take care of endpoints done as 
> pb. Test at least.
> Let me raise this one on the dev list too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)