[
https://issues.apache.org/jira/browse/CRUNCH-534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Micah Whitacre updated CRUNCH-534:
----------------------------------
Description:
With HBase's switch to ProtocolBuffers there is now an imposed restriction on
the size of data (64MB) which can be stored in protocol buffers and therefore
Cells/Puts/etc.
{quote}
Call to failed on local exception:
com.google.protobuf.InvalidProtocolBufferException: Protocol message was too
large. May be malicious. Use CodedInputStream.setSizeLimit() to increase the
size limit.
{quote}
While a single cell being that big seems extreme this limit also counts towards
Result objects so an entire row.
It is definitely an HBase problem[1] but Crunch also might have changes needed
since we use protobuf serialization in our HBaseTypes.[2] So might need to
make sure error isn't only on our side as they work through the issue or might
need to look at MOB support[3].
[1] - https://issues.apache.org/jira/browse/HBASE-13825
[2] -
https://github.com/apache/crunch/blob/d176778cf803374506cb7743069a05e28e07e2cf/crunch-hbase/src/main/java/org/apache/crunch/io/hbase/HBaseTypes.java#L33
[3] - https://issues.apache.org/jira/browse/HBASE-11339
was:
With HBase's switch to ProtocolBuffers there is now an imposed restriction on
the size of data which can be stored in protocol buffers and therefore
Cells/Puts/etc.
{quote}
Call to failed on local exception:
com.google.protobuf.InvalidProtocolBufferException: Protocol message was too
large. May be malicious. Use CodedInputStream.setSizeLimit() to increase the
size limit.
{quote}
It is definitely an HBase problem[1] but Crunch also might have changes needed
since we use protobuf serialization in our HBaseTypes.[2] So might need to
make sure error isn't only on our side as they work through the issue or might
need to look at MOB support[3].
[1] - https://issues.apache.org/jira/browse/HBASE-13825
[2] -
https://github.com/apache/crunch/blob/d176778cf803374506cb7743069a05e28e07e2cf/crunch-hbase/src/main/java/org/apache/crunch/io/hbase/HBaseTypes.java#L33
[3] - https://issues.apache.org/jira/browse/HBASE-11339
> Protobuf Size Limit Exception
> ------------------------------
>
> Key: CRUNCH-534
> URL: https://issues.apache.org/jira/browse/CRUNCH-534
> Project: Crunch
> Issue Type: Bug
> Components: IO
> Reporter: Micah Whitacre
>
> With HBase's switch to ProtocolBuffers there is now an imposed restriction on
> the size of data (64MB) which can be stored in protocol buffers and therefore
> Cells/Puts/etc.
> {quote}
> Call to failed on local exception:
> com.google.protobuf.InvalidProtocolBufferException: Protocol message was too
> large. May be malicious. Use CodedInputStream.setSizeLimit() to increase
> the size limit.
> {quote}
> While a single cell being that big seems extreme this limit also counts
> towards Result objects so an entire row.
> It is definitely an HBase problem[1] but Crunch also might have changes
> needed since we use protobuf serialization in our HBaseTypes.[2] So might
> need to make sure error isn't only on our side as they work through the issue
> or might need to look at MOB support[3].
> [1] - https://issues.apache.org/jira/browse/HBASE-13825
> [2] -
> https://github.com/apache/crunch/blob/d176778cf803374506cb7743069a05e28e07e2cf/crunch-hbase/src/main/java/org/apache/crunch/io/hbase/HBaseTypes.java#L33
> [3] - https://issues.apache.org/jira/browse/HBASE-11339
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
