Re: Large number of column qualifiers

[Gaurav Agarwal]

Hi Ted,

Thanks for the reply. As I understood from 11544 description, it deals with
optimising the server response by chunking it - in order to avoid very
large memory allocations..

In my case, I have enough memory to hold the results; that is not the
problem. Rather the Cell object itself has a hard-limit of holding 32KB of
columnQualifier metadata. In my case the number of columns (and thus their
size) far exceeds this limit.

Please let me know if I missed something in your suggestion.

On Wed, Sep 23, 2015 at 10:54 PM, Ted Yu  wrote:

> Please take a look at HBASE-11544 which is in hbase 1.1
>
> Cheers
>
> On Wed, Sep 23, 2015 at 10:18 AM, Gaurav Agarwal  wrote:
>
> > Hi All,
> >
> > I have Column Family with very large number of column qualifiers (>
> > 50,000). Each column qualifier is 8 bytes long. The problem is the when I
> > do a scan operation to fetch some rows, the client side Cell object does
> > not have enough space allocated in it to hold all the columnQaulifiers
> for
> > a given row and hence I cannot read all the columns back for a given row.
> >
> > Please see the code snippet that I am using:
> >
> >  final ResultScanner rs = htable.getScanner(scan);
> >  for (Result row = rs.next(); row != null; row = rs.next()) {
> > final Cell[] cells = row.rawCells();
> > if (cells != null) {
> > for (final Cell cell : cells) {
> > final long c = Bytes.toLong(
> > *cell.getQualifierArray()*,
> cell.getQualifierOffset(),
> > cell.getQualifierLength());
> > final long v = Bytes.toLong(cell.getValueArray(),
> > cell.getValueOffset());
> > points.put(c, v);
> > }
> > }
> > }
> >
> > The cell.getQualifierArray() method says that it's 'Max length is
> > Short.MAX_VALUE which is 32,767 bytes.'. Hence it can only hold around
> > 4,000 columnQualfiers.
> >
> > Is there an alternate API that I should be using or am I missing some
> > setting here? Note that in worst case I need to read all the
> > columnQualifiers in a row and I may or may not know a subset to fetch in
> > advance.
> >
> > Even if this is not possible in a single call, is there a way to cursor
> > through the columnQualifiers?
> >
> > I am presently using Hbase 0.96 client but can switch to Hbase 1.x if
> there
> > is an API in the newer version.
> >
> > --cheers, gaurav
> >
> > --
> > --cheers, gaurav
> >
>



-- 
--cheers, gaurav

Large number of column qualifiers

[Gaurav Agarwal]

Hi All,

I have Column Family with very large number of column qualifiers (>
50,000). Each column qualifier is 8 bytes long. The problem is the when I
do a scan operation to fetch some rows, the client side Cell object does
not have enough space allocated in it to hold all the columnQaulifiers for
a given row and hence I cannot read all the columns back for a given row.

Please see the code snippet that I am using:

 final ResultScanner rs = htable.getScanner(scan);
 for (Result row = rs.next(); row != null; row = rs.next()) {
final Cell[] cells = row.rawCells();
if (cells != null) {
for (final Cell cell : cells) {
final long c = Bytes.toLong(
*cell.getQualifierArray()*, cell.getQualifierOffset(),
cell.getQualifierLength());
final long v = Bytes.toLong(cell.getValueArray(),
cell.getValueOffset());
points.put(c, v);
}
}
}

The cell.getQualifierArray() method says that it's 'Max length is
Short.MAX_VALUE which is 32,767 bytes.'. Hence it can only hold around
4,000 columnQualfiers.

Is there an alternate API that I should be using or am I missing some
setting here? Note that in worst case I need to read all the
columnQualifiers in a row and I may or may not know a subset to fetch in
advance.

Even if this is not possible in a single call, is there a way to cursor
through the columnQualifiers?

I am presently using Hbase 0.96 client but can switch to Hbase 1.x if there
is an API in the newer version.

--cheers, gaurav

-- 
--cheers, gaurav

Re: Large number of column qualifiers

[Gaurav Agarwal]

Thanks Vlad. Could you please point me the KV size setting (default 1MB)?
Just to make sure that I understand correct, are you suggesting that the
following comment is incorrect in Cell.java?

 /**
   * Contiguous raw bytes that may start at any index in the containing
array. Max length is
   * Short.MAX_VALUE which is 32,767 bytes.
   * @return The array containing the qualifier bytes.
   */
  byte[] getQualifierArray();

On Thu, Sep 24, 2015 at 12:10 AM, Gaurav Agarwal  wrote:

> Thanks Vlad. Could you please point me the KV size setting (default 1MB)?
> Just to make sure that I understand correct - the following comment is
> incorrect in Cell.java:
>
>  /**
>* Contiguous raw bytes that may start at any index in the containing
> array. Max length is
>* Short.MAX_VALUE which is 32,767 bytes.
>* @return The array containing the qualifier bytes.
>*/
>   byte[] getQualifierArray();
>
> On Wed, Sep 23, 2015 at 11:43 PM, Vladimir Rodionov <
> vladrodio...@gmail.com> wrote:
>
>> Check KeyValue class (Cell's implementation). getQualifierArray() returns
>> kv's backing array. There is no SHORT limit on a size of this array, but
>> there are other limits in  HBase - maximum KV size, for example, which is
>> configurable, but, by default, is 1MB. Having 50K qualifiers is a bad
>> idea.
>> Consider redesigning your data model and use rowkey instead.
>>
>> -Vlad
>>
>> On Wed, Sep 23, 2015 at 10:24 AM, Ted Yu  wrote:
>>
>> > Please take a look at HBASE-11544 which is in hbase 1.1
>> >
>> > Cheers
>> >
>> > On Wed, Sep 23, 2015 at 10:18 AM, Gaurav Agarwal 
>> wrote:
>> >
>> > > Hi All,
>> > >
>> > > I have Column Family with very large number of column qualifiers (>
>> > > 50,000). Each column qualifier is 8 bytes long. The problem is the
>> when I
>> > > do a scan operation to fetch some rows, the client side Cell object
>> does
>> > > not have enough space allocated in it to hold all the columnQaulifiers
>> > for
>> > > a given row and hence I cannot read all the columns back for a given
>> row.
>> > >
>> > > Please see the code snippet that I am using:
>> > >
>> > >  final ResultScanner rs = htable.getScanner(scan);
>> > >  for (Result row = rs.next(); row != null; row = rs.next()) {
>> > > final Cell[] cells = row.rawCells();
>> > > if (cells != null) {
>> > > for (final Cell cell : cells) {
>> > > final long c = Bytes.toLong(
>> > > *cell.getQualifierArray()*,
>> > cell.getQualifierOffset(),
>> > > cell.getQualifierLength());
>> > > final long v = Bytes.toLong(cell.getValueArray(),
>> > > cell.getValueOffset());
>> > > points.put(c, v);
>> > > }
>> > > }
>> > > }
>> > >
>> > > The cell.getQualifierArray() method says that it's 'Max length is
>> > > Short.MAX_VALUE which is 32,767 bytes.'. Hence it can only hold around
>> > > 4,000 columnQualfiers.
>> > >
>> > > Is there an alternate API that I should be using or am I missing some
>> > > setting here? Note that in worst case I need to read all the
>> > > columnQualifiers in a row and I may or may not know a subset to fetch
>> in
>> > > advance.
>> > >
>> > > Even if this is not possible in a single call, is there a way to
>> cursor
>> > > through the columnQualifiers?
>> > >
>> > > I am presently using Hbase 0.96 client but can switch to Hbase 1.x if
>> > there
>> > > is an API in the newer version.
>> > >
>> > > --cheers, gaurav
>> > >
>> > > --
>> > > --cheers, gaurav
>> > >
>> >
>>
>
>
>
> --
> --cheers, gaurav
>



-- 
--cheers, gaurav

Re: Large number of column qualifiers

[Vladimir Rodionov]

Check KeyValue class (Cell's implementation). getQualifierArray() returns
kv's backing array. There is no SHORT limit on a size of this array, but
there are other limits in  HBase - maximum KV size, for example, which is
configurable, but, by default, is 1MB. Having 50K qualifiers is a bad idea.
Consider redesigning your data model and use rowkey instead.

-Vlad

On Wed, Sep 23, 2015 at 10:24 AM, Ted Yu  wrote:

> Please take a look at HBASE-11544 which is in hbase 1.1
>
> Cheers
>
> On Wed, Sep 23, 2015 at 10:18 AM, Gaurav Agarwal  wrote:
>
> > Hi All,
> >
> > I have Column Family with very large number of column qualifiers (>
> > 50,000). Each column qualifier is 8 bytes long. The problem is the when I
> > do a scan operation to fetch some rows, the client side Cell object does
> > not have enough space allocated in it to hold all the columnQaulifiers
> for
> > a given row and hence I cannot read all the columns back for a given row.
> >
> > Please see the code snippet that I am using:
> >
> >  final ResultScanner rs = htable.getScanner(scan);
> >  for (Result row = rs.next(); row != null; row = rs.next()) {
> > final Cell[] cells = row.rawCells();
> > if (cells != null) {
> > for (final Cell cell : cells) {
> > final long c = Bytes.toLong(
> > *cell.getQualifierArray()*,
> cell.getQualifierOffset(),
> > cell.getQualifierLength());
> > final long v = Bytes.toLong(cell.getValueArray(),
> > cell.getValueOffset());
> > points.put(c, v);
> > }
> > }
> > }
> >
> > The cell.getQualifierArray() method says that it's 'Max length is
> > Short.MAX_VALUE which is 32,767 bytes.'. Hence it can only hold around
> > 4,000 columnQualfiers.
> >
> > Is there an alternate API that I should be using or am I missing some
> > setting here? Note that in worst case I need to read all the
> > columnQualifiers in a row and I may or may not know a subset to fetch in
> > advance.
> >
> > Even if this is not possible in a single call, is there a way to cursor
> > through the columnQualifiers?
> >
> > I am presently using Hbase 0.96 client but can switch to Hbase 1.x if
> there
> > is an API in the newer version.
> >
> > --cheers, gaurav
> >
> > --
> > --cheers, gaurav
> >
>

????: HBase Filter Problem

[Fulin Sun]

Hi , there

How many rows are there in the hbase table ? You want to achive the default 
FilterList.Operator.MUST_PASS_ALL or 
you just want to use or conditions for these filters ? 

I think the reason is that this kind of filter list just go more scan work and 
lower performance. 

Best,
Sun.




CertusNet

 donhoff_h
?? 2015-09-23 16:33
 user
?? HBase Filter Problem
Hi??
 
I wrote a program which function is to extract some data from a HBase table. 
According to business requirements I had to use the PrefixFilter and the 
SingleColumnValueFilter to filter the data.  The program ran very fast and 
returned in 1 sec. 
 
Considering I just need the rowkey of each record in my final result, I tried 
to improve my program by using the PrefixFilter + SingleColumnValueFilter + 
FirstKeyOnlyFitler. To my surprise the program ran very slow this time. It run 
about 20min and still not finished. So I had to kill it.
 
Does anybody know the reason that cause my program run such slow?  Since I set 
the PrefixFilter as the first filter in the FilterList object, I think the 
program should ran fast.
 
Many Thanks!

?????? HBase Filter Problem

[donhoff_h]

Hi,

There are 90 Million records in the table. And I use the the MUST_PASS_ALL for 
all my filters.  When I use PrefixFilter + SingleColumnValueFilter, it returned 
fast. So I supposed that the combination of PrefixFilter + 
SingleColumnValueFilter + FirstKeyOnlyFilter should be fast. But the fact is 
just in contrast. Do you know the reason that cause it?

Thanks!



--  --
??: "Fulin Sun";;
: 2015??9??23??(??) 4:53
??: "HBase User"; 

: : HBase Filter Problem



Hi , there

How many rows are there in the hbase table ? You want to achive the default 
FilterList.Operator.MUST_PASS_ALL or 
you just want to use or conditions for these filters ? 

I think the reason is that this kind of filter list just go more scan work and 
lower performance. 

Best,
Sun.




CertusNet

 donhoff_h
?? 2015-09-23 16:33
 user
?? HBase Filter Problem
Hi??
 
I wrote a program which function is to extract some data from a HBase table. 
According to business requirements I had to use the PrefixFilter and the 
SingleColumnValueFilter to filter the data.  The program ran very fast and 
returned in 1 sec. 
 
Considering I just need the rowkey of each record in my final result, I tried 
to improve my program by using the PrefixFilter + SingleColumnValueFilter + 
FirstKeyOnlyFitler. To my surprise the program ran very slow this time. It run 
about 20min and still not finished. So I had to kill it.
 
Does anybody know the reason that cause my program run such slow?  Since I set 
the PrefixFilter as the first filter in the FilterList object, I think the 
program should ran fast.
 
Many Thanks!

HBase Filter Problem

[donhoff_h]

Hi??

I wrote a program which function is to extract some data from a HBase table. 
According to business requirements I had to use the PrefixFilter and the 
SingleColumnValueFilter to filter the data.  The program ran very fast and 
returned in 1 sec. 

Considering I just need the rowkey of each record in my final result, I tried 
to improve my program by using the PrefixFilter + SingleColumnValueFilter + 
FirstKeyOnlyFitler. To my surprise the program ran very slow this time. It run 
about 20min and still not finished. So I had to kill it.

Does anybody know the reason that cause my program run such slow?  Since I set 
the PrefixFilter as the first filter in the FilterList object, I think the 
program should ran fast.

Many Thanks!

Exporting a snapshot to external cluster

[Akmal Abbasov]

Hi all,
I would like to know the best practice when exporting a snapshot to remote 
hbase cluster with ha configuration.
My assumption is:
1. to know which of the HDFS namenode is active
2. export snapshot to active namenode

Since I need to do this programmatically what is the best way to know which 
namenode is active? 
Afaik, it should be done through zookeeper, but through which API it will be 
more convenient?

Thanks.

Re: Large number of column qualifiers

[Anoop John]

>>I have Column Family with very large number of column qualifiers (>
50,000). Each column qualifier is 8 bytes long.

When u say u have 5 qualifiers in a CF, means u will have those many
cells coming under that CF per row.  So am not getting what is the
qualifier length limit as such coming. Per qualifier, you will have a diff
cell and its qualifier.

-Anoop-


On Thu, Sep 24, 2015 at 1:13 AM, Vladimir Rodionov 
wrote:

> Yes, the comment is incorrect.
>
> hbase.client.keyvalue.maxsize controls max key-value size, but its
> unlimited in a master (I was wrong about 1MB, this is probably for older
> versions of HBase)
>
>
> -Vlad
>
> On Wed, Sep 23, 2015 at 11:45 AM, Gaurav Agarwal  wrote:
>
> > Thanks Vlad. Could you please point me the KV size setting (default 1MB)?
> > Just to make sure that I understand correct, are you suggesting that the
> > following comment is incorrect in Cell.java?
> >
> >  /**
> >* Contiguous raw bytes that may start at any index in the containing
> > array. Max length is
> >* Short.MAX_VALUE which is 32,767 bytes.
> >* @return The array containing the qualifier bytes.
> >*/
> >   byte[] getQualifierArray();
> >
> > On Thu, Sep 24, 2015 at 12:10 AM, Gaurav Agarwal 
> wrote:
> >
> > > Thanks Vlad. Could you please point me the KV size setting (default
> 1MB)?
> > > Just to make sure that I understand correct - the following comment is
> > > incorrect in Cell.java:
> > >
> > >  /**
> > >* Contiguous raw bytes that may start at any index in the containing
> > > array. Max length is
> > >* Short.MAX_VALUE which is 32,767 bytes.
> > >* @return The array containing the qualifier bytes.
> > >*/
> > >   byte[] getQualifierArray();
> > >
> > > On Wed, Sep 23, 2015 at 11:43 PM, Vladimir Rodionov <
> > > vladrodio...@gmail.com> wrote:
> > >
> > >> Check KeyValue class (Cell's implementation). getQualifierArray()
> > returns
> > >> kv's backing array. There is no SHORT limit on a size of this array,
> but
> > >> there are other limits in  HBase - maximum KV size, for example, which
> > is
> > >> configurable, but, by default, is 1MB. Having 50K qualifiers is a bad
> > >> idea.
> > >> Consider redesigning your data model and use rowkey instead.
> > >>
> > >> -Vlad
> > >>
> > >> On Wed, Sep 23, 2015 at 10:24 AM, Ted Yu  wrote:
> > >>
> > >> > Please take a look at HBASE-11544 which is in hbase 1.1
> > >> >
> > >> > Cheers
> > >> >
> > >> > On Wed, Sep 23, 2015 at 10:18 AM, Gaurav Agarwal 
> > >> wrote:
> > >> >
> > >> > > Hi All,
> > >> > >
> > >> > > I have Column Family with very large number of column qualifiers
> (>
> > >> > > 50,000). Each column qualifier is 8 bytes long. The problem is the
> > >> when I
> > >> > > do a scan operation to fetch some rows, the client side Cell
> object
> > >> does
> > >> > > not have enough space allocated in it to hold all the
> > columnQaulifiers
> > >> > for
> > >> > > a given row and hence I cannot read all the columns back for a
> given
> > >> row.
> > >> > >
> > >> > > Please see the code snippet that I am using:
> > >> > >
> > >> > >  final ResultScanner rs = htable.getScanner(scan);
> > >> > >  for (Result row = rs.next(); row != null; row = rs.next()) {
> > >> > > final Cell[] cells = row.rawCells();
> > >> > > if (cells != null) {
> > >> > > for (final Cell cell : cells) {
> > >> > > final long c = Bytes.toLong(
> > >> > > *cell.getQualifierArray()*,
> > >> > cell.getQualifierOffset(),
> > >> > > cell.getQualifierLength());
> > >> > > final long v = Bytes.toLong(cell.getValueArray(),
> > >> > > cell.getValueOffset());
> > >> > > points.put(c, v);
> > >> > > }
> > >> > > }
> > >> > > }
> > >> > >
> > >> > > The cell.getQualifierArray() method says that it's 'Max length is
> > >> > > Short.MAX_VALUE which is 32,767 bytes.'. Hence it can only hold
> > around
> > >> > > 4,000 columnQualfiers.
> > >> > >
> > >> > > Is there an alternate API that I should be using or am I missing
> > some
> > >> > > setting here? Note that in worst case I need to read all the
> > >> > > columnQualifiers in a row and I may or may not know a subset to
> > fetch
> > >> in
> > >> > > advance.
> > >> > >
> > >> > > Even if this is not possible in a single call, is there a way to
> > >> cursor
> > >> > > through the columnQualifiers?
> > >> > >
> > >> > > I am presently using Hbase 0.96 client but can switch to Hbase 1.x
> > if
> > >> > there
> > >> > > is an API in the newer version.
> > >> > >
> > >> > > --cheers, gaurav
> > >> > >
> > >> > > --
> > >> > > --cheers, gaurav
> > >> > >
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > --cheers, gaurav
> > >
> >
> >
> >
> > --
> > --cheers, gaurav
> >
>

Hbck and exceptions

[Sumit Nigam]

Hi,
I have a few questions/ observations with respect to some of the Hbase 
exceptions. 
I have used various hbck options with varied success in case of Hbase master 
failing to startup after a shutdown. However, it has been more of trial and 
error. Is there any recommendation document as to what options to use when 
getting such startup errors which are seemingly related to data corruption 
(such as say, TableExistsException, TableDoesNotExistException, TableEnabled, 
Disabled exceptions etc.)? In some of the forums that I have read, these kind 
of exceptions can happen due to a unclean shutdown of Hbase. If that is true, 
then kill -9 should be avoided in case of an OOME? Also, using zkcli to clear 
(rmr) entire /hbase could cause data loss because we would lose all znodes 
related to Hbase (including meta)? Or is Hbase able to rebuild that information 
without any worry of data loss?
Thanks,Sumit

Re: Thinking of HBase 1.1.3 [was HBASE-14317 and 1.1.3]

[Nick Dimiduk]

I've run the compatibility checking tool [0] between branch-1.1
(0bf97bac2ed564994a0bcda5f1993260bf0b448f) and 1.1.0
(e860c66d41ddc8231004b646098a58abca7fb523). There has been a little bit of
drift, but nothing that I think is release-blocking. However, I'd like to
bring it to your attention here, before it sinks an RC. You can compare
this to the run between 1.1.0 and 1.1.2RC2, which became 1.1.2 [1]. Notice
we've added a handful of methods, which is acceptable according to our
guidelines [2].The question I have is about adding throws IOException
to TableRecordReader.close(). IOException is in the interface declaration
of the super type, but this will require a source code change for anyone
consuming our type directly. I believe, according to [2], this breaks our
guidelines for a patch release.

I've also sent a note over to HBASE-14394 [3] regarding the added public
and undocumented method to TableRecordReader, so there's potentially two
addendum's required for this patch.

How would the community like to proceed?

[0]: http://people.apache.org/~ndimiduk/1.1.0_branch-1.1_compat_report.html
[1]: http://people.apache.org/~ndimiduk/1.1.0_1.1.2RC2_compat_report.html
[2]: http://hbase.apache.org/book.html#hbase.versioning
[3]:
https://issues.apache.org/jira/browse/HBASE-14394?focusedCommentId=14905429=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14905429

On Mon, Sep 21, 2015 at 3:18 PM, Nick Dimiduk  wrote:

> Hi folks,
>
> It's that time again, I'm looking at spinning 1.1.3 bit this week, with
> hopes that we can get a release out in early October. The only issue I'm
> actively tracking as a must for this release is HBASE-14374, the back port
> for HBASE-14317. Is there anything else you're planning to get in for this
> one that's not been committed yet? Please speak up. I'll be starting my
> pre-release validations tomorrow or Wednesday.
>
> Thanks,
> Nick
>
> On Fri, Sep 4, 2015 at 4:08 PM, Andrew Purtell 
> wrote:
>
>> > PMC: do you have bandwidth to test yet another round of RC's?
>>
>> Yes, absolutely, and if you'd also like help making the RCs mail me
>> privately.
>>
>> On Fri, Sep 4, 2015 at 8:11 AM, Nick Dimiduk  wrote:
>>
>> > Hi folks,
>> >
>> > I know we just got through voting periods on three patch releases, but
>> > HBASE-14317 is looking pretty bad by my eye. Given we have a fix on our
>> > end, I'm up for spinning 1.1.3 a couple weeks early. How does the
>> community
>> > feel about it? Users: do you need this patch immediately? PMC: do you
>> have
>> > bandwidth to test yet another round of RC's? I'm not on JIRA yet this
>> > morning; is there other nastiness we should get fixed in an accelerated
>> .3
>> > as well?
>> >
>> > Thanks for your thoughts and your time.
>> > -n
>> >
>>
>>
>>
>> --
>> Best regards,
>>
>>- Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>>
>
>

Re: Thinking of HBase 1.1.3 [was HBASE-14317 and 1.1.3]

[Enis Söztutar]

Agreed that we should not change the declared interface for TRR in patch
releases. Ugly, but we can rethrow as RuntimeException or ignore in 1.1 and
before.

I think this is also a blocker:
https://issues.apache.org/jira/browse/HBASE-14474

Enis

On Wed, Sep 23, 2015 at 3:50 PM, Nick Dimiduk  wrote:

> I've run the compatibility checking tool [0] between branch-1.1
> (0bf97bac2ed564994a0bcda5f1993260bf0b448f) and 1.1.0
> (e860c66d41ddc8231004b646098a58abca7fb523). There has been a little bit of
> drift, but nothing that I think is release-blocking. However, I'd like to
> bring it to your attention here, before it sinks an RC. You can compare
> this to the run between 1.1.0 and 1.1.2RC2, which became 1.1.2 [1]. Notice
> we've added a handful of methods, which is acceptable according to our
> guidelines [2].The question I have is about adding throws IOException
> to TableRecordReader.close(). IOException is in the interface declaration
> of the super type, but this will require a source code change for anyone
> consuming our type directly. I believe, according to [2], this breaks our
> guidelines for a patch release.
>
> I've also sent a note over to HBASE-14394 [3] regarding the added public
> and undocumented method to TableRecordReader, so there's potentially two
> addendum's required for this patch.
>
> How would the community like to proceed?
>
> [0]:
> http://people.apache.org/~ndimiduk/1.1.0_branch-1.1_compat_report.html
> [1]: http://people.apache.org/~ndimiduk/1.1.0_1.1.2RC2_compat_report.html
> [2]: http://hbase.apache.org/book.html#hbase.versioning
> [3]:
>
> https://issues.apache.org/jira/browse/HBASE-14394?focusedCommentId=14905429=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14905429
>
> On Mon, Sep 21, 2015 at 3:18 PM, Nick Dimiduk  wrote:
>
> > Hi folks,
> >
> > It's that time again, I'm looking at spinning 1.1.3 bit this week, with
> > hopes that we can get a release out in early October. The only issue I'm
> > actively tracking as a must for this release is HBASE-14374, the back
> port
> > for HBASE-14317. Is there anything else you're planning to get in for
> this
> > one that's not been committed yet? Please speak up. I'll be starting my
> > pre-release validations tomorrow or Wednesday.
> >
> > Thanks,
> > Nick
> >
> > On Fri, Sep 4, 2015 at 4:08 PM, Andrew Purtell 
> > wrote:
> >
> >> > PMC: do you have bandwidth to test yet another round of RC's?
> >>
> >> Yes, absolutely, and if you'd also like help making the RCs mail me
> >> privately.
> >>
> >> On Fri, Sep 4, 2015 at 8:11 AM, Nick Dimiduk 
> wrote:
> >>
> >> > Hi folks,
> >> >
> >> > I know we just got through voting periods on three patch releases, but
> >> > HBASE-14317 is looking pretty bad by my eye. Given we have a fix on
> our
> >> > end, I'm up for spinning 1.1.3 a couple weeks early. How does the
> >> community
> >> > feel about it? Users: do you need this patch immediately? PMC: do you
> >> have
> >> > bandwidth to test yet another round of RC's? I'm not on JIRA yet this
> >> > morning; is there other nastiness we should get fixed in an
> accelerated
> >> .3
> >> > as well?
> >> >
> >> > Thanks for your thoughts and your time.
> >> > -n
> >> >
> >>
> >>
> >>
> >> --
> >> Best regards,
> >>
> >>- Andy
> >>
> >> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> >> (via Tom White)
> >>
> >
> >
>

Re: Large number of column qualifiers

[ramkrishna vasudevan]

Am not sure whether you have tried it. the scan API has got an API called
'batching'. Did you try it?  So per row if there are more columns you can
still limit the amount of data being sent to the client. I think the main
issue you are facing is that the qualifiers getting returned are more in
number and so the client is not able to accept them?

'Short.MAX_VALUE which is 32,767 bytes.'
This comment applies for the qualifier length ie. the name that you specify
for the qualifier not on the number of qualifiers.

Regards
Ram

On Thu, Sep 24, 2015 at 8:52 AM, Anoop John  wrote:

> >>I have Column Family with very large number of column qualifiers (>
> 50,000). Each column qualifier is 8 bytes long.
>
> When u say u have 5 qualifiers in a CF, means u will have those many
> cells coming under that CF per row.  So am not getting what is the
> qualifier length limit as such coming. Per qualifier, you will have a diff
> cell and its qualifier.
>
> -Anoop-
>
>
> On Thu, Sep 24, 2015 at 1:13 AM, Vladimir Rodionov  >
> wrote:
>
> > Yes, the comment is incorrect.
> >
> > hbase.client.keyvalue.maxsize controls max key-value size, but its
> > unlimited in a master (I was wrong about 1MB, this is probably for older
> > versions of HBase)
> >
> >
> > -Vlad
> >
> > On Wed, Sep 23, 2015 at 11:45 AM, Gaurav Agarwal 
> wrote:
> >
> > > Thanks Vlad. Could you please point me the KV size setting (default
> 1MB)?
> > > Just to make sure that I understand correct, are you suggesting that
> the
> > > following comment is incorrect in Cell.java?
> > >
> > >  /**
> > >* Contiguous raw bytes that may start at any index in the containing
> > > array. Max length is
> > >* Short.MAX_VALUE which is 32,767 bytes.
> > >* @return The array containing the qualifier bytes.
> > >*/
> > >   byte[] getQualifierArray();
> > >
> > > On Thu, Sep 24, 2015 at 12:10 AM, Gaurav Agarwal 
> > wrote:
> > >
> > > > Thanks Vlad. Could you please point me the KV size setting (default
> > 1MB)?
> > > > Just to make sure that I understand correct - the following comment
> is
> > > > incorrect in Cell.java:
> > > >
> > > >  /**
> > > >* Contiguous raw bytes that may start at any index in the
> containing
> > > > array. Max length is
> > > >* Short.MAX_VALUE which is 32,767 bytes.
> > > >* @return The array containing the qualifier bytes.
> > > >*/
> > > >   byte[] getQualifierArray();
> > > >
> > > > On Wed, Sep 23, 2015 at 11:43 PM, Vladimir Rodionov <
> > > > vladrodio...@gmail.com> wrote:
> > > >
> > > >> Check KeyValue class (Cell's implementation). getQualifierArray()
> > > returns
> > > >> kv's backing array. There is no SHORT limit on a size of this array,
> > but
> > > >> there are other limits in  HBase - maximum KV size, for example,
> which
> > > is
> > > >> configurable, but, by default, is 1MB. Having 50K qualifiers is a
> bad
> > > >> idea.
> > > >> Consider redesigning your data model and use rowkey instead.
> > > >>
> > > >> -Vlad
> > > >>
> > > >> On Wed, Sep 23, 2015 at 10:24 AM, Ted Yu 
> wrote:
> > > >>
> > > >> > Please take a look at HBASE-11544 which is in hbase 1.1
> > > >> >
> > > >> > Cheers
> > > >> >
> > > >> > On Wed, Sep 23, 2015 at 10:18 AM, Gaurav Agarwal <
> gau...@arkin.net>
> > > >> wrote:
> > > >> >
> > > >> > > Hi All,
> > > >> > >
> > > >> > > I have Column Family with very large number of column qualifiers
> > (>
> > > >> > > 50,000). Each column qualifier is 8 bytes long. The problem is
> the
> > > >> when I
> > > >> > > do a scan operation to fetch some rows, the client side Cell
> > object
> > > >> does
> > > >> > > not have enough space allocated in it to hold all the
> > > columnQaulifiers
> > > >> > for
> > > >> > > a given row and hence I cannot read all the columns back for a
> > given
> > > >> row.
> > > >> > >
> > > >> > > Please see the code snippet that I am using:
> > > >> > >
> > > >> > >  final ResultScanner rs = htable.getScanner(scan);
> > > >> > >  for (Result row = rs.next(); row != null; row = rs.next()) {
> > > >> > > final Cell[] cells = row.rawCells();
> > > >> > > if (cells != null) {
> > > >> > > for (final Cell cell : cells) {
> > > >> > > final long c = Bytes.toLong(
> > > >> > > *cell.getQualifierArray()*,
> > > >> > cell.getQualifierOffset(),
> > > >> > > cell.getQualifierLength());
> > > >> > > final long v = Bytes.toLong(cell.getValueArray(),
> > > >> > > cell.getValueOffset());
> > > >> > > points.put(c, v);
> > > >> > > }
> > > >> > > }
> > > >> > > }
> > > >> > >
> > > >> > > The cell.getQualifierArray() method says that it's 'Max length
> is
> > > >> > > Short.MAX_VALUE which is 32,767 bytes.'. Hence it can only hold
> > > around
> > > >> > > 4,000 columnQualfiers.
> > > >> > >
> > > >> > > Is there an alternate API that I should be using or am I missing
> > > some