Re: Need hlep in substring filter
filterRowKey what is these arguments? Can you please explain lets say I
have data like abcde_12345 similary way
and I want to scan all the records ending with 12345 what should I pass
to this function?
On 16 April 2015 at 20:12, Ted Yu yuzhih...@gmail.com wrote:
Please take a look
at
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/SuffixFilter.java
in Phoenix repo.
Processing is done in the following method:
public boolean filterRowKey(byte[] buffer, int offset, int length) {
Note: the suffix is expressed in byte array.
FYI
On Thu, Apr 16, 2015 at 7:27 AM, Jeetendra Gangele gangele...@gmail.com
wrote:
Thanks for your reply.Ya I am filtering on row-key which ended with
12567
I did not understand properly what do you mean by You can compose
SuffixFilter which filters row key by comparing suffix.
can you elaborate .
On 16 April 2015 at 19:49, Ted Yu yuzhih...@gmail.com wrote:
bq. record key ending with 12567
Can you clarify whether you're filtering by row key ?
If so, please take a look at PrefixFilter.
You can compose SuffixFilter which filters row key by comparing suffix.
Cheers
On Thu, Apr 16, 2015 at 7:06 AM, Jeetendra Gangele
gangele...@gmail.com
wrote:
Hi All I have get all the records using scan with end with particular
string let
say record key ending with 12567. I need to scan and get all such
result.
I am values filter and do the substring there but i want to avoid
this
because its slow.
Regards
jeetendra
Re: Need hlep in substring filter
Here is an example
from
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
:
byte[] suffix =
ByteUtil.concat(QueryConstants.SEPARATOR_BYTE_ARRAY,
SchemaUtil.getTableNameAsBytes(schemaName, tableName));
SuffixFilter rowFilter = new SuffixFilter(suffix);
Filter filter = new FilterList(linkFilter, rowFilter);
You can use the following method from Bytes.java (in hbase-common) to
obtain byte[] from 12345
public static byte[] toBytes(String s) {
Cheers
On Fri, Apr 17, 2015 at 7:50 AM, Jeetendra Gangele gangele...@gmail.com
wrote:
filterRowKey what is these arguments? Can you please explain lets say I
have data like abcde_12345 similary way
and I want to scan all the records ending with 12345 what should I pass
to this function?
On 16 April 2015 at 20:12, Ted Yu yuzhih...@gmail.com wrote:
Please take a look
at
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/SuffixFilter.java
in Phoenix repo.
Processing is done in the following method:
public boolean filterRowKey(byte[] buffer, int offset, int length) {
Note: the suffix is expressed in byte array.
FYI
On Thu, Apr 16, 2015 at 7:27 AM, Jeetendra Gangele gangele...@gmail.com
wrote:
Thanks for your reply.Ya I am filtering on row-key which ended with
12567
I did not understand properly what do you mean by You can compose
SuffixFilter which filters row key by comparing suffix.
can you elaborate .
On 16 April 2015 at 19:49, Ted Yu yuzhih...@gmail.com wrote:
bq. record key ending with 12567
Can you clarify whether you're filtering by row key ?
If so, please take a look at PrefixFilter.
You can compose SuffixFilter which filters row key by comparing
suffix.
Cheers
On Thu, Apr 16, 2015 at 7:06 AM, Jeetendra Gangele
gangele...@gmail.com
wrote:
Hi All I have get all the records using scan with end with
particular
string let
say record key ending with 12567. I need to scan and get all such
result.
I am values filter and do the substring there but i want to avoid
this
because its slow.
Regards
jeetendra
Add keys to column family in HBase using Python
Dear HBase experts, I have a Hadoop cluster which has Hive, HBase installed along with other Hadoop components. I am currently exploring ways to automate a data migration process from Hive to HBase which involves new columns of data added ever so often. I was successful in creating a HBase table using Hive and load data into the HBase table, on these lines I tried to add new columns to the HBase table(from Hive) using the alter table syntax and I got the error message, ALTER TABLE cannot be used for a non-native table temp_testing. As an alternative to this I am also trying to do this programmatically using Python, I have explored the libraries HappyBasehttps://happybase.readthedocs.org/en/latest/index.html and starbasehttp://pythonhosted.org//starbase/. These libraries provide functionality for creating, deleting and other features but none of these provide an option to add a key to a column family. Does anybody know of a better way of achieving this with Python, say libraries or through other means. Thanks in advance, Manoj The information transmitted in this email is intended only for the person or entity to which it is addressed, and may contain material confidential to Xoom Corporation, and/or its subsidiary, buyindiaonline.com Inc. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient(s) is prohibited. If you received this email in error, please contact the sender and delete the material from your files.
Re: Need hlep in substring filter
I am using Hbase 0.98.4 so this feature already should be there?
problem is FilterList class(when we add this filter) not able to find the
SuffixFilter
On 17 April 2015 at 22:47, Jeetendra Gangele gangele...@gmail.com wrote:
Yes I used inside my project. This project will build the jar that jar I
added with classpath when running the job
On 17 April 2015 at 22:40, Ted Yu yuzhih...@gmail.com wrote:
bq. java.io.IOException: java.lang.ClassNotFoundException:
com.insideview.yam.HbaseClient.SuffixFilter
I assume you copied SuffixFilter class to your insideview project. How did
you deploy the jar containing SuffixFilter to hbase cluster ?
See HBASE-1936 for more details where hbase.dynamic.jars.dir was
introduced.
Cheers
On Fri, Apr 17, 2015 at 9:57 AM, Jeetendra Gangele gangele...@gmail.com
wrote:
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1439)
... 8 more
Caused by: org.apache.hadoop.hbase.exceptions.DeserializationException:
java.io.IOException: java.lang.ClassNotFoundException:
com.insideview.yam.HbaseClient.SuffixFilter
at
org.apache.hadoop.hbase.filter.FilterList.parseFrom(FilterList.java:406)
... 12 more
Caused by: java.io.IOException: java.lang.ClassNotFoundException:
com.insideview.yam.HbaseClient.SuffixFilter
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1441)
at
org.apache.hadoop.hbase.filter.FilterList.parseFrom(FilterList.java:403)
This is the error i m getting
On 17 April 2015 at 20:32, Ted Yu yuzhih...@gmail.com wrote:
Here is an example
from
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
:
byte[] suffix =
ByteUtil.concat(QueryConstants.SEPARATOR_BYTE_ARRAY,
SchemaUtil.getTableNameAsBytes(schemaName, tableName));
SuffixFilter rowFilter = new SuffixFilter(suffix);
Filter filter = new FilterList(linkFilter, rowFilter);
You can use the following method from Bytes.java (in hbase-common) to
obtain byte[] from 12345
public static byte[] toBytes(String s) {
Cheers
On Fri, Apr 17, 2015 at 7:50 AM, Jeetendra Gangele
gangele...@gmail.com
wrote:
filterRowKey what is these arguments? Can you please explain lets
say I
have data like abcde_12345 similary way
and I want to scan all the records ending with 12345 what should I
pass
to this function?
On 16 April 2015 at 20:12, Ted Yu yuzhih...@gmail.com wrote:
Please take a look
at
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/SuffixFilter.java
in Phoenix repo.
Processing is done in the following method:
public boolean filterRowKey(byte[] buffer, int offset, int
length)
{
Note: the suffix is expressed in byte array.
FYI
On Thu, Apr 16, 2015 at 7:27 AM, Jeetendra Gangele
gangele...@gmail.com
wrote:
Thanks for your reply.Ya I am filtering on row-key which ended
with
12567
I did not understand properly what do you mean by You can
compose
SuffixFilter which filters row key by comparing suffix.
can you elaborate .
On 16 April 2015 at 19:49, Ted Yu yuzhih...@gmail.com wrote:
bq. record key ending with 12567
Can you clarify whether you're filtering by row key ?
If so, please take a look at PrefixFilter.
You can compose SuffixFilter which filters row key by
comparing
suffix.
Cheers
On Thu, Apr 16, 2015 at 7:06 AM, Jeetendra Gangele
gangele...@gmail.com
wrote:
Hi All I have get all the records using scan with end with
particular
string let
say record key ending with 12567. I need to scan and get
all
such
result.
I am values filter and do the substring there but i want to
avoid
this
because its slow.
Regards
jeetendra
Re: Need hlep in substring filter
bq. build the jar
The jar needs to be accessible by each region server.
See HBASE-1936 for details.
On Fri, Apr 17, 2015 at 10:17 AM, Jeetendra Gangele gangele...@gmail.com
wrote:
Yes I used inside my project. This project will build the jar that jar I
added with classpath when running the job
On 17 April 2015 at 22:40, Ted Yu yuzhih...@gmail.com wrote:
bq. java.io.IOException: java.lang.ClassNotFoundException:
com.insideview.yam.HbaseClient.SuffixFilter
I assume you copied SuffixFilter class to your insideview project. How
did
you deploy the jar containing SuffixFilter to hbase cluster ?
See HBASE-1936 for more details where hbase.dynamic.jars.dir was
introduced.
Cheers
On Fri, Apr 17, 2015 at 9:57 AM, Jeetendra Gangele gangele...@gmail.com
wrote:
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1439)
... 8 more
Caused by: org.apache.hadoop.hbase.exceptions.DeserializationException:
java.io.IOException: java.lang.ClassNotFoundException:
com.insideview.yam.HbaseClient.SuffixFilter
at
org.apache.hadoop.hbase.filter.FilterList.parseFrom(FilterList.java:406)
... 12 more
Caused by: java.io.IOException: java.lang.ClassNotFoundException:
com.insideview.yam.HbaseClient.SuffixFilter
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1441)
at
org.apache.hadoop.hbase.filter.FilterList.parseFrom(FilterList.java:403)
This is the error i m getting
On 17 April 2015 at 20:32, Ted Yu yuzhih...@gmail.com wrote:
Here is an example
from
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
:
byte[] suffix =
ByteUtil.concat(QueryConstants.SEPARATOR_BYTE_ARRAY,
SchemaUtil.getTableNameAsBytes(schemaName, tableName));
SuffixFilter rowFilter = new SuffixFilter(suffix);
Filter filter = new FilterList(linkFilter, rowFilter);
You can use the following method from Bytes.java (in hbase-common) to
obtain byte[] from 12345
public static byte[] toBytes(String s) {
Cheers
On Fri, Apr 17, 2015 at 7:50 AM, Jeetendra Gangele
gangele...@gmail.com
wrote:
filterRowKey what is these arguments? Can you please explain lets
say I
have data like abcde_12345 similary way
and I want to scan all the records ending with 12345 what should
I
pass
to this function?
On 16 April 2015 at 20:12, Ted Yu yuzhih...@gmail.com wrote:
Please take a look
at
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/SuffixFilter.java
in Phoenix repo.
Processing is done in the following method:
public boolean filterRowKey(byte[] buffer, int offset, int
length)
{
Note: the suffix is expressed in byte array.
FYI
On Thu, Apr 16, 2015 at 7:27 AM, Jeetendra Gangele
gangele...@gmail.com
wrote:
Thanks for your reply.Ya I am filtering on row-key which ended
with
12567
I did not understand properly what do you mean by You can
compose
SuffixFilter which filters row key by comparing suffix.
can you elaborate .
On 16 April 2015 at 19:49, Ted Yu yuzhih...@gmail.com wrote:
bq. record key ending with 12567
Can you clarify whether you're filtering by row key ?
If so, please take a look at PrefixFilter.
You can compose SuffixFilter which filters row key by
comparing
suffix.
Cheers
On Thu, Apr 16, 2015 at 7:06 AM, Jeetendra Gangele
gangele...@gmail.com
wrote:
Hi All I have get all the records using scan with end with
particular
string let
say record key ending with 12567. I need to scan and get
all
such
result.
I am values filter and do the substring there but i want to
avoid
this
because its slow.
Regards
jeetendra
Re: Need hlep in substring filter
bq. java.io.IOException: java.lang.ClassNotFoundException:
com.insideview.yam.HbaseClient.SuffixFilter
I assume you copied SuffixFilter class to your insideview project. How did
you deploy the jar containing SuffixFilter to hbase cluster ?
See HBASE-1936 for more details where hbase.dynamic.jars.dir was introduced.
Cheers
On Fri, Apr 17, 2015 at 9:57 AM, Jeetendra Gangele gangele...@gmail.com
wrote:
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1439)
... 8 more
Caused by: org.apache.hadoop.hbase.exceptions.DeserializationException:
java.io.IOException: java.lang.ClassNotFoundException:
com.insideview.yam.HbaseClient.SuffixFilter
at
org.apache.hadoop.hbase.filter.FilterList.parseFrom(FilterList.java:406)
... 12 more
Caused by: java.io.IOException: java.lang.ClassNotFoundException:
com.insideview.yam.HbaseClient.SuffixFilter
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1441)
at
org.apache.hadoop.hbase.filter.FilterList.parseFrom(FilterList.java:403)
This is the error i m getting
On 17 April 2015 at 20:32, Ted Yu yuzhih...@gmail.com wrote:
Here is an example
from
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
:
byte[] suffix =
ByteUtil.concat(QueryConstants.SEPARATOR_BYTE_ARRAY,
SchemaUtil.getTableNameAsBytes(schemaName, tableName));
SuffixFilter rowFilter = new SuffixFilter(suffix);
Filter filter = new FilterList(linkFilter, rowFilter);
You can use the following method from Bytes.java (in hbase-common) to
obtain byte[] from 12345
public static byte[] toBytes(String s) {
Cheers
On Fri, Apr 17, 2015 at 7:50 AM, Jeetendra Gangele gangele...@gmail.com
wrote:
filterRowKey what is these arguments? Can you please explain lets say I
have data like abcde_12345 similary way
and I want to scan all the records ending with 12345 what should I
pass
to this function?
On 16 April 2015 at 20:12, Ted Yu yuzhih...@gmail.com wrote:
Please take a look
at
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/SuffixFilter.java
in Phoenix repo.
Processing is done in the following method:
public boolean filterRowKey(byte[] buffer, int offset, int length)
{
Note: the suffix is expressed in byte array.
FYI
On Thu, Apr 16, 2015 at 7:27 AM, Jeetendra Gangele
gangele...@gmail.com
wrote:
Thanks for your reply.Ya I am filtering on row-key which ended with
12567
I did not understand properly what do you mean by You can compose
SuffixFilter which filters row key by comparing suffix.
can you elaborate .
On 16 April 2015 at 19:49, Ted Yu yuzhih...@gmail.com wrote:
bq. record key ending with 12567
Can you clarify whether you're filtering by row key ?
If so, please take a look at PrefixFilter.
You can compose SuffixFilter which filters row key by comparing
suffix.
Cheers
On Thu, Apr 16, 2015 at 7:06 AM, Jeetendra Gangele
gangele...@gmail.com
wrote:
Hi All I have get all the records using scan with end with
particular
string let
say record key ending with 12567. I need to scan and get all
such
result.
I am values filter and do the substring there but i want to
avoid
this
because its slow.
Regards
jeetendra
Re: Need hlep in substring filter
Yes I used inside my project. This project will build the jar that jar I
added with classpath when running the job
On 17 April 2015 at 22:40, Ted Yu yuzhih...@gmail.com wrote:
bq. java.io.IOException: java.lang.ClassNotFoundException:
com.insideview.yam.HbaseClient.SuffixFilter
I assume you copied SuffixFilter class to your insideview project. How did
you deploy the jar containing SuffixFilter to hbase cluster ?
See HBASE-1936 for more details where hbase.dynamic.jars.dir was
introduced.
Cheers
On Fri, Apr 17, 2015 at 9:57 AM, Jeetendra Gangele gangele...@gmail.com
wrote:
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1439)
... 8 more
Caused by: org.apache.hadoop.hbase.exceptions.DeserializationException:
java.io.IOException: java.lang.ClassNotFoundException:
com.insideview.yam.HbaseClient.SuffixFilter
at
org.apache.hadoop.hbase.filter.FilterList.parseFrom(FilterList.java:406)
... 12 more
Caused by: java.io.IOException: java.lang.ClassNotFoundException:
com.insideview.yam.HbaseClient.SuffixFilter
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1441)
at
org.apache.hadoop.hbase.filter.FilterList.parseFrom(FilterList.java:403)
This is the error i m getting
On 17 April 2015 at 20:32, Ted Yu yuzhih...@gmail.com wrote:
Here is an example
from
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
:
byte[] suffix =
ByteUtil.concat(QueryConstants.SEPARATOR_BYTE_ARRAY,
SchemaUtil.getTableNameAsBytes(schemaName, tableName));
SuffixFilter rowFilter = new SuffixFilter(suffix);
Filter filter = new FilterList(linkFilter, rowFilter);
You can use the following method from Bytes.java (in hbase-common) to
obtain byte[] from 12345
public static byte[] toBytes(String s) {
Cheers
On Fri, Apr 17, 2015 at 7:50 AM, Jeetendra Gangele
gangele...@gmail.com
wrote:
filterRowKey what is these arguments? Can you please explain lets
say I
have data like abcde_12345 similary way
and I want to scan all the records ending with 12345 what should I
pass
to this function?
On 16 April 2015 at 20:12, Ted Yu yuzhih...@gmail.com wrote:
Please take a look
at
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/SuffixFilter.java
in Phoenix repo.
Processing is done in the following method:
public boolean filterRowKey(byte[] buffer, int offset, int
length)
{
Note: the suffix is expressed in byte array.
FYI
On Thu, Apr 16, 2015 at 7:27 AM, Jeetendra Gangele
gangele...@gmail.com
wrote:
Thanks for your reply.Ya I am filtering on row-key which ended
with
12567
I did not understand properly what do you mean by You can
compose
SuffixFilter which filters row key by comparing suffix.
can you elaborate .
On 16 April 2015 at 19:49, Ted Yu yuzhih...@gmail.com wrote:
bq. record key ending with 12567
Can you clarify whether you're filtering by row key ?
If so, please take a look at PrefixFilter.
You can compose SuffixFilter which filters row key by comparing
suffix.
Cheers
On Thu, Apr 16, 2015 at 7:06 AM, Jeetendra Gangele
gangele...@gmail.com
wrote:
Hi All I have get all the records using scan with end with
particular
string let
say record key ending with 12567. I need to scan and get
all
such
result.
I am values filter and do the substring there but i want to
avoid
this
because its slow.
Regards
jeetendra
Re: Rowkey design question
Sorry, but … We are in violent agreement. If done wrong it can and will kill you. Murphy’s law. If there’s more than one way to do something … the wrong way will be chosen, so where does that leave you? And then what hasn’t been said is the security concern which is odd because with XASecure now Ranger (until they try a new and different name), you need to use coprocessors on a trigger to see if you have permission to write to, or read data in a table. If you’re new to HBase… don’t use a coprocessor. That’s just asking for trouble. (And I know everyone here knows that to be the truth.) On Apr 12, 2015, at 1:45 AM, lars hofhansl la...@apache.org wrote: After the fun interlude (sorry about that) let me get back to the issue. There a multiple consideration: 1. row vs column. If in doubt err on the side of more rows. Only use many columns in a row when you need transaction over the data in the columns. 2. Value sizes. HBase is good at dealing with many small things. 1-5mb values here and there are OK, but most rows should be a few dozen KBs. Otherwise you'll see too much write amplification. 3. Column families. Place columns you typically access together in the same column family, and try to keep columns you don't access together mostly in different families. HBase can than efficiently rule out a large body of data to scan, by avoiding scanning families that are not needed. 4. Coprocessors and filters let you transform/filter things where the data is. The benefit can be huge. With coprocessors you can trap scan requests (next() calls) and inject your own logic. Thats what Phoenix does for example, and it's pretty efficient if done right (if you do it wrong you can kill your region server). On #2. You might want to invent a scheme where you store smaller values by value (i.e. in HBase) and larger ones by reference. I would put the column with the large value in its own family so that you could scan the rest of the metadata without requiring HBase to read the large value. You can follow a simple protocol: A. If the value is small (pick some notion of small between 1 and 10mb), store it in HBase, in a separate familY. B. Otherwise: 1. Write a row with the intended location of the file holding the value in HDFS. 2. Write the value into the HDFS file. Make sure the file location has a random element to avoid races. 3. Update the row created in #1 with a commit column (just a column you set to true), this is like a commit. (only when a writer reaches this point should the value be considered written) Note the everything is idempotent. The worst that can happen is that the process fails between #2 and #3. Now you have orphaned data in HDFS. Since the HDFS location has a random element in it, you can just retry. You can either leave orphaned data (since the commit bit is not set, it's not visible to a client), or you periodically look for those and clean them up. Hope this helps. Please let us know how it goes. -- Lars From: Kristoffer Sjögren sto...@gmail.com To: user@hbase.apache.org Sent: Wednesday, April 8, 2015 6:41 AM Subject: Re: Rowkey design question Yes, I think you're right. Adding one or more dimensions to the rowkey would indeed make the table narrower. And I guess it also make sense to store actual values (bigger qualifiers) outside HBase. Keeping them in Hadoop why not? Pulling hot ones out on SSD caches would be an interesting solution. And quite a bit simpler. Good call and thanks for the tip! :-) On Wed, Apr 8, 2015 at 1:45 PM, Michael Segel michael_se...@hotmail.com wrote: Ok… First, I’d suggest you rethink your schema by adding an additional dimension. You’ll end up with more rows, but a narrower table. In terms of compaction… if the data is relatively static, you won’t have compactions because nothing changed. But if your data is that static… why not put the data in sequence files and use HBase as the index. Could be faster. HTH -Mike On Apr 8, 2015, at 3:26 AM, Kristoffer Sjögren sto...@gmail.com wrote: I just read through HBase MOB design document and one thing that caught my attention was the following statement. When HBase deals with large numbers of values 100kb and up to ~10MB of data, it encounters performance degradations due to write amplification caused by splits and compactions. Is there any chance to run into this problem in the read path for data that is written infrequently and never changed? On Wed, Apr 8, 2015 at 9:30 AM, Kristoffer Sjögren sto...@gmail.com wrote: A small set of qualifiers will be accessed frequently so keeping them in block cache would be very beneficial. Some very seldom. So this sounds very promising! The reason why i'm considering a coprocessor is that I need to provide very specific information in the query request. Same thing with the response.
Re: Need hlep in substring filter
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1439)
... 8 more
Caused by: org.apache.hadoop.hbase.exceptions.DeserializationException:
java.io.IOException: java.lang.ClassNotFoundException:
com.insideview.yam.HbaseClient.SuffixFilter
at
org.apache.hadoop.hbase.filter.FilterList.parseFrom(FilterList.java:406)
... 12 more
Caused by: java.io.IOException: java.lang.ClassNotFoundException:
com.insideview.yam.HbaseClient.SuffixFilter
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1441)
at
org.apache.hadoop.hbase.filter.FilterList.parseFrom(FilterList.java:403)
This is the error i m getting
On 17 April 2015 at 20:32, Ted Yu yuzhih...@gmail.com wrote:
Here is an example
from
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
:
byte[] suffix =
ByteUtil.concat(QueryConstants.SEPARATOR_BYTE_ARRAY,
SchemaUtil.getTableNameAsBytes(schemaName, tableName));
SuffixFilter rowFilter = new SuffixFilter(suffix);
Filter filter = new FilterList(linkFilter, rowFilter);
You can use the following method from Bytes.java (in hbase-common) to
obtain byte[] from 12345
public static byte[] toBytes(String s) {
Cheers
On Fri, Apr 17, 2015 at 7:50 AM, Jeetendra Gangele gangele...@gmail.com
wrote:
filterRowKey what is these arguments? Can you please explain lets say I
have data like abcde_12345 similary way
and I want to scan all the records ending with 12345 what should I pass
to this function?
On 16 April 2015 at 20:12, Ted Yu yuzhih...@gmail.com wrote:
Please take a look
at
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/SuffixFilter.java
in Phoenix repo.
Processing is done in the following method:
public boolean filterRowKey(byte[] buffer, int offset, int length) {
Note: the suffix is expressed in byte array.
FYI
On Thu, Apr 16, 2015 at 7:27 AM, Jeetendra Gangele
gangele...@gmail.com
wrote:
Thanks for your reply.Ya I am filtering on row-key which ended with
12567
I did not understand properly what do you mean by You can compose
SuffixFilter which filters row key by comparing suffix.
can you elaborate .
On 16 April 2015 at 19:49, Ted Yu yuzhih...@gmail.com wrote:
bq. record key ending with 12567
Can you clarify whether you're filtering by row key ?
If so, please take a look at PrefixFilter.
You can compose SuffixFilter which filters row key by comparing
suffix.
Cheers
On Thu, Apr 16, 2015 at 7:06 AM, Jeetendra Gangele
gangele...@gmail.com
wrote:
Hi All I have get all the records using scan with end with
particular
string let
say record key ending with 12567. I need to scan and get all
such
result.
I am values filter and do the substring there but i want to avoid
this
because its slow.
Regards
jeetendra
Re: Need hlep in substring filter
This will @ hbase-site.xml?
can I pur my jar containing SuffixFilter class into hbase.rootdir?
On 18 April 2015 at 02:19, Ted Yu yuzhih...@gmail.com wrote:
bq. Hbase 0.98.4 so this feature already should be there?
Yes.
Drop the class containing SuffixFilter class in location specified by
hbase.dynamic.jars.dir
config. Then region server should be able to find it.
On Fri, Apr 17, 2015 at 10:22 AM, Jeetendra Gangele gangele...@gmail.com
wrote:
I am using Hbase 0.98.4 so this feature already should be there?
problem is FilterList class(when we add this filter) not able to find the
SuffixFilter
On 17 April 2015 at 22:47, Jeetendra Gangele gangele...@gmail.com
wrote:
Yes I used inside my project. This project will build the jar that jar
I
added with classpath when running the job
On 17 April 2015 at 22:40, Ted Yu yuzhih...@gmail.com wrote:
bq. java.io.IOException: java.lang.ClassNotFoundException:
com.insideview.yam.HbaseClient.SuffixFilter
I assume you copied SuffixFilter class to your insideview project. How
did
you deploy the jar containing SuffixFilter to hbase cluster ?
See HBASE-1936 for more details where hbase.dynamic.jars.dir was
introduced.
Cheers
On Fri, Apr 17, 2015 at 9:57 AM, Jeetendra Gangele
gangele...@gmail.com
wrote:
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown
Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1439)
... 8 more
Caused by:
org.apache.hadoop.hbase.exceptions.DeserializationException:
java.io.IOException: java.lang.ClassNotFoundException:
com.insideview.yam.HbaseClient.SuffixFilter
at
org.apache.hadoop.hbase.filter.FilterList.parseFrom(FilterList.java:406)
... 12 more
Caused by: java.io.IOException: java.lang.ClassNotFoundException:
com.insideview.yam.HbaseClient.SuffixFilter
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1441)
at
org.apache.hadoop.hbase.filter.FilterList.parseFrom(FilterList.java:403)
This is the error i m getting
On 17 April 2015 at 20:32, Ted Yu yuzhih...@gmail.com wrote:
Here is an example
from
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
:
byte[] suffix =
ByteUtil.concat(QueryConstants.SEPARATOR_BYTE_ARRAY,
SchemaUtil.getTableNameAsBytes(schemaName, tableName));
SuffixFilter rowFilter = new SuffixFilter(suffix);
Filter filter = new FilterList(linkFilter, rowFilter);
You can use the following method from Bytes.java (in hbase-common)
to
obtain byte[] from 12345
public static byte[] toBytes(String s) {
Cheers
On Fri, Apr 17, 2015 at 7:50 AM, Jeetendra Gangele
gangele...@gmail.com
wrote:
filterRowKey what is these arguments? Can you please explain
lets
say I
have data like abcde_12345 similary way
and I want to scan all the records ending with 12345 what
should I
pass
to this function?
On 16 April 2015 at 20:12, Ted Yu yuzhih...@gmail.com wrote:
Please take a look
at
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/SuffixFilter.java
in Phoenix repo.
Processing is done in the following method:
public boolean filterRowKey(byte[] buffer, int offset, int
length)
{
Note: the suffix is expressed in byte array.
FYI
On Thu, Apr 16, 2015 at 7:27 AM, Jeetendra Gangele
gangele...@gmail.com
wrote:
Thanks for your reply.Ya I am filtering on row-key which
ended
with
12567
I did not understand properly what do you mean by You can
compose
SuffixFilter which filters row key by comparing suffix.
can you elaborate .
On 16 April 2015 at 19:49, Ted Yu yuzhih...@gmail.com
wrote:
bq. record key ending with 12567
Can you clarify whether you're filtering by row key ?
If so, please take a look at PrefixFilter.
You can compose SuffixFilter which filters row key by
comparing
suffix.
Cheers
On Thu, Apr 16, 2015 at 7:06 AM, Jeetendra Gangele
gangele...@gmail.com
wrote:
Hi All I have get all the records using scan with end
with
particular
string
Re: Need hlep in substring filter
bq. Hbase 0.98.4 so this feature already should be there?
Yes.
Drop the class containing SuffixFilter class in location specified by
hbase.dynamic.jars.dir
config. Then region server should be able to find it.
On Fri, Apr 17, 2015 at 10:22 AM, Jeetendra Gangele gangele...@gmail.com
wrote:
I am using Hbase 0.98.4 so this feature already should be there?
problem is FilterList class(when we add this filter) not able to find the
SuffixFilter
On 17 April 2015 at 22:47, Jeetendra Gangele gangele...@gmail.com wrote:
Yes I used inside my project. This project will build the jar that jar I
added with classpath when running the job
On 17 April 2015 at 22:40, Ted Yu yuzhih...@gmail.com wrote:
bq. java.io.IOException: java.lang.ClassNotFoundException:
com.insideview.yam.HbaseClient.SuffixFilter
I assume you copied SuffixFilter class to your insideview project. How
did
you deploy the jar containing SuffixFilter to hbase cluster ?
See HBASE-1936 for more details where hbase.dynamic.jars.dir was
introduced.
Cheers
On Fri, Apr 17, 2015 at 9:57 AM, Jeetendra Gangele
gangele...@gmail.com
wrote:
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown
Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1439)
... 8 more
Caused by:
org.apache.hadoop.hbase.exceptions.DeserializationException:
java.io.IOException: java.lang.ClassNotFoundException:
com.insideview.yam.HbaseClient.SuffixFilter
at
org.apache.hadoop.hbase.filter.FilterList.parseFrom(FilterList.java:406)
... 12 more
Caused by: java.io.IOException: java.lang.ClassNotFoundException:
com.insideview.yam.HbaseClient.SuffixFilter
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1441)
at
org.apache.hadoop.hbase.filter.FilterList.parseFrom(FilterList.java:403)
This is the error i m getting
On 17 April 2015 at 20:32, Ted Yu yuzhih...@gmail.com wrote:
Here is an example
from
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
:
byte[] suffix =
ByteUtil.concat(QueryConstants.SEPARATOR_BYTE_ARRAY,
SchemaUtil.getTableNameAsBytes(schemaName, tableName));
SuffixFilter rowFilter = new SuffixFilter(suffix);
Filter filter = new FilterList(linkFilter, rowFilter);
You can use the following method from Bytes.java (in hbase-common)
to
obtain byte[] from 12345
public static byte[] toBytes(String s) {
Cheers
On Fri, Apr 17, 2015 at 7:50 AM, Jeetendra Gangele
gangele...@gmail.com
wrote:
filterRowKey what is these arguments? Can you please explain lets
say I
have data like abcde_12345 similary way
and I want to scan all the records ending with 12345 what
should I
pass
to this function?
On 16 April 2015 at 20:12, Ted Yu yuzhih...@gmail.com wrote:
Please take a look
at
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/SuffixFilter.java
in Phoenix repo.
Processing is done in the following method:
public boolean filterRowKey(byte[] buffer, int offset, int
length)
{
Note: the suffix is expressed in byte array.
FYI
On Thu, Apr 16, 2015 at 7:27 AM, Jeetendra Gangele
gangele...@gmail.com
wrote:
Thanks for your reply.Ya I am filtering on row-key which ended
with
12567
I did not understand properly what do you mean by You can
compose
SuffixFilter which filters row key by comparing suffix.
can you elaborate .
On 16 April 2015 at 19:49, Ted Yu yuzhih...@gmail.com
wrote:
bq. record key ending with 12567
Can you clarify whether you're filtering by row key ?
If so, please take a look at PrefixFilter.
You can compose SuffixFilter which filters row key by
comparing
suffix.
Cheers
On Thu, Apr 16, 2015 at 7:06 AM, Jeetendra Gangele
gangele...@gmail.com
wrote:
Hi All I have get all the records using scan with end with
particular
string let
say record key ending with 12567. I need to scan and get
all
such
result.
I am values filter and do the substring there but i want
to
avoid
this
because its slow.
Regards
jeetendra
Re: Need hlep in substring filter
bq. This will @ hbase-site.xml?
Yes.
bq. can I pur my jar containing SuffixFilter class into hbase.rootdir?
Better create a sub-folder under the rootdir.
On Fri, Apr 17, 2015 at 1:58 PM, Jeetendra Gangele gangele...@gmail.com
wrote:
This will @ hbase-site.xml?
can I pur my jar containing SuffixFilter class into hbase.rootdir?
On 18 April 2015 at 02:19, Ted Yu yuzhih...@gmail.com wrote:
bq. Hbase 0.98.4 so this feature already should be there?
Yes.
Drop the class containing SuffixFilter class in location specified by
hbase.dynamic.jars.dir
config. Then region server should be able to find it.
On Fri, Apr 17, 2015 at 10:22 AM, Jeetendra Gangele
gangele...@gmail.com
wrote:
I am using Hbase 0.98.4 so this feature already should be there?
problem is FilterList class(when we add this filter) not able to find
the
SuffixFilter
On 17 April 2015 at 22:47, Jeetendra Gangele gangele...@gmail.com
wrote:
Yes I used inside my project. This project will build the jar that
jar
I
added with classpath when running the job
On 17 April 2015 at 22:40, Ted Yu yuzhih...@gmail.com wrote:
bq. java.io.IOException: java.lang.ClassNotFoundException:
com.insideview.yam.HbaseClient.SuffixFilter
I assume you copied SuffixFilter class to your insideview project.
How
did
you deploy the jar containing SuffixFilter to hbase cluster ?
See HBASE-1936 for more details where hbase.dynamic.jars.dir was
introduced.
Cheers
On Fri, Apr 17, 2015 at 9:57 AM, Jeetendra Gangele
gangele...@gmail.com
wrote:
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor68.invoke(Unknown
Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1439)
... 8 more
Caused by:
org.apache.hadoop.hbase.exceptions.DeserializationException:
java.io.IOException: java.lang.ClassNotFoundException:
com.insideview.yam.HbaseClient.SuffixFilter
at
org.apache.hadoop.hbase.filter.FilterList.parseFrom(FilterList.java:406)
... 12 more
Caused by: java.io.IOException: java.lang.ClassNotFoundException:
com.insideview.yam.HbaseClient.SuffixFilter
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1441)
at
org.apache.hadoop.hbase.filter.FilterList.parseFrom(FilterList.java:403)
This is the error i m getting
On 17 April 2015 at 20:32, Ted Yu yuzhih...@gmail.com wrote:
Here is an example
from
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/MetaDataEndpointImpl.java
:
byte[] suffix =
ByteUtil.concat(QueryConstants.SEPARATOR_BYTE_ARRAY,
SchemaUtil.getTableNameAsBytes(schemaName, tableName));
SuffixFilter rowFilter = new SuffixFilter(suffix);
Filter filter = new FilterList(linkFilter, rowFilter);
You can use the following method from Bytes.java (in
hbase-common)
to
obtain byte[] from 12345
public static byte[] toBytes(String s) {
Cheers
On Fri, Apr 17, 2015 at 7:50 AM, Jeetendra Gangele
gangele...@gmail.com
wrote:
filterRowKey what is these arguments? Can you please explain
lets
say I
have data like abcde_12345 similary way
and I want to scan all the records ending with 12345 what
should I
pass
to this function?
On 16 April 2015 at 20:12, Ted Yu yuzhih...@gmail.com
wrote:
Please take a look
at
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/SuffixFilter.java
in Phoenix repo.
Processing is done in the following method:
public boolean filterRowKey(byte[] buffer, int offset, int
length)
{
Note: the suffix is expressed in byte array.
FYI
On Thu, Apr 16, 2015 at 7:27 AM, Jeetendra Gangele
gangele...@gmail.com
wrote:
Thanks for your reply.Ya I am filtering on row-key which
ended
with
12567
I did not understand properly what do you mean by You can
compose
SuffixFilter which filters row key by comparing suffix.
can you elaborate .
On 16 April 2015 at 19:49, Ted Yu yuzhih...@gmail.com
wrote:
bq. record key ending with 12567
Can you clarify whether you're filtering
Re: hbase.apache.org homepage looks weird on Chrome and Firefox
It is also possible because of HTTP resources are not loaded. I run into this daily, because of an extension I am using which refuses to load unsafe scripts from HTTPS links which I default to. Usually, web sites also host the CSS and javascript files referred instead of hot linking them. Maybe we should do the same. Enis On Thu, Apr 16, 2015 at 12:13 PM, anil gupta anilgupt...@gmail.com wrote: In chrome, i did Clear Browsing Data and then revisited http://hbase.apache.org/;. It came up properly. Thanks for the pointer, Nick. On Thu, Apr 16, 2015 at 11:05 AM, Andrew Purtell apurt...@apache.org wrote: Looks fine for me, Chrome and Firefox tested. As Nick says Looks like the CSS asset didn't load at Anil's location for whatever reason. On Thu, Apr 16, 2015 at 8:36 AM, Stack st...@duboce.net wrote: Are others running into the issue Anil sees? Thanks, St.Ack On Thu, Apr 16, 2015 at 8:13 AM, anil gupta anilgupt...@gmail.com wrote: Chrome: Version 42.0.2311.90 (64-bit) on Mac But, firefox(34.0.5) also displays the page in same way. On Thu, Apr 16, 2015 at 12:58 AM, Ted Yu yuzhih...@gmail.com wrote: Which Chrome version do you use ? I use 41.0.2272.104 (64-bit) (on Mac) and the page renders fine. Cheers On Wed, Apr 15, 2015 at 11:27 PM, anil gupta anilgupt...@gmail.com wrote: Hi, I am aware that recently there were some updates done on HBase website. For last few months, more often than not, the homepage is displayed in weird way in chrome and firefox. Is there a bug on homepage that is leading to this view: https://www.dropbox.com/s/jcpfnu4jwim28zg/Screen%20Shot%202015-04-15%20at%2011.18.46%20PM.png?dl=0 https://www.dropbox.com/s/o7xminppnzll6x7/Screen%20Shot%202015-04-15%20at%2011.19.55%20PM.png?dl=0 IMO, if the homepage looks broken then its hard to proceed ahead to read the docs. My two cents. Also, it would be nice if we could move docs of startgate from here: https://wiki.apache.org/hadoop/Hbase/Stargate to hbase.apache.org. -- Thanks Regards, Anil Gupta -- Thanks Regards, Anil Gupta -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White) -- Thanks Regards, Anil Gupta
