Classification: UNCLASSIFIED
Caveats: NONE
I checked the server logs (smacking myself on the forehead for not doing
so before) and I see this exception:
ERROR org.apache.hadoop.hbase.regionserver.HRegionServer:
org.apache.hadoop.hbase.UnknownScannerException: Name: -1
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja
va:1889)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav
a:57)
at
sun.reflec.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI
mpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915
)
And that jibes with the idea that the server needs to be aware of my
custom filter. I searched the mailing list archive for more information
on this and the only comment I could find was that the jar needs to be
in the "CLASSPATH on the RegionServer". I took this to mean that I
should create a jar with my filter in it, and put it in a folder on the
same computer that is running the Hbase RegionServer and make sure that
folder is in a CLASSPATH environment variable.
I'm still a novice java programmer (C# is my area), so I did the best I
could. I moved my SubstringDuplicate class to its own jar and put it in
a folder (since I'm running in pseudo-distributed mode while I get all
this working, this is all on one computer). I created a CLASSPATH
environment variable and added the path the jar is in to it. Restarted
everything, and tested it.
I'm still getting the same crash.
I tried to follow up on the comment to "You also may need to add it to
HbaseObjectWritable.java so it can be sent across the RPC layer." a bit.
I looked at the HBaseObjectWritable.java file and it looks like the
WritableByteArrayComparable class is already in that file as it should
be. Since that's my super class, I would think my filter would be
covered, wouldn't it? I believe that's what was meant when it was said
that 0.20.4 would fix that problem. Since it went from an interface to a
base abstract super class for writable purposes.
So I'm still stuck.
Does the CLASSPATH need to be set in etc instead of my login
.bash_profile? I have the variable set in the same account I'm starting
hadoop and hbase up in.
It seems to me that the HBase server needs something more to know that
this filter exists, but I don't know how to tell it, and I can't find
any good examples for creating a custom filter for my scanner.
I'd appreciate some more help. Thanks in advance.
Chris Schlesiger
[email protected]
Electronics/Software Engineer
U.S. Army Research Lab
Voc: 301-394-2473
Fax: 301-394-3591
DSN 290-2473
-----Original Message-----
From: Jonathan Gray [mailto:[email protected]]
Sent: Friday, June 11, 2010 1:31 PM
To: [email protected]
Subject: RE: Custom WritableByteArrayComparable crashing. Need help
please (UNCLASSIFIED)
Hey Chris,
Did you check out the logs on the server side? This should give some
more insight into what's going on.
One thing to keep in mind is that the server needs to know about the
filter you are using (it needs to be in a jar in the classpath). You
also may need to add it to HbaseObjectWritable.java so it can be sent
across the RPC layer.
There should be some previous emails on the mailing list about this.
JG
> -----Original Message-----
> From: Schlesiger, Chris (Civ, ARL/CISD)
> [mailto:[email protected]]
> Sent: Friday, June 11, 2010 10:13 AM
> To: [email protected]
> Subject: Custom WritableByteArrayComparable crashing. Need help please
> (UNCLASSIFIED)
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
> Our HBase contains people names for RowIDs, and we need our
> application to be able to try other types of string matching to find
> rows other than than the straight Substring and Regex matching that
> comes with the HBase API in the filters package.
>
> We were first working with HBase 0.20.1 where
> WritableByteArrayComparable was an interface. I tried implementing the
> interface and had no luck. I figured I'd better upgrade to the newest
> HBase, which I just did. Now in 0.20.4, WritableByteArrayComparable is
> an abstract class. Okay, so I changed my code to create a new child
> class and I still ran into the same problem with crashing and no rows
> returned by the scanner.
>
> So I decided to just try out the mere idea of creating a new child
> comparator to see if the same problem arose. I went to the HBase
> source code for the 0.20.4 tag, and looked at the SubstringComparator
> class in the filters package. I re-created it line by line in my own
> java project. The class I created was the same except that since I
> wasn't in the same package as the super class, I couldn't access the
> .value property as on line 74. But it looked to me that it wasn't
> needed since the getValue() method was overridden and ignoring .value
anyway.
>
> I created a scanner, created a RowFilter and passed in my
> SubstringComparatorDuplicate, and ran the program. Same exact crash.
> If I change my code only to create a SubstringComparator from the
> hbase jar instead, it works perfectly and I get rows returned by the
> scanner.
>
> Here are my symptoms:
>
> The program halts for almost 30 seconds, then I get the following
> exception:
>
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
> contact region server 127.0.0.1:60062 for region
> People,,1276369002801, row '', but failed after 10 attempts.
> Exceptions:
> java.io.IOException: Call to /127.0.0.1:60062 failed on local
> exception:
> java.io.EOFException
> java.io.IOException: Call to /127.0.0.1:60062 failed on local
> exception:
> java.io.EOFException
> java.io.IOException: Call to /127.0.0.1:60062 failed on local
> exception:
> java.io.EOFException
> java.io.IOException: Call to /127.0.0.1:60062 failed on local
> exception:
> java.io.EOFException
> java.io.IOException: Call to /127.0.0.1:60062 failed on local
> exception:
> java.io.EOFException
> java.io.IOException: Call to /127.0.0.1:60062 failed on local
> exception:
> java.io.EOFException
> java.io.IOException: Call to /127.0.0.1:60062 failed on local
> exception:
> java.io.EOFException
> java.io.IOException: Call to /127.0.0.1:60062 failed on local
> exception:
> java.io.EOFException
> java.io.IOException: Call to /127.0.0.1:60062 failed on local
> exception:
> java.io.EOFException
> java.io.IOException: Call to /127.0.0.1:60062 failed on local
> exception:
> java.io.EOFException
>
> at
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegi
> o
> n
> ServerWithRetries(HConnectionManager.java:1055)
> at
>
org.apache.hadoop.hbase.client.Htable$ClientScanner.nextScanner(HTable.
> j
> ava:2003)
> at
> org.apache.hadoop.hbase.client.Htable$ClientScanner.initialize(HTable.
> j
> a
> va:1923)
> at
> org.apache.hadoop.hbase.client.Htable.getScanner(HTable.java:403)
> at org.mycompany.test.search(Search.java:209)
> at org.mycompany.test.main(Main.java:40)
>
> I tried putting break points on each of the methods in my
> SubstringComparatorDuplicate, and the breakpoint in the write method
> gets hit 10 times and then I get the exception.
>
> I'm running HBase in pseudo-distributed mode on my development
> computer until I get the code working. The database is small. I'm
> running Hadoop
> 0.20.2 in dfs mode only for the same reason.
>
> I can't see any reason why my SubstringComparatorDuplicate class
> shouldn't work. It is identical to the code found here
>
> http://svn.apache.org/viewvc/hbase/tags/0.20.4/src/java/org/apache/had
> o
> o
> p/hbase/filter/SubstringComparator.java?view=markup
>
> With only 2 exceptions.
> 1) I'm in a different package, project, and jar file.
> 2) Line 74 is commented out
>
> Has anyone got any great ideas? I'm pretty much stuck now, and it's
> looking like there's a bug in the HBase code as near as I can tell.
>
> I'd appreciate the help. Thanks in advance.
>
> Chris Schlesiger
> [email protected]
> Electronics/Software Engineer
> U.S. Army Research Lab
> Voc: 301-394-2473
> Fax: 301-394-3591
> DSN 290-2473
> Classification: UNCLASSIFIED
> Caveats: NONE
Classification: UNCLASSIFIED
Caveats: NONE
