Chris: Use jps command to find the process Id of region server. Then use the following to find out classpath passed to region server (look for -classpath): ps aux | grep <pid>
Simple test would be placing your jar under $HBASE_HOME/lib and see what happens. On Wed, Jun 16, 2010 at 9:58 AM, Schlesiger, Chris (Civ, ARL/CISD) < [email protected]> wrote: > Classification: UNCLASSIFIED > Caveats: NONE > > I checked the server logs (smacking myself on the forehead for not doing > so before) and I see this exception: > > ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: > org.apache.hadoop.hbase.UnknownScannerException: Name: -1 > at > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.ja > va:1889) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav > a:57) > at > sun.reflec.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI > mpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:616) > at > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915 > ) > > And that jibes with the idea that the server needs to be aware of my > custom filter. I searched the mailing list archive for more information > on this and the only comment I could find was that the jar needs to be > in the "CLASSPATH on the RegionServer". I took this to mean that I > should create a jar with my filter in it, and put it in a folder on the > same computer that is running the Hbase RegionServer and make sure that > folder is in a CLASSPATH environment variable. > > I'm still a novice java programmer (C# is my area), so I did the best I > could. I moved my SubstringDuplicate class to its own jar and put it in > a folder (since I'm running in pseudo-distributed mode while I get all > this working, this is all on one computer). I created a CLASSPATH > environment variable and added the path the jar is in to it. Restarted > everything, and tested it. > > I'm still getting the same crash. > > I tried to follow up on the comment to "You also may need to add it to > HbaseObjectWritable.java so it can be sent across the RPC layer." a bit. > I looked at the HBaseObjectWritable.java file and it looks like the > WritableByteArrayComparable class is already in that file as it should > be. Since that's my super class, I would think my filter would be > covered, wouldn't it? I believe that's what was meant when it was said > that 0.20.4 would fix that problem. Since it went from an interface to a > base abstract super class for writable purposes. > > So I'm still stuck. > > Does the CLASSPATH need to be set in etc instead of my login > .bash_profile? I have the variable set in the same account I'm starting > hadoop and hbase up in. > > It seems to me that the HBase server needs something more to know that > this filter exists, but I don't know how to tell it, and I can't find > any good examples for creating a custom filter for my scanner. > > I'd appreciate some more help. Thanks in advance. > > Chris Schlesiger > [email protected] > Electronics/Software Engineer > U.S. Army Research Lab > Voc: 301-394-2473 > Fax: 301-394-3591 > DSN 290-2473 > > -----Original Message----- > From: Jonathan Gray [mailto:[email protected]] > Sent: Friday, June 11, 2010 1:31 PM > To: [email protected] > Subject: RE: Custom WritableByteArrayComparable crashing. Need help > please (UNCLASSIFIED) > > Hey Chris, > > Did you check out the logs on the server side? This should give some > more insight into what's going on. > > One thing to keep in mind is that the server needs to know about the > filter you are using (it needs to be in a jar in the classpath). You > also may need to add it to HbaseObjectWritable.java so it can be sent > across the RPC layer. > > There should be some previous emails on the mailing list about this. > > JG > > > -----Original Message----- > > From: Schlesiger, Chris (Civ, ARL/CISD) > > [mailto:[email protected]] > > Sent: Friday, June 11, 2010 10:13 AM > > To: [email protected] > > Subject: Custom WritableByteArrayComparable crashing. Need help please > > (UNCLASSIFIED) > > > > Classification: UNCLASSIFIED > > Caveats: NONE > > > > Our HBase contains people names for RowIDs, and we need our > > application to be able to try other types of string matching to find > > rows other than than the straight Substring and Regex matching that > > comes with the HBase API in the filters package. > > > > We were first working with HBase 0.20.1 where > > WritableByteArrayComparable was an interface. I tried implementing the > > > interface and had no luck. I figured I'd better upgrade to the newest > > HBase, which I just did. Now in 0.20.4, WritableByteArrayComparable is > > > an abstract class. Okay, so I changed my code to create a new child > > class and I still ran into the same problem with crashing and no rows > > returned by the scanner. > > > > So I decided to just try out the mere idea of creating a new child > > comparator to see if the same problem arose. I went to the HBase > > source code for the 0.20.4 tag, and looked at the SubstringComparator > > class in the filters package. I re-created it line by line in my own > > java project. The class I created was the same except that since I > > wasn't in the same package as the super class, I couldn't access the > > .value property as on line 74. But it looked to me that it wasn't > > needed since the getValue() method was overridden and ignoring .value > anyway. > > > > I created a scanner, created a RowFilter and passed in my > > SubstringComparatorDuplicate, and ran the program. Same exact crash. > > If I change my code only to create a SubstringComparator from the > > hbase jar instead, it works perfectly and I get rows returned by the > > scanner. > > > > Here are my symptoms: > > > > The program halts for almost 30 seconds, then I get the following > > exception: > > > > org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to > > contact region server 127.0.0.1:60062 for region > > People,,1276369002801, row '', but failed after 10 attempts. > > Exceptions: > > java.io.IOException: Call to /127.0.0.1:60062 failed on local > > exception: > > java.io.EOFException > > java.io.IOException: Call to /127.0.0.1:60062 failed on local > > exception: > > java.io.EOFException > > java.io.IOException: Call to /127.0.0.1:60062 failed on local > > exception: > > java.io.EOFException > > java.io.IOException: Call to /127.0.0.1:60062 failed on local > > exception: > > java.io.EOFException > > java.io.IOException: Call to /127.0.0.1:60062 failed on local > > exception: > > java.io.EOFException > > java.io.IOException: Call to /127.0.0.1:60062 failed on local > > exception: > > java.io.EOFException > > java.io.IOException: Call to /127.0.0.1:60062 failed on local > > exception: > > java.io.EOFException > > java.io.IOException: Call to /127.0.0.1:60062 failed on local > > exception: > > java.io.EOFException > > java.io.IOException: Call to /127.0.0.1:60062 failed on local > > exception: > > java.io.EOFException > > java.io.IOException: Call to /127.0.0.1:60062 failed on local > > exception: > > java.io.EOFException > > > > at > > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegi > > o > > n > > ServerWithRetries(HConnectionManager.java:1055) > > at > > > org.apache.hadoop.hbase.client.Htable$ClientScanner.nextScanner(HTable. > > j > > ava:2003) > > at > > org.apache.hadoop.hbase.client.Htable$ClientScanner.initialize(HTable. > > j > > a > > va:1923) > > at > > org.apache.hadoop.hbase.client.Htable.getScanner(HTable.java:403) > > at org.mycompany.test.search(Search.java:209) > > at org.mycompany.test.main(Main.java:40) > > > > I tried putting break points on each of the methods in my > > SubstringComparatorDuplicate, and the breakpoint in the write method > > gets hit 10 times and then I get the exception. > > > > I'm running HBase in pseudo-distributed mode on my development > > computer until I get the code working. The database is small. I'm > > running Hadoop > > 0.20.2 in dfs mode only for the same reason. > > > > I can't see any reason why my SubstringComparatorDuplicate class > > shouldn't work. It is identical to the code found here > > > > http://svn.apache.org/viewvc/hbase/tags/0.20.4/src/java/org/apache/had > > o > > o > > p/hbase/filter/SubstringComparator.java?view=markup > > > > With only 2 exceptions. > > 1) I'm in a different package, project, and jar file. > > 2) Line 74 is commented out > > > > Has anyone got any great ideas? I'm pretty much stuck now, and it's > > looking like there's a bug in the HBase code as near as I can tell. > > > > I'd appreciate the help. Thanks in advance. > > > > Chris Schlesiger > > [email protected] > > Electronics/Software Engineer > > U.S. Army Research Lab > > Voc: 301-394-2473 > > Fax: 301-394-3591 > > DSN 290-2473 > > Classification: UNCLASSIFIED > > Caveats: NONE > > Classification: UNCLASSIFIED > Caveats: NONE > >
