Dear Lars,
Some exceptions are raised when I concurrently read data from HBase.
Each thread to read is assigned a HTable instance. The version of
HBase I used is 0.92.0.
I cannot fix the problem. Could you please help me?
Thanks so much!
Best wishes,
Bing
Feb 6, 2013 12:21:31 AM
org.apache.hadoop.hbase.ipc.HBaseClient$Connection run
WARNING: Unexpected exception receiving call responses
java.lang.NullPointerException
at
org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521)
at
org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:297)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:593)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
Feb 6, 2013 12:21:31 AM
org.apache.hadoop.hbase.client.ScannerCallable close
WARNING: Ignore, probably already closed
java.io.IOException: Call to greatfreeweb/127.0.1.1:60020
failed on local exception: java.io.IOException: Unexpected exception
receiving call responses
at
org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:934)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:903)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
at $Proxy6.close(Unknown Source)
at
org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:112)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:74)
at
org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:39)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1325)
at
org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:1167)
at
org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1296)
at
org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1356)
at
com.greatfree.hbase.rank.NodeRankRetriever.loadNodeGroupNodeRankRowKeys(NodeRankRetriever.java:348)
at
com.greatfree.ranking.PersistNodeGroupNodeRanksThread.run(PersistNodeGroupNodeRanksThread.java:29)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) Caused by:
java.io.IOException: Unexpected exception receiving call responses
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:509)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521)
at
org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:297)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:593)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
I read data from HBase concurrently with the following code.
...
ExecutorService threadPool = Executors.newFixedThreadPool(100);
LoadNodeGroupNodeRankRowKeyThread thread;
for (String nodeKey : nodeKeys)
{
// Threads are initialized and executed here.
thread = new LoadNodeGroupNodeRankRowKeyThread(nodeKey);
threadPool.execute(thread);
}
Scanner in = new Scanner(System.in);
in.nextLine();
threadPool.shutdownNow();
...
The code of LoadNodeGroupNodeRankRowKeyThread is as follows,
...
public void run()
{
NodeRankRetriever retriever = new NodeRankRetriever();
// The following line reader data from HBase.
Set<String> rowKeys =
retriever.loadNodeGroupNodeRankRowKeys(this.hostNodeKey);
if (rowKeys.size() > 0)
{
for (String rowKey : rowKeys)
{
System.out.println(rowKey);
}
}
else
{
System.out.println("No data loaded");
}
retriever.dispose();
}
...
The constructor of NodeRankRetriever() just got an instance of HTable
from HTablePool from the following method.
...
public HTableInterface getTable(String tableName)
{
return this.hTablePool.getTable(tableName);
}
...
The method dispose() of NodeRankRetriever() just close the
HTableInterface created by HTablePool.
...
public void dispose()
{
try
{
this.rankTable.close();
}
catch (IOException e)
{
e.printStackTrace();
}
}
...
On Wed, Feb 6, 2013 at 1:05 PM, lars hofhansl <[email protected]> wrote:
> Are you sharing this.rankTable between threads? HTable is not thread safe.
>
> -- Lars
>
>
>
> ________________________________
> From: Bing Li <[email protected]>
> To: "[email protected]" <[email protected]>; user
> <[email protected]>
> Sent: Tuesday, February 5, 2013 8:54 AM
> Subject: Re: Is "synchronized" required?
>
> Dear all,
>
> After "synchronized" is removed from the method of writing, I get the
> following exceptions when reading. Before the removal, no such
> exceptions.
>
> Could you help me how to solve it?
>
> Thanks so much!
>
> Best wishes,
> Bing
>
> [java] Feb 6, 2013 12:21:31 AM
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection run
> [java] WARNING: Unexpected exception receiving call responses
> [java] java.lang.NullPointerException
> [java] at
> org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521)
> [java] at
> org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:297)
> [java] at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:593)
> [java] at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
> [java] Feb 6, 2013 12:21:31 AM
> org.apache.hadoop.hbase.client.ScannerCallable close
> [java] WARNING: Ignore, probably already closed
> [java] java.io.IOException: Call to greatfreeweb/127.0.1.1:60020
> failed on local exception: java.io.IOException: Unexpected exception
> receiving call responses
> [java] at
> org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:934)
> [java] at
> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:903)
> [java] at
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
> [java] at $Proxy6.close(Unknown Source)
> [java] at
> org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:112)
> [java] at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:74)
> [java] at
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:39)
> [java] at
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1325)
> [java] at
> org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:1167)
> [java] at
> org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1296)
> [java] at
> org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1356)
> [java] at
> com.greatfree.hbase.rank.NodeRankRetriever.LoadNodeGroupNodeRankRowKeys(NodeRankRetriever.java:348)
> [java] at
> com.greatfree.ranking.PersistNodeGroupNodeRanksThread.run(PersistNodeGroupNodeRanksThread.java:29)
> [java] at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> [java] at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> [java] at java.lang.Thread.run(Thread.java:662)
> [java] Caused by: java.io.IOException: Unexpected exception
> receiving call responses
> [java] at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:509)
> [java] Caused by: java.lang.NullPointerException
> [java] at
> org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:521)
> [java] at
> org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseObjectWritable.java:297)
> [java] at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:593)
> [java] at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:505)
>
>
> The code that causes the exceptions is as follows.
>
> public Set<String> LoadNodeGroupNodeRankRowKeys(String
> hostNodeKey, String groupKey, int timingScale)
> {
> List<Filter> nodeGroupFilterList = new ArrayList<Filter>();
>
> SingleColumnValueFilter hostNodeKeyFilter = new
> SingleColumnValueFilter(RankStructure.NODE_GROUP_NODE_RANK_FAMILY,
> RankStructure.NODE_GROUP_NODE_RANK_HOST_NODE_KEY_COLUMN,
> CompareFilter.CompareOp.EQUAL, new SubstringComparator(hostNodeKey));
> hostNodeKeyFilter.setFilterIfMissing(true);
> nodeGroupFilterList.add(hostNodeKeyFilter);
>
> SingleColumnValueFilter groupKeyFilter = new
> SingleColumnValueFilter(RankStructure.NODE_GROUP_NODE_RANK_FAMILY,
> RankStructure.NODE_GROUP_NODE_RANK_GROUP_KEY_COLUMN,
> CompareFilter.CompareOp.EQUAL, new SubstringComparator(groupKey));
> groupKeyFilter.setFilterIfMissing(true);
> nodeGroupFilterList.add(groupKeyFilter);
>
> SingleColumnValueFilter timingScaleFilter = new
> SingleColumnValueFilter(RankStructure.NODE_GROUP_NODE_RANK_FAMILY,
> RankStructure.NODE_GROUP_NODE_RANK_TIMING_SCALE_COLUMN,
> CompareFilter.CompareOp.EQUAL, new
> BinaryComparator(Bytes.toBytes(timingScale)));
> timingScaleFilter.setFilterIfMissing(true);
> nodeGroupFilterList.add(timingScaleFilter);
>
> FilterList nodeGroupFilter = new
> FilterList(nodeGroupFilterList);
> Scan scan = new Scan();
> scan.setFilter(nodeGroupFilter);
> scan.setCaching(Parameters.CACHING_SIZE);
> scan.setBatch(Parameters.BATCHING_SIZE);
>
> Set<String> rowKeySet = Sets.newHashSet();
> try
> {
> ResultScanner scanner =
> this.rankTable.getScanner(scan);
> for (Result result : scanner) //
> <---- EXCEPTIONS are raised at this line.
> {
> for (KeyValue kv : result.raw())
> {
>
> rowKeySet.add(Bytes.toString(kv.getRow()));
> break;
> }
> }
> scanner.close();
> }
> catch (IOException e)
> {
> e.printStackTrace();
> }
> return rowKeySet;
> }
>
>
> On Tue, Feb 5, 2013 at 4:20 AM, Bing Li <[email protected]> wrote:
>> Dear all,
>>
>> When writing data into HBase, sometimes I got exceptions. I guess they
>> might be caused by concurrent writings. But I am not sure.
>>
>> My question is whether it is necessary to put "synchronized" before
>> the writing methods? The following lines are the sample code.
>>
>> I think the directive, synchronized, must lower the performance of
>> writing. Sometimes concurrent writing is needed in my system.
>>
>> Thanks so much!
>>
>> Best wishes,
>> Bing
>>
>> public synchronized void AddDomainNodeRanks(String domainKey, int
>> timingScale, Map<String, Double> nodeRankMap)
>> {
>> List<Put> puts = new ArrayList<Put>();
>> Put domainKeyPut;
>> Put timingScalePut;
>> Put nodeKeyPut;
>> Put rankPut;
>>
>> byte[] domainNodeRankRowKey;
>>
>> for (Map.Entry<String, Double> nodeRankEntry : nodeRankMap.entrySet())
>> {
>> domainNodeRankRowKey =
>> Bytes.toBytes(RankStructure.DOMAIN_NODE_RANK_ROW +
>> Tools.GetAHash(domainKey + timingScale + nodeRankEntry.getKey()));
>>
>> domainKeyPut = new Put(domainNodeRankRowKey);
>> domainKeyPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
>> RankStructure.DOMAIN_NODE_RANK_DOMAIN_KEY_COLUMN,
>> Bytes.toBytes(domainKey));
>> puts.add(domainKeyPut);
>>
>> timingScalePut = new Put(domainNodeRankRowKey);
>> timingScalePut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
>> RankStructure.DOMAIN_NODE_RANK_TIMING_SCALE_COLUMN,
>> Bytes.toBytes(timingScale));
>> puts.add(timingScalePut);
>>
>> nodeKeyPut = new Put(domainNodeRankRowKey);
>> nodeKeyPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
>> RankStructure.DOMAIN_NODE_RANK_NODE_KEY_COLUMN,
>> Bytes.toBytes(nodeRankEntry.getKey()));
>> puts.add(nodeKeyPut);
>>
>> rankPut = new Put(domainNodeRankRowKey);
>> rankPut.add(RankStructure.DOMAIN_NODE_RANK_FAMILY,
>> RankStructure.DOMAIN_NODE_RANK_RANKS_COLUMN,
>> Bytes.toBytes(nodeRankEntry.getValue()));
>> puts.add(rankPut);
>> }
>>
>> try
>> {
>> this.rankTable.put(puts);
>> }
>> catch (IOException e)
>> {
>> e.printStackTrace();
>> }
>> }
