date:20130914

[jira] [Commented] (HBASE-9461) Some doc and cleanup in RPCServer

2013-09-14 Thread Nicolas Liochon (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-9461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13767422#comment-13767422
 ] 

Nicolas Liochon commented on HBASE-9461:


bq. was going to commit this since it some progress
sure. I was mainly hijacking the jira :-)

bq. The Delay stuff is unused I think. It was an experiment. Maybe I'll look at 
that next and purge it if I can.
It's my impression as well (the code is HBASE-3899). The idea seems very good, 
but if it's not used the ratio complexity vs. usefulness can't be good.

 Some doc and cleanup in RPCServer
 -

 Key: HBASE-9461
 URL: https://issues.apache.org/jira/browse/HBASE-9461
 Project: HBase
  Issue Type: Bug
Reporter: stack
Assignee: stack
 Attachments: 9461.txt, 9461v2.txt, ipc2.ucls


 RPC is a dog to follow.  I want to do buffer pooling for reading requests but 
 its tough drawing the diagram of who is doing what when.  HBASE-8884 seems to 
 have made it more involved still.  This issue is about doing a bit of 
 untangling.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-9390) coprocessors observers are not called during a recovery with the new log replay algorithm

2013-09-14 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-9390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13767479#comment-13767479
 ] 

Hudson commented on HBASE-9390:
---

SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #728 (See 
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/728/])
hbase-9390: coprocessors observers are not called during a recovery with the 
new log replay algorithm - 1 (jeffreyz: rev 1523172)
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java


 coprocessors observers are not called during a recovery with the new log 
 replay algorithm
 -

 Key: HBASE-9390
 URL: https://issues.apache.org/jira/browse/HBASE-9390
 Project: HBase
  Issue Type: Bug
  Components: Coprocessors, MTTR
Affects Versions: 0.95.2
Reporter: Nicolas Liochon
Assignee: Jeffrey Zhong
 Attachments: copro.patch, hbase-9390.patch, hbase-9390-v2.patch


 See the patch to reproduce the issue: If we activate log replay we don't have 
 the events on WAL restore.
 Pinging [~jeffreyz], we discussed this offline.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-9518) getFakedKey() improvement

2013-09-14 Thread Liang Xie (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-9518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13767482#comment-13767482
 ] 

Liang Xie commented on HBASE-9518:
--

Hi [~stack] you can see the new TestKeyValue case:
if the last kv of previous block and the first kv of current block have same 
postfix and just 1 offset diff, e.g.  100abcdefg and 101abcdefg,
before 9518, the getShortMidpointKey() will fallback to the default right kv, 
say 101abcdefg.
after 9518, it'll return 101, a shorter faked value, still reasonable, right? 
:)
And i found this corner case existing in current hbase test cases as well, so 
i'd like to let it go into community codebase also.

 getFakedKey() improvement
 -

 Key: HBASE-9518
 URL: https://issues.apache.org/jira/browse/HBASE-9518
 Project: HBase
  Issue Type: Improvement
  Components: regionserver
Affects Versions: 0.98.0, 0.96.1
Reporter: Liang Xie
Assignee: Liang Xie
 Attachments: HBASE-9518.txt, HBASE-9518-v2.txt


 make generating faked key algo more aggressive

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-9502) HStore.seekToScanner should handle magic value

2013-09-14 Thread Liang Xie (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-9502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13767483#comment-13767483
 ] 

Liang Xie commented on HBASE-9502:
--

Hi [~stack], i thought you need to patch HBASE-9518 first, then run this test, 
and i thought it'll fail w/o the patch.

 HStore.seekToScanner should handle magic value
 --

 Key: HBASE-9502
 URL: https://issues.apache.org/jira/browse/HBASE-9502
 Project: HBase
  Issue Type: Bug
  Components: regionserver, Scanners
Affects Versions: 0.98.0, 0.95.2, 0.96.1
Reporter: Liang Xie
Assignee: Liang Xie
 Attachments: HBASE-9502.txt


 due to faked key, the seekTo probably reture -2, and HStore.seekToScanner 
 should handle this corner case.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-9519) fix NPE in EncodedScannerV2.getFirstKeyInBlock()

2013-09-14 Thread Liang Xie (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-9519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13767503#comment-13767503
 ] 

Liang Xie commented on HBASE-9519:
--

take it easy, nobody is pissed off :)

could we kick off another QA run manually on build server?

 fix NPE in EncodedScannerV2.getFirstKeyInBlock()
 

 Key: HBASE-9519
 URL: https://issues.apache.org/jira/browse/HBASE-9519
 Project: HBase
  Issue Type: Bug
  Components: HFile
Affects Versions: 0.98.0, 0.96.1
Reporter: Liang Xie
Assignee: Liang Xie
 Attachments: HBASE-9519.txt, HBASE-9519-v2.txt


 we observed a reproducable NPE while scanning special table under special 
 condition in our IntegratedTesting scenario, it was fixed by appling the 
 attached patch.
 org.apache.hadoop.hbase.client.ScannerCallable@67ee75a5, java.io.IOException: 
 java.io.IOException: java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1186)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:1175)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2391)
 at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source)
 at 
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at 
 org.apache.hadoop.hbase.ipc.SecureRpcEngine$Server.call(SecureRpcEngine.java:456)
 at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
 Caused by: java.lang.NullPointerException
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$EncodedScannerV2.getFirstKeyInBlock(HFileReaderV2.java:1071)
 at 
 org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekBefore(HFileReaderV2.java:547)
 at 
 org.apache.hadoop.hbase.io.HalfStoreFileReader$1.seekBefore(HalfStoreFileReader.java:159)
 at 
 org.apache.hadoop.hbase.io.HalfStoreFileReader$1.seekBefore(HalfStoreFileReader.java:142)
 at 
 org.apache.hadoop.hbase.io.HalfStoreFileReader.getLastKey(HalfStoreFileReader.java:267)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFile$Reader.passesKeyRangeFilter(StoreFile.java:1543)
 at 
 org.apache.hadoop.hbase.regionserver.StoreFileScanner.shouldUseScanner(StoreFileScanner.java:375)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.selectScannersFrom(StoreScanner.java:298)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.getScannersNoCompaction(StoreScanner.java:262)
 at 
 org.apache.hadoop.hbase.regionserver.StoreScanner.init(StoreScanner.java:149)
 at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:2122)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.init(HRegion.java:3460)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1645)
 at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1635)
 at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1610)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2377)
 ... 5 more

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-9528) Adaptive compaction

2013-09-14 Thread Liang Xie (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-9528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13767504#comment-13767504
 ] 

Liang Xie commented on HBASE-9528:
--

yeh, both central planning compaction and compaction scheduler seems more 
suitable:)


 Adaptive compaction
 ---

 Key: HBASE-9528
 URL: https://issues.apache.org/jira/browse/HBASE-9528
 Project: HBase
  Issue Type: Sub-task
Affects Versions: 0.98.0
Reporter: Liang Xie

 Currently, the compaction policy granularity is based on single machine. we 
 had a thought that introduce a new cluster granularity decision, such that we 
 could improve those case per cluster running status:
 1) many nodes are compacting aggressive, we call it cluster compaction storm, 
 we should throttle it.
 2) do more compaction if low traffic in current cluster(similar with off-peak 
 feature), not limit by config timerange(like off-peak timerange), just 
 trigger by load or qps or other stuff.
 comments? thanks

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-9047) Tool to handle finishing replication when the cluster is offline

2013-09-14 Thread Demai Ni (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-9047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13767521#comment-13767521
]

Demai Ni commented on HBASE-9047:
-

[~jdcryans]

thank you so much for the review comments and suggestions. I will remove the
'system.out.println', fix the typo, and remove the 'copyright' line. Also I
will remove 'conf.setBoolean(HConstants.REPLICATION_ENABLE_KEY, true)', and
change the testcase per your suggestion.

about the Thread.sleep(3), many thanks for pointing it out. It would be a
bug in making. I will put some zookeeper checking with a timeout loop.

Let me look into the 'tool' class. I assume to make it a Runnable class, and
use run() method as the main body is the key here. Thanks for the suggestion.

As for the code style, I am using eclipse and import a
hbase_eclipse_formatter.xml(http://hbase.apache.org/book/ides.html), but I
realized that I must miss something from the experience of this and past patch
submission. Is it the right way to follow? Is there a style checking script
that I can run before submit? Thanks

Have a nice weekend

Demai

Tool to handle finishing replication when the cluster is offline

Key: HBASE-9047
URL: https://issues.apache.org/jira/browse/HBASE-9047
Project: HBase
Issue Type: New Feature
Reporter: Jean-Daniel Cryans
Assignee: Demai Ni
Attachments: HBASE-9047-0.94.9-v0.PATCH, HBASE-9047-trunk-v0.patch

We're having a discussion on the mailing list about replicating the data on a
cluster that was shut down in an offline fashion. The motivation could be
that you don't want to bring HBase back up but still need that data on the
slave.
So I have this idea of a tool that would be running on the master cluster
while it is down, although it could also run at any time. Basically it would
be able to read the replication state of each master region server, finish
replicating what's missing to all the slave, and then clear that state in
zookeeper.
The code that handles replication does most of that already, see
ReplicationSourceManager and ReplicationSource. Basically when
ReplicationSourceManager.init() is called, it will check all the queues in ZK
and try to grab those that aren't attached to a region server. If the whole
cluster is down, it will grab all of them.
The beautiful thing here is that you could start that tool on all your
machines and the load will be spread out, but that might not be a big concern
if replication wasn't lagging since it would take a few seconds to finish
replicating the missing data for each region server.
I'm guessing when starting ReplicationSourceManager you'd give it a fake
region server ID, and you'd tell it not to start its own source.
FWIW the main difference in how replication is handled between Apache's HBase
and Facebook's is that the latter is always done separately of HBase itself.
This jira isn't about doing that.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

42 matches

Mail list logo