[jira] [Commented] (HBASE-7404) Bucket Cache:A solution about CMS,Heap Fragment and Big Cache on HBASE
[ https://issues.apache.org/jira/browse/HBASE-7404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13803905#comment-13803905 ] Jerry He commented on HBASE-7404: - Any intention and activity putting this in 0.94? Bucket Cache:A solution about CMS,Heap Fragment and Big Cache on HBASE -- Key: HBASE-7404 URL: https://issues.apache.org/jira/browse/HBASE-7404 Project: HBase Issue Type: New Feature Affects Versions: 0.94.3 Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.95.0 Attachments: 7404-0.94-fixed-lines.txt, 7404-trunk-v10.patch, 7404-trunk-v11.patch, 7404-trunk-v12.patch, 7404-trunk-v13.patch, 7404-trunk-v13.txt, 7404-trunk-v14.patch, BucketCache.pdf, hbase-7404-94v2.patch, HBASE-7404-backport-0.94.patch, hbase-7404-trunkv2.patch, hbase-7404-trunkv9.patch, Introduction of Bucket Cache.pdf First, thanks @neil from Fusion-IO share the source code. Usage: 1.Use bucket cache as main memory cache, configured as the following: –hbase.bucketcache.ioengine heap –hbase.bucketcache.size 0.4 (size for bucket cache, 0.4 is a percentage of max heap size) 2.Use bucket cache as a secondary cache, configured as the following: –hbase.bucketcache.ioengine file:/disk1/hbase/cache.data(The file path where to store the block data) –hbase.bucketcache.size 1024 (size for bucket cache, unit is MB, so 1024 means 1GB) –hbase.bucketcache.combinedcache.enabled false (default value being true) See more configurations from org.apache.hadoop.hbase.io.hfile.CacheConfig and org.apache.hadoop.hbase.io.hfile.bucket.BucketCache What's Bucket Cache? It could greatly decrease CMS and heap fragment by GC It support a large cache space for High Read Performance by using high speed disk like Fusion-io 1.An implementation of block cache like LruBlockCache 2.Self manage blocks' storage position through Bucket Allocator 3.The cached blocks could be stored in the memory or file system 4.Bucket Cache could be used as a mainly block cache(see CombinedBlockCache), combined with LruBlockCache to decrease CMS and fragment by GC. 5.BucketCache also could be used as a secondary cache(e.g. using Fusionio to store block) to enlarge cache space How about SlabCache? We have studied and test SlabCache first, but the result is bad, because: 1.SlabCache use SingleSizeCache, its use ratio of memory is low because kinds of block size, especially using DataBlockEncoding 2.SlabCache is uesd in DoubleBlockCache, block is cached both in SlabCache and LruBlockCache, put the block to LruBlockCache again if hit in SlabCache , it causes CMS and heap fragment don't get any better 3.Direct heap performance is not good as heap, and maybe cause OOM, so we recommend using heap engine See more in the attachment and in the patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9742) Add Documentation For Simple User Access
[
https://issues.apache.org/jira/browse/HBASE-9742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13803907#comment-13803907
]
stack commented on HBASE-9742:
--
You want me to apply this stuff to 0.94 Jesse? Usually we just let the trunk
doc do the job for users. I tried applying the above but got this:
{code}
durruti:0.94 stack$ patch -p0 ~/Downloads/simplewith0_94.diff
can't find file to patch at input line 5
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--
|Index: src/main/docbkx/security.xml
|===
|--- src/main/docbkx/security.xml (revision 1531333)
|+++ src/main/docbkx/security.xml (working copy)
--
File to patch: ^C
durruti:0.94 stack$
durruti:0.94 stack$ patch -p1 ~/Downloads/simplewith0_94.diff
can't find file to patch at input line 5
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--
|Index: src/main/docbkx/security.xml
|===
|--- src/main/docbkx/security.xml (revision 1531333)
|+++ src/main/docbkx/security.xml (working copy)
--
{code}
It works for you? Thanks boss.
Add Documentation For Simple User Access
Key: HBASE-9742
URL: https://issues.apache.org/jira/browse/HBASE-9742
Project: HBase
Issue Type: Improvement
Components: documentation
Affects Versions: 0.94.12, 0.96.0
Reporter: Jesse Anderson
Assignee: Jesse Anderson
Fix For: 0.96.0
Attachments: simple0_94.diff, simple.diff, simplewith0_94.diff
The [security section|http://hbase.apache.org/book/security.html] in the
HBase book only talks about using Kerberos. There is a simple user access
mode too that is not documented.
This should be documented for development systems and HBase installs where
security is not an issue, but they want to prevent user mistakes.
I've added a section to the security chapter that talks about simple user
access. The new section makes it very clear it is not secure.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-8894) Forward port compressed l2 cache from 0.89fb
[
https://issues.apache.org/jira/browse/HBASE-8894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13803910#comment-13803910
]
Liang Xie commented on HBASE-8894:
--
0) hardware or os:
System | Supermicro; X9SCD; v0123456789 (Desktop)
Service Tag | 0123456789
Platform | Linux
Release | CentOS release 6.2 (Final)
Kernel | 2.6.32-220.el6.x86_64
Architecture | CPU = 64-bit, OS = 64-bit
Threading | NPTL 2.12
SELinux | Disabled
Virtualized | No virtualization detected
# Processor ##
Processors | physical = 1, cores = 4, virtual = 8, hyperthreading = yes
Speeds | 8x3292.141
Models | 8xIntel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz
Caches | 8x8192 KB
# Memory #
Total | 31.3G
Free | 219.9M
Used | physical = 31.1G, swap allocated = 33.5G, swap used = 213.6M, virtual =
31.3G
Buffers | 438.0M
Caches | 20.4G
Dirty | 1340 kB
UsedRSS | 9.2G
Swappiness | 60
DirtyPolicy | 20, 10
DirtyStatus | 0, 0
Locator Size Speed Form Factor Type Type Detail
= = = = ===
1333 MHz
1333 MHz
1333 MHz
1333 MHz
ChannelA-DIMM0 8192 MB 1333 MHz DIMM DDR3
Synchronous
ChannelA-DIMM1 8192 MB 1333 MHz DIMM DDR3
Synchronous
ChannelB-DIMM0 8192 MB 1333 MHz DIMM DDR3
Synchronous
ChannelB-DIMM1 8192 MB 1333 MHz DIMM DDR3
Synchronous
1) important vm options:
-Xmx2560m -XX:MaxDirectMemorySize=2560m -Xms2560m -Xmn512m -Xss256k
-XX:MaxPermSize=128m -XX:PermSize=128m -XX:+UseConcMarkSweepGC -verbose:gc
-XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:SurvivorRatio=1
-XX:+UseCMSCompactAtFullCollection -XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly -XX:+CMSParallelRemarkEnabled
-XX:TargetSurvivorRatio=80 -XX:+CMSScavengeBeforeRemark -XX:ConcGCThreads=8
-XX:ParallelGCThreads=8 -XX:PretenureSizeThreshold=4m
-XX:+CMSConcurrentMTEnabled -XX:+ExplicitGCInvokesConcurrent
-XX:-UseBiasedLocking ...
I used the absolutely the same vm options during both bucket cache and l2 cache
testing.
2) important config:
hbase.bucketcache.ioengine=offheap
hfile.l2.bucketcache.ioengine=offheap
hbase.bucketcache.size=0.7
hfile.l2.bucketcache.size=0.7
hfile.block.cache.size=0.7
hbase.regionserver.global.memstore.lowerLimit=0.1
hbase.regionserver.global.memstore.upperLimit=0.1
3) jdk version:
java -version
java version 1.6.0_37
Java(TM) SE Runtime Environment (build 1.6.0_37-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.12-b01, mixed mode)
4) i just tested 100% read, no write at all.
additional, my test data size :
dfs -du -s -h /hbase/hytst-replication/ycsb_xieliang
3.6g /hbase/hytst-replication/ycsb_xieliang
it's a YCSB table with 500w records.
I ensure there's a read ops warmup before test running, until almost all data
be loaded into os cache. so in this scenario the data could not fit into hbase
cache all, but no disk io to dominate the test result:)
Forward port compressed l2 cache from 0.89fb
Key: HBASE-8894
URL: https://issues.apache.org/jira/browse/HBASE-8894
Project: HBase
Issue Type: New Feature
Reporter: stack
Assignee: Liang Xie
Priority: Critical
Attachments: HBASE-8894-0.94-v1.txt, HBASE-8894-0.94-v2.txt
Forward port Alex's improvement on hbase-7407 from 0.89-fb branch:
{code}
1 r1492797 | liyin | 2013-06-13 11:18:20 -0700 (Thu, 13 Jun 2013) | 43 lines
2
3 [master] Implements a secondary compressed cache (L2 cache)
4
5 Author: avf
6
7 Summary:
8 This revision implements compressed and encoded second-level cache with
off-heap
9 (and optionally on-heap) storage and a bucket-allocator based on
HBASE-7404.
10
11 BucketCache from HBASE-7404 is extensively modified to:
12
13 * Only handle byte arrays (i.e., no more serialization/deserialization
within)
14 * Remove persistence support for the time being
15 * Keep an index of hfilename to blocks for efficient eviction on close
16
17 A new interface (L2Cache) is introduced in order to separate it from the
current
18 implementation. The L2 cache is then integrated into the classes that
handle
19 reading from and writing to HFiles to allow cache-on-write as well as
20 cache-on-read. Metrics for the L2 cache are integrated into
RegionServerMetrics
21 much in the same fashion as metrics for the existing (L2) BlockCache.
22
23 Additionally, CacheConfig class is re-refactored to configure the L2
cache,
24 replace multile constructors with a Builder, as well as replace static
methods
25 for instantiating the caches with abstract factories (with singleton
26 implementations for
[jira] [Commented] (HBASE-8859) truncate_preserve should get table split keys as it is instead of converting them to string type and then again to bytes
[
https://issues.apache.org/jira/browse/HBASE-8859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13803912#comment-13803912
]
stack commented on HBASE-8859:
--
Should the below call be in the HTableInterface too?
getSplitKeys
truncate_preserve should get table split keys as it is instead of converting
them to string type and then again to bytes
Key: HBASE-8859
URL: https://issues.apache.org/jira/browse/HBASE-8859
Project: HBase
Issue Type: Bug
Components: scripts
Affects Versions: 0.95.1
Reporter: rajeshbabu
Assignee: rajeshbabu
Fix For: 0.98.0, 0.96.1
Attachments: HBASE-8859-Test_to_reproduce.patch,
HBASE-8859_trunk_2.patch, HBASE-8859_trunk.patch
If we take int,long or double bytes as split keys then we are not creating
table with same split keys because converting them to strings directly and to
bytes is giving different split keys, sometimes getting IllegalArgument
exception because of same split keys(converted). Instead we can get split
keys directly from HTable and pass them while creating table.
{code}
h_table = org.apache.hadoop.hbase.client.HTable.new(conf, table_name)
splits = h_table.getRegionLocations().keys().map{|i| i.getStartKey}
:byte
splits = org.apache.hadoop.hbase.util.Bytes.toByteArrays(splits)
{code}
{code}
Truncating 'emp3' table (it may take a while):
- Disabling table...
- Dropping table...
- Creating table with region boundaries...
ERROR: java.lang.IllegalArgumentException: All split keys must be unique,
found duplicate: B\x11S\xEF\xBF\xBD\xEF\xBF\xBD\xEF\xBF\xBD\x00\x00,
B\x11S\xEF\xBF\xBD\xEF\xBF\xBD\xEF\xBF\xBD\x00\x00
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9645) Regionserver halt because of HLog's Logic Error Snapshot seq id from earlier flush still present!
[
https://issues.apache.org/jira/browse/HBASE-9645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-9645:
-
Resolution: Fixed
Fix Version/s: (was: 0.96.1)
(was: 0.94.13)
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Applied to trunk.
Regionserver halt because of HLog's Logic Error Snapshot seq id from earlier
flush still present!
---
Key: HBASE-9645
URL: https://issues.apache.org/jira/browse/HBASE-9645
Project: HBase
Issue Type: Bug
Components: regionserver, wal
Affects Versions: 0.94.10
Environment: Linux 2.6.32-el5.x86_64
Reporter: Victor Xu
Priority: Critical
Fix For: 0.98.0
Attachments: 9645v2.txt, HBASE_9645-0.94.10.patch
I upgrade my hbase cluster to 0.94.10 three weeks ago, and this case happened
several days after that. I change the bug's priority to 'Critical' because
every time it happens, a regionserver halt down. All of them have the same
log:
{noformat}
ERROR org.apache.hadoop.hbase.regionserver.wal.HLog: Logic Error Snapshot seq
id from earlier flush still present! for region
c0d88db4ce3606842fbec9d34c38f707 overwritten oldseq=80114270537with new
seq=80115066829
{noformat}
I check the code finding that it locates at HLog.startCacheFlush method. The
'lastSeqWritten' has been locked. Maybe something wrong happened outside the
HLog that change it by mistake.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-8894) Forward port compressed l2 cache from 0.89fb
[
https://issues.apache.org/jira/browse/HBASE-8894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13803923#comment-13803923
]
Alex Feinberg commented on HBASE-8894:
--
Hi [~xieliang007]
This configuration creates way too memory pressure. I'd also suggest using
Java7 (this is the setup that I used at FB).
I'll try to come up with actual JVM options I used, but I used:
- Total JVM heap size: ~14gb (Xmx and Xms)
space available to the l1 cache (the regular block cache): 0.4-0.5 (we have
never went above 0.5 as that caused too many problems)
- New gen size: 4gb (I *think*, not too sure)
- Direct memory: 10gb (I am roughly scaling down to your machine -- you want to
leave some memory available to OS)
- I gave 0.9% of direct memory to the L2 cache
I did use CMS, but I don't remember the CMS initiating ratio. G1 might also work
I will try to find the exact JVM configuration
How much memory did you give to memstore?
Thanks!
- af
Forward port compressed l2 cache from 0.89fb
Key: HBASE-8894
URL: https://issues.apache.org/jira/browse/HBASE-8894
Project: HBase
Issue Type: New Feature
Reporter: stack
Assignee: Liang Xie
Priority: Critical
Attachments: HBASE-8894-0.94-v1.txt, HBASE-8894-0.94-v2.txt
Forward port Alex's improvement on hbase-7407 from 0.89-fb branch:
{code}
1 r1492797 | liyin | 2013-06-13 11:18:20 -0700 (Thu, 13 Jun 2013) | 43 lines
2
3 [master] Implements a secondary compressed cache (L2 cache)
4
5 Author: avf
6
7 Summary:
8 This revision implements compressed and encoded second-level cache with
off-heap
9 (and optionally on-heap) storage and a bucket-allocator based on
HBASE-7404.
10
11 BucketCache from HBASE-7404 is extensively modified to:
12
13 * Only handle byte arrays (i.e., no more serialization/deserialization
within)
14 * Remove persistence support for the time being
15 * Keep an index of hfilename to blocks for efficient eviction on close
16
17 A new interface (L2Cache) is introduced in order to separate it from the
current
18 implementation. The L2 cache is then integrated into the classes that
handle
19 reading from and writing to HFiles to allow cache-on-write as well as
20 cache-on-read. Metrics for the L2 cache are integrated into
RegionServerMetrics
21 much in the same fashion as metrics for the existing (L2) BlockCache.
22
23 Additionally, CacheConfig class is re-refactored to configure the L2
cache,
24 replace multile constructors with a Builder, as well as replace static
methods
25 for instantiating the caches with abstract factories (with singleton
26 implementations for both the existing LruBlockCache and the newly
introduced
27 BucketCache based L2 cache)
28
29 Test Plan:
30 1) Additional unit tests
31 2) Stress test on a single devserver
32 3) Test on a single-node in shadow cluster
33 4) Test on a whole shadow cluster
34
35 Revert Plan:
36
37 Reviewers: liyintang, aaiyer, rshroff, manukranthk, adela
38
39 Reviewed By: liyintang
40
41 CC: gqchen, hbase-eng@
42
43 Differential Revision: https://phabricator.fb.com/D837264
44
45 Task ID: 2325295
7
6 r1492340 | liyin | 2013-06-12 11:36:03 -0700 (Wed, 12 Jun 2013) | 21 lines
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-8755) A new write thread model for HLog to improve the overall HBase write throughput
[ https://issues.apache.org/jira/browse/HBASE-8755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13803932#comment-13803932 ] stack commented on HBASE-8755: -- Five datanodes with fusionio. hbase tip of 0.96 branch and hadoop-2.1.0-beta. HLogPE on master node. ||Threads||w/o patch time||w/o patch ops||w/ patch time||w/ patch ops|| |1|1048.033s|954.168ops/s|1100.423s|908.741ops/s| |1|1042.126s|959.577ops/s|1156.557s|864.635ops/s| |1|1052.601s|950.028ops/s|1143.271s|874.683ops/s| |5|904.176s|5529.896ops/s|1916.229s|2609.292ops/s| |5|910.469s|5491.675ops/s|1911.841s|2615.280ops/s| |5|925.778s|5400.863ops/s|1970.565s|2537.344ops/s| |50|2699.752s|18520.221ops/s|1889.877s|26456.748ops/s| |50|2689.678s|18589.586ops/s|1922.716s|26004.881ops/s| |50|2711.144s|18442.398ops/s|1893.439s|26406.977ops/s| |75|4945.563s|15165.108ops/s|1997.553s|37545.938ops/s| |75|4852.779s|15455.063ops/s|1992.425s|37642.570ops/s| |75|4921.685s|15238.684ops/s|-|-| |100|6224.527s|16065.479ops/s|2086.691s|47922.766ops/s| |100|6195.727s|16140.156ops/s|2091.869s|47804.145ops/s| Diffs are small when 1 thread only. Its bad at 5 threads but thereafter the patch starts to shine. If we could make the 5 threads better, we could commit this patch. A new write thread model for HLog to improve the overall HBase write throughput --- Key: HBASE-8755 URL: https://issues.apache.org/jira/browse/HBASE-8755 Project: HBase Issue Type: Improvement Components: Performance, wal Reporter: Feng Honghua Assignee: stack Priority: Critical Fix For: 0.96.1 Attachments: 8755trunkV2.txt, HBASE-8755-0.94-V0.patch, HBASE-8755-0.94-V1.patch, HBASE-8755-trunk-V0.patch, HBASE-8755-trunk-V1.patch In current write model, each write handler thread (executing put()) will individually go through a full 'append (hlog local buffer) = HLog writer append (write to hdfs) = HLog writer sync (sync hdfs)' cycle for each write, which incurs heavy race condition on updateLock and flushLock. The only optimization where checking if current syncTillHere txid in expectation for other thread help write/sync its own txid to hdfs and omitting the write/sync actually help much less than expectation. Three of my colleagues(Ye Hangjun / Wu Zesheng / Zhang Peng) at Xiaomi proposed a new write thread model for writing hdfs sequence file and the prototype implementation shows a 4X improvement for throughput (from 17000 to 7+). I apply this new write thread model in HLog and the performance test in our test cluster shows about 3X throughput improvement (from 12150 to 31520 for 1 RS, from 22000 to 7 for 5 RS), the 1 RS write throughput (1K row-size) even beats the one of BigTable (Precolator published in 2011 says Bigtable's write throughput then is 31002). I can provide the detailed performance test results if anyone is interested. The change for new write thread model is as below: 1 All put handler threads append the edits to HLog's local pending buffer; (it notifies AsyncWriter thread that there is new edits in local buffer) 2 All put handler threads wait in HLog.syncer() function for underlying threads to finish the sync that contains its txid; 3 An single AsyncWriter thread is responsible for retrieve all the buffered edits in HLog's local pending buffer and write to the hdfs (hlog.writer.append); (it notifies AsyncFlusher thread that there is new writes to hdfs that needs a sync) 4 An single AsyncFlusher thread is responsible for issuing a sync to hdfs to persist the writes by AsyncWriter; (it notifies the AsyncNotifier thread that sync watermark increases) 5 An single AsyncNotifier thread is responsible for notifying all pending put handler threads which are waiting in the HLog.syncer() function 6 No LogSyncer thread any more (since there is always AsyncWriter/AsyncFlusher threads do the same job it does) -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-8894) Forward port compressed l2 cache from 0.89fb
[
https://issues.apache.org/jira/browse/HBASE-8894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13803933#comment-13803933
]
Liang Xie commented on HBASE-8894:
--
bq. How much memory did you give to memstore?
0.1 * 2560m - 256m
bq. This configuration creates way too memory pressure
yes, per my point of view, the biggest motive to use l2 cache or bucket cache
probably is to alleviate gc hurt caused by read ops. so i need to watch what's
the diff between bucket cache and l2 cache under heavy read scenario:)
Forward port compressed l2 cache from 0.89fb
Key: HBASE-8894
URL: https://issues.apache.org/jira/browse/HBASE-8894
Project: HBase
Issue Type: New Feature
Reporter: stack
Assignee: Liang Xie
Priority: Critical
Attachments: HBASE-8894-0.94-v1.txt, HBASE-8894-0.94-v2.txt
Forward port Alex's improvement on hbase-7407 from 0.89-fb branch:
{code}
1 r1492797 | liyin | 2013-06-13 11:18:20 -0700 (Thu, 13 Jun 2013) | 43 lines
2
3 [master] Implements a secondary compressed cache (L2 cache)
4
5 Author: avf
6
7 Summary:
8 This revision implements compressed and encoded second-level cache with
off-heap
9 (and optionally on-heap) storage and a bucket-allocator based on
HBASE-7404.
10
11 BucketCache from HBASE-7404 is extensively modified to:
12
13 * Only handle byte arrays (i.e., no more serialization/deserialization
within)
14 * Remove persistence support for the time being
15 * Keep an index of hfilename to blocks for efficient eviction on close
16
17 A new interface (L2Cache) is introduced in order to separate it from the
current
18 implementation. The L2 cache is then integrated into the classes that
handle
19 reading from and writing to HFiles to allow cache-on-write as well as
20 cache-on-read. Metrics for the L2 cache are integrated into
RegionServerMetrics
21 much in the same fashion as metrics for the existing (L2) BlockCache.
22
23 Additionally, CacheConfig class is re-refactored to configure the L2
cache,
24 replace multile constructors with a Builder, as well as replace static
methods
25 for instantiating the caches with abstract factories (with singleton
26 implementations for both the existing LruBlockCache and the newly
introduced
27 BucketCache based L2 cache)
28
29 Test Plan:
30 1) Additional unit tests
31 2) Stress test on a single devserver
32 3) Test on a single-node in shadow cluster
33 4) Test on a whole shadow cluster
34
35 Revert Plan:
36
37 Reviewers: liyintang, aaiyer, rshroff, manukranthk, adela
38
39 Reviewed By: liyintang
40
41 CC: gqchen, hbase-eng@
42
43 Differential Revision: https://phabricator.fb.com/D837264
44
45 Task ID: 2325295
7
6 r1492340 | liyin | 2013-06-12 11:36:03 -0700 (Wed, 12 Jun 2013) | 21 lines
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-8859) truncate_preserve should get table split keys as it is instead of converting them to string type and then again to bytes
[
https://issues.apache.org/jira/browse/HBASE-8859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13803977#comment-13803977
]
rajeshbabu commented on HBASE-8859:
---
[~saint@gmail.com]
Since no split keys related APIs in HTableInterface I didn't add there. I will
raise a separate issue for adding all split key related APIs in HTableInterface
and work on it. Is it ok?
truncate_preserve should get table split keys as it is instead of converting
them to string type and then again to bytes
Key: HBASE-8859
URL: https://issues.apache.org/jira/browse/HBASE-8859
Project: HBase
Issue Type: Bug
Components: scripts
Affects Versions: 0.95.1
Reporter: rajeshbabu
Assignee: rajeshbabu
Fix For: 0.98.0, 0.96.1
Attachments: HBASE-8859-Test_to_reproduce.patch,
HBASE-8859_trunk_2.patch, HBASE-8859_trunk.patch
If we take int,long or double bytes as split keys then we are not creating
table with same split keys because converting them to strings directly and to
bytes is giving different split keys, sometimes getting IllegalArgument
exception because of same split keys(converted). Instead we can get split
keys directly from HTable and pass them while creating table.
{code}
h_table = org.apache.hadoop.hbase.client.HTable.new(conf, table_name)
splits = h_table.getRegionLocations().keys().map{|i| i.getStartKey}
:byte
splits = org.apache.hadoop.hbase.util.Bytes.toByteArrays(splits)
{code}
{code}
Truncating 'emp3' table (it may take a while):
- Disabling table...
- Dropping table...
- Creating table with region boundaries...
ERROR: java.lang.IllegalArgumentException: All split keys must be unique,
found duplicate: B\x11S\xEF\xBF\xBD\xEF\xBF\xBD\xEF\xBF\xBD\x00\x00,
B\x11S\xEF\xBF\xBD\xEF\xBF\xBD\xEF\xBF\xBD\x00\x00
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9767) Support OperationAttributes in ImportTSV Parser
[ https://issues.apache.org/jira/browse/HBASE-9767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13803981#comment-13803981 ] ramkrishna.s.vasudevan commented on HBASE-9767: --- Will commit this later today. Support OperationAttributes in ImportTSV Parser --- Key: HBASE-9767 URL: https://issues.apache.org/jira/browse/HBASE-9767 Project: HBase Issue Type: Improvement Affects Versions: 0.98.0 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Attachments: HBASE-9767_1.patch, HBASE-9767.patch This JIRA aims at supporting the operation attributes in bulk loads. Ideally this operation attributes once extracted has to be set in the mappers/reducers. In case of mappers using TableOutPutFormat this would be very helpful when the puts are done through HTable. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HBASE-9830) Backport HBASE-9605 to 0.94
chendihao created HBASE-9830: Summary: Backport HBASE-9605 to 0.94 Key: HBASE-9830 URL: https://issues.apache.org/jira/browse/HBASE-9830 Project: HBase Issue Type: Improvement Affects Versions: 0.94.3 Reporter: chendihao Priority: Minor Backport HBASE-9605 which is about Allow AggregationClient to skip specifying column family for row count aggregate -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HBASE-9831) 'hbasefsck.numthreads' property can not pass to hbck via generic option
takeshi.miao created HBASE-9831:
---
Summary: 'hbasefsck.numthreads' property can not pass to hbck via
generic option
Key: HBASE-9831
URL: https://issues.apache.org/jira/browse/HBASE-9831
Project: HBase
Issue Type: Bug
Components: hbck
Affects Versions: 0.94.12
Reporter: takeshi.miao
Priority: Minor
Fix For: 0.94.13
We use generic option way to pass _'hbasefsck.numthreads'_ property to _'hbase
hbck'_, but it does not accept our new setting value
{code}
hbase hbck -D hbasefsck.numthreads=5
{code}
We can still find there are threads over than 5 we already set via generic
opttion
{code}
[2013-10-24
09:25:02,561][pool-2-thread-6][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com (auth:KERBEROS)
from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
(UserGroupInformation.java:1430)
[2013-10-24
09:25:02,562][pool-2-thread-10][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com (auth:KERBEROS)
from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
(UserGroupInformation.java:1430)
[2013-10-24
09:25:02,565][pool-2-thread-13][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com (auth:KERBEROS)
from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
(UserGroupInformation.java:1430)
[2013-10-24
09:25:02,566][pool-2-thread-11][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com (auth:KERBEROS)
from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
(UserGroupInformation.java:1430)
[2013-10-24
09:25:02,567][pool-2-thread-9][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com (auth:KERBEROS)
from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
(UserGroupInformation.java:1430)
[2013-10-24
09:25:02,568][pool-2-thread-12][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com (auth:KERBEROS)
from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
(UserGroupInformation.java:1430)
[2013-10-24
09:25:02,570][pool-2-thread-7][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com (auth:KERBEROS)
from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
(UserGroupInformation.java:1430)
[2013-10-24
09:25:02,571][pool-2-thread-14][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com (auth:KERBEROS)
from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
(UserGroupInformation.java:1430)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9831) 'hbasefsck.numthreads' property can not pass to hbck via generic option
[
https://issues.apache.org/jira/browse/HBASE-9831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
takeshi.miao updated HBASE-9831:
Attachment: HBASE-9831.v01.patch
add patch
'hbasefsck.numthreads' property can not pass to hbck via generic option
---
Key: HBASE-9831
URL: https://issues.apache.org/jira/browse/HBASE-9831
Project: HBase
Issue Type: Bug
Components: hbck
Affects Versions: 0.94.12
Reporter: takeshi.miao
Priority: Minor
Labels: hbck
Fix For: 0.94.13
Attachments: HBASE-9831.v01.patch
We use generic option way to pass _'hbasefsck.numthreads'_ property to
_'hbase hbck'_, but it does not accept our new setting value
{code}
hbase hbck -D hbasefsck.numthreads=5
{code}
We can still find there are threads over than 5 we already set via generic
opttion
{code}
[2013-10-24
09:25:02,561][pool-2-thread-6][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,562][pool-2-thread-10][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,565][pool-2-thread-13][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,566][pool-2-thread-11][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,567][pool-2-thread-9][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,568][pool-2-thread-12][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,570][pool-2-thread-7][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,571][pool-2-thread-14][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9831) 'hbasefsck.numthreads' property can not pass to hbck via generic option
[
https://issues.apache.org/jira/browse/HBASE-9831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
takeshi.miao updated HBASE-9831:
Issue Type: Improvement (was: Bug)
'hbasefsck.numthreads' property can not pass to hbck via generic option
---
Key: HBASE-9831
URL: https://issues.apache.org/jira/browse/HBASE-9831
Project: HBase
Issue Type: Improvement
Components: hbck
Affects Versions: 0.94.12
Reporter: takeshi.miao
Priority: Minor
Labels: hbck
Fix For: 0.94.13
Attachments: HBASE-9831.v01.patch
We use generic option way to pass _'hbasefsck.numthreads'_ property to
_'hbase hbck'_, but it does not accept our new setting value
{code}
hbase hbck -D hbasefsck.numthreads=5
{code}
We can still find there are threads over than 5 we already set via generic
opttion
{code}
[2013-10-24
09:25:02,561][pool-2-thread-6][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,562][pool-2-thread-10][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,565][pool-2-thread-13][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,566][pool-2-thread-11][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,567][pool-2-thread-9][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,568][pool-2-thread-12][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,570][pool-2-thread-7][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,571][pool-2-thread-14][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9645) Regionserver halt because of HLog's Logic Error Snapshot seq id from earlier flush still present!
[
https://issues.apache.org/jira/browse/HBASE-9645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804037#comment-13804037
]
Hudson commented on HBASE-9645:
---
SUCCESS: Integrated in HBase-TRUNK #4642 (See
[https://builds.apache.org/job/HBase-TRUNK/4642/])
HBASE-9645 Regionserver halt because of HLogs Logic Error Snapshot seq id from
earlier flush still present (stack: rev 1535288)
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
Regionserver halt because of HLog's Logic Error Snapshot seq id from earlier
flush still present!
---
Key: HBASE-9645
URL: https://issues.apache.org/jira/browse/HBASE-9645
Project: HBase
Issue Type: Bug
Components: regionserver, wal
Affects Versions: 0.94.10
Environment: Linux 2.6.32-el5.x86_64
Reporter: Victor Xu
Priority: Critical
Fix For: 0.98.0
Attachments: 9645v2.txt, HBASE_9645-0.94.10.patch
I upgrade my hbase cluster to 0.94.10 three weeks ago, and this case happened
several days after that. I change the bug's priority to 'Critical' because
every time it happens, a regionserver halt down. All of them have the same
log:
{noformat}
ERROR org.apache.hadoop.hbase.regionserver.wal.HLog: Logic Error Snapshot seq
id from earlier flush still present! for region
c0d88db4ce3606842fbec9d34c38f707 overwritten oldseq=80114270537with new
seq=80115066829
{noformat}
I check the code finding that it locates at HLog.startCacheFlush method. The
'lastSeqWritten' has been locked. Maybe something wrong happened outside the
HLog that change it by mistake.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9830) Backport HBASE-9605 to 0.94
[ https://issues.apache.org/jira/browse/HBASE-9830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chendihao updated HBASE-9830: - Attachment: HBASE-9830-0.94-v1.patch patch for 0.94 Backport HBASE-9605 to 0.94 --- Key: HBASE-9830 URL: https://issues.apache.org/jira/browse/HBASE-9830 Project: HBase Issue Type: Improvement Affects Versions: 0.94.3 Reporter: chendihao Priority: Minor Attachments: HBASE-9830-0.94-v1.patch Backport HBASE-9605 which is about Allow AggregationClient to skip specifying column family for row count aggregate -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9830) Backport HBASE-9605 to 0.94
[ https://issues.apache.org/jira/browse/HBASE-9830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chendihao updated HBASE-9830: - Status: Patch Available (was: Open) Backport HBASE-9605 to 0.94 --- Key: HBASE-9830 URL: https://issues.apache.org/jira/browse/HBASE-9830 Project: HBase Issue Type: Improvement Affects Versions: 0.94.3 Reporter: chendihao Priority: Minor Attachments: HBASE-9830-0.94-v1.patch Backport HBASE-9605 which is about Allow AggregationClient to skip specifying column family for row count aggregate -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HBASE-9832) Add MR support for Visibility labels
ramkrishna.s.vasudevan created HBASE-9832: - Summary: Add MR support for Visibility labels Key: HBASE-9832 URL: https://issues.apache.org/jira/browse/HBASE-9832 Project: HBase Issue Type: Bug Components: mapreduce Affects Versions: 0.98.0 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.98.0 MR needs to support adding the visibility labels through TableOutputFormat and HfileOutPutformat. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9830) Backport HBASE-9605 to 0.94
[
https://issues.apache.org/jira/browse/HBASE-9830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804066#comment-13804066
]
Hadoop QA commented on HBASE-9830:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12610076/HBASE-9830-0.94-v1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7614//console
This message is automatically generated.
Backport HBASE-9605 to 0.94
---
Key: HBASE-9830
URL: https://issues.apache.org/jira/browse/HBASE-9830
Project: HBase
Issue Type: Improvement
Affects Versions: 0.94.3
Reporter: chendihao
Priority: Minor
Attachments: HBASE-9830-0.94-v1.patch
Backport HBASE-9605 which is about Allow AggregationClient to skip
specifying column family for row count aggregate
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9832) Add MR support for Visibility labels
[ https://issues.apache.org/jira/browse/HBASE-9832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-9832: -- Issue Type: Improvement (was: Bug) Add MR support for Visibility labels Key: HBASE-9832 URL: https://issues.apache.org/jira/browse/HBASE-9832 Project: HBase Issue Type: Improvement Components: mapreduce Affects Versions: 0.98.0 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.98.0 MR needs to support adding the visibility labels through TableOutputFormat and HfileOutPutformat. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9830) Backport HBASE-9605 to 0.94
[ https://issues.apache.org/jira/browse/HBASE-9830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804108#comment-13804108 ] chendihao commented on HBASE-9830: -- The patch is for 0.94 so HadoopQA may not handle it. Hope someone review and commit it. Backport HBASE-9605 to 0.94 --- Key: HBASE-9830 URL: https://issues.apache.org/jira/browse/HBASE-9830 Project: HBase Issue Type: Improvement Affects Versions: 0.94.3 Reporter: chendihao Priority: Minor Attachments: HBASE-9830-0.94-v1.patch Backport HBASE-9605 which is about Allow AggregationClient to skip specifying column family for row count aggregate -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9830) Backport HBASE-9605 to 0.94
[ https://issues.apache.org/jira/browse/HBASE-9830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804110#comment-13804110 ] chendihao commented on HBASE-9830: -- [~yuzhih...@gmail.com] Any suggestions? Backport HBASE-9605 to 0.94 --- Key: HBASE-9830 URL: https://issues.apache.org/jira/browse/HBASE-9830 Project: HBase Issue Type: Improvement Affects Versions: 0.94.3 Reporter: chendihao Priority: Minor Attachments: HBASE-9830-0.94-v1.patch Backport HBASE-9605 which is about Allow AggregationClient to skip specifying column family for row count aggregate -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9808) org.apache.hadoop.hbase.rest.PerformanceEvaluation is out of sync with org.apache.hadoop.hbase.PerformanceEvaluation
[
https://issues.apache.org/jira/browse/HBASE-9808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804113#comment-13804113
]
Gustavo Anatoly commented on HBASE-9808:
Ted, I'm still working with the aggregate diff file, because I'm having
problems with malformed patch when applied, on my working copy.
But, I'm doing efforts to finish this issue fast.
org.apache.hadoop.hbase.rest.PerformanceEvaluation is out of sync with
org.apache.hadoop.hbase.PerformanceEvaluation
Key: HBASE-9808
URL: https://issues.apache.org/jira/browse/HBASE-9808
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Gustavo Anatoly
Here is list of JIRAs whose fixes might have gone into
rest.PerformanceEvaluation :
{code}
r1527817 | mbertozzi | 2013-09-30 15:57:44 -0700 (Mon, 30 Sep 2013) | 1 line
HBASE-9663 PerformanceEvaluation does not properly honor specified table name
parameter
r1526452 | mbertozzi | 2013-09-26 04:58:50 -0700 (Thu, 26 Sep 2013) | 1 line
HBASE-9662 PerformanceEvaluation input do not handle tags properties
r1525269 | ramkrishna | 2013-09-21 11:01:32 -0700 (Sat, 21 Sep 2013) | 3 lines
HBASE-8496 - Implement tags and the internals of how a tag should look like
(Ram)
r1524985 | nkeywal | 2013-09-20 06:02:54 -0700 (Fri, 20 Sep 2013) | 1 line
HBASE-9558 PerformanceEvaluation is in hbase-server, and creates a
dependency to MiniDFSCluster
r1523782 | nkeywal | 2013-09-16 13:07:13 -0700 (Mon, 16 Sep 2013) | 1 line
HBASE-9521 clean clearBufferOnFail behavior and deprecate it
r1518341 | jdcryans | 2013-08-28 12:46:55 -0700 (Wed, 28 Aug 2013) | 2 lines
HBASE-9330 Refactor PE to create HTable the correct way
{code}
Long term, we may consider consolidating the two PerformanceEvaluation
classes so that such maintenance work can be reduced.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9827) Intermittent TestLogRollingNoCluster#testContendedLogRolling failure
[
https://issues.apache.org/jira/browse/HBASE-9827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804144#comment-13804144
]
Hudson commented on HBASE-9827:
---
SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #807 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/807/])
HBASE-9827 Intermittent TestLogRollingNoCluster#testContendedLogRolling failure
(jxiang: rev 1535254)
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
Intermittent TestLogRollingNoCluster#testContendedLogRolling failure
Key: HBASE-9827
URL: https://issues.apache.org/jira/browse/HBASE-9827
Project: HBase
Issue Type: Test
Reporter: Ted Yu
Assignee: Jimmy Xiang
Fix For: 0.98.0, 0.96.1
Attachments: trunk-9827.patch
From
https://builds.apache.org/job/HBase-TRUNK/lastCompletedBuild/testReport/org.apache.hadoop.hbase.regionserver.wal/TestLogRollingNoCluster/testContendedLogRolling/
:
{code}
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertFalse(Assert.java:64)
at org.junit.Assert.assertFalse(Assert.java:74)
at
org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster.testContendedLogRolling(TestLogRollingNoCluster.java:79)
...
2013-10-23 00:20:36,872 INFO [18] wal.FSHLog(527): Rolled WAL
/home/jenkins/jenkins-slave/workspace/HBase-TRUNK/trunk/hbase-server/target/test-data/39875b32-a156-48ac-875f-9f67689d3a2e/logs/hlog.1382487636870
with entries=28, filesize=30.0k; new WAL
/home/jenkins/jenkins-slave/workspace/HBase-TRUNK/trunk/hbase-server/target/test-data/39875b32-a156-48ac-875f-9f67689d3a2e/logs/hlog.1382487636872
2013-10-23 00:20:36,873 INFO [51] wal.TestLogRollingNoCluster$Appender(135):
Caught exception from Appender:51
java.io.IOException: cannot get log writer
at
org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createWriter(HLogFactory.java:197)
at
org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createWALWriter(HLogFactory.java:177)
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog.createWriterInstance(FSHLog.java:566)
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:509)
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:470)
at
org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster$Appender.run(TestLogRollingNoCluster.java:118)
Caused by: java.io.IOException: File already
exists:/home/jenkins/jenkins-slave/workspace/HBase-TRUNK/trunk/hbase-server/target/test-data/39875b32-a156-48ac-875f-9f67689d3a2e/logs/hlog.1382487636872
at
org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:249)
at
org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:241)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.init(ChecksumFileSystem.java:335)
at
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:381)
at
org.apache.hadoop.fs.ChecksumFileSystem.createNonRecursive(ChecksumFileSystem.java:395)
at
org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:610)
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.init(ProtobufLogWriter.java:68)
at
org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createWriter(HLogFactory.java:194)
{code}
Looks like
target/test-data/39875b32-a156-48ac-875f-9f67689d3a2e/logs/hlog.1382487636872
was created already and the second FileSystem.createNonRecursive() call
failed.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9827) Intermittent TestLogRollingNoCluster#testContendedLogRolling failure
[
https://issues.apache.org/jira/browse/HBASE-9827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804146#comment-13804146
]
Hudson commented on HBASE-9827:
---
FAILURE: Integrated in hbase-0.96-hadoop2 #99 (See
[https://builds.apache.org/job/hbase-0.96-hadoop2/99/])
HBASE-9827 Intermittent TestLogRollingNoCluster#testContendedLogRolling failure
(jxiang: rev 1535256)
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
Intermittent TestLogRollingNoCluster#testContendedLogRolling failure
Key: HBASE-9827
URL: https://issues.apache.org/jira/browse/HBASE-9827
Project: HBase
Issue Type: Test
Reporter: Ted Yu
Assignee: Jimmy Xiang
Fix For: 0.98.0, 0.96.1
Attachments: trunk-9827.patch
From
https://builds.apache.org/job/HBase-TRUNK/lastCompletedBuild/testReport/org.apache.hadoop.hbase.regionserver.wal/TestLogRollingNoCluster/testContendedLogRolling/
:
{code}
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertFalse(Assert.java:64)
at org.junit.Assert.assertFalse(Assert.java:74)
at
org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster.testContendedLogRolling(TestLogRollingNoCluster.java:79)
...
2013-10-23 00:20:36,872 INFO [18] wal.FSHLog(527): Rolled WAL
/home/jenkins/jenkins-slave/workspace/HBase-TRUNK/trunk/hbase-server/target/test-data/39875b32-a156-48ac-875f-9f67689d3a2e/logs/hlog.1382487636870
with entries=28, filesize=30.0k; new WAL
/home/jenkins/jenkins-slave/workspace/HBase-TRUNK/trunk/hbase-server/target/test-data/39875b32-a156-48ac-875f-9f67689d3a2e/logs/hlog.1382487636872
2013-10-23 00:20:36,873 INFO [51] wal.TestLogRollingNoCluster$Appender(135):
Caught exception from Appender:51
java.io.IOException: cannot get log writer
at
org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createWriter(HLogFactory.java:197)
at
org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createWALWriter(HLogFactory.java:177)
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog.createWriterInstance(FSHLog.java:566)
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:509)
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog.rollWriter(FSHLog.java:470)
at
org.apache.hadoop.hbase.regionserver.wal.TestLogRollingNoCluster$Appender.run(TestLogRollingNoCluster.java:118)
Caused by: java.io.IOException: File already
exists:/home/jenkins/jenkins-slave/workspace/HBase-TRUNK/trunk/hbase-server/target/test-data/39875b32-a156-48ac-875f-9f67689d3a2e/logs/hlog.1382487636872
at
org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:249)
at
org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:241)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.init(ChecksumFileSystem.java:335)
at
org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:381)
at
org.apache.hadoop.fs.ChecksumFileSystem.createNonRecursive(ChecksumFileSystem.java:395)
at
org.apache.hadoop.fs.FileSystem.createNonRecursive(FileSystem.java:610)
at
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.init(ProtobufLogWriter.java:68)
at
org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createWriter(HLogFactory.java:194)
{code}
Looks like
target/test-data/39875b32-a156-48ac-875f-9f67689d3a2e/logs/hlog.1382487636872
was created already and the second FileSystem.createNonRecursive() call
failed.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9645) Regionserver halt because of HLog's Logic Error Snapshot seq id from earlier flush still present!
[
https://issues.apache.org/jira/browse/HBASE-9645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804145#comment-13804145
]
Hudson commented on HBASE-9645:
---
SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #807 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/807/])
HBASE-9645 Regionserver halt because of HLogs Logic Error Snapshot seq id from
earlier flush still present (stack: rev 1535288)
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
Regionserver halt because of HLog's Logic Error Snapshot seq id from earlier
flush still present!
---
Key: HBASE-9645
URL: https://issues.apache.org/jira/browse/HBASE-9645
Project: HBase
Issue Type: Bug
Components: regionserver, wal
Affects Versions: 0.94.10
Environment: Linux 2.6.32-el5.x86_64
Reporter: Victor Xu
Priority: Critical
Fix For: 0.98.0
Attachments: 9645v2.txt, HBASE_9645-0.94.10.patch
I upgrade my hbase cluster to 0.94.10 three weeks ago, and this case happened
several days after that. I change the bug's priority to 'Critical' because
every time it happens, a regionserver halt down. All of them have the same
log:
{noformat}
ERROR org.apache.hadoop.hbase.regionserver.wal.HLog: Logic Error Snapshot seq
id from earlier flush still present! for region
c0d88db4ce3606842fbec9d34c38f707 overwritten oldseq=80114270537with new
seq=80115066829
{noformat}
I check the code finding that it locates at HLog.startCacheFlush method. The
'lastSeqWritten' has been locked. Maybe something wrong happened outside the
HLog that change it by mistake.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9830) Backport HBASE-9605 to 0.94
[ https://issues.apache.org/jira/browse/HBASE-9830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804197#comment-13804197 ] Jean-Marc Spaggiari commented on HBASE-9830: [~lhofhansl] can you have a look? Backport HBASE-9605 to 0.94 --- Key: HBASE-9830 URL: https://issues.apache.org/jira/browse/HBASE-9830 Project: HBase Issue Type: Improvement Affects Versions: 0.94.3 Reporter: chendihao Priority: Minor Attachments: HBASE-9830-0.94-v1.patch Backport HBASE-9605 which is about Allow AggregationClient to skip specifying column family for row count aggregate -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9831) 'hbasefsck.numthreads' property can not pass to hbck via generic option
[
https://issues.apache.org/jira/browse/HBASE-9831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804202#comment-13804202
]
Jean-Marc Spaggiari commented on HBASE-9831:
Have you tried it? Does it works for you?
Is this required?
{code}
executor = null;
{code}
Can you do a trunk version to run hadoopqa?
Looks good to me!
'hbasefsck.numthreads' property can not pass to hbck via generic option
---
Key: HBASE-9831
URL: https://issues.apache.org/jira/browse/HBASE-9831
Project: HBase
Issue Type: Improvement
Components: hbck
Affects Versions: 0.94.12
Reporter: takeshi.miao
Priority: Minor
Labels: hbck
Fix For: 0.94.13
Attachments: HBASE-9831.v01.patch
We use generic option way to pass _'hbasefsck.numthreads'_ property to
_'hbase hbck'_, but it does not accept our new setting value
{code}
hbase hbck -D hbasefsck.numthreads=5
{code}
We can still find there are threads over than 5 we already set via generic
opttion
{code}
[2013-10-24
09:25:02,561][pool-2-thread-6][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,562][pool-2-thread-10][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,565][pool-2-thread-13][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,566][pool-2-thread-11][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,567][pool-2-thread-9][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,568][pool-2-thread-12][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,570][pool-2-thread-7][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,571][pool-2-thread-14][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9806) Add PerfEval tool for BlockCache
[ https://issues.apache.org/jira/browse/HBASE-9806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804219#comment-13804219 ] Jean-Marc Spaggiari commented on HBASE-9806: Sure! I will create a frankenbox and try that. I will have physical access to the computers only Friday evening. So I might be able to configure that this week-end. Also, I found a motherboard where I can put 64GB A bit expensive, but I will most probably buy it so we can test with high memory values and see how it reacts! Donations are welcome ;) Hahaha. I will need some details on what you want me to run 0.96.0 vs 0.96.0+9806 ? Anything else? Specific settings? Add PerfEval tool for BlockCache Key: HBASE-9806 URL: https://issues.apache.org/jira/browse/HBASE-9806 Project: HBase Issue Type: Test Components: Performance, test Reporter: Nick Dimiduk Assignee: Nick Dimiduk Attachments: HBASE-9806.00.patch, HBASE-9806.01.patch We have at least three different block caching layers with myriad configuration settings. Let's add a tool for evaluating memory allocations and configuration combinations with different access patterns. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9543) Impl unique aggregation
[ https://issues.apache.org/jira/browse/HBASE-9543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804220#comment-13804220 ] Jean-Marc Spaggiari commented on HBASE-9543: I also prefer to NOT return partial results. There will be pretty useless and will just use network and resources for nothing. +1 for adding comments, +1 for the method name, and +1 overall with the proposes fixes. Impl unique aggregation Key: HBASE-9543 URL: https://issues.apache.org/jira/browse/HBASE-9543 Project: HBase Issue Type: New Feature Components: Coprocessors Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Attachments: HBASE-9543-0.94-v1.diff, HBASE-9543-trunk-v1.diff, HBASE-9543-trunk-v2.diff, HBASE-9543-trunk-v3.diff, HBASE-9543-trunk-v4.diff Impl unique aggregation: return a set of all columns' values in a scan. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9820) Method Replication.decorateMasterConfiguration can fail with NPE in case that property HBASE_MASTER_LOGCLEANER_PLUGINS is not defined
[
https://issues.apache.org/jira/browse/HBASE-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804241#comment-13804241
]
Jean-Marc Spaggiari commented on HBASE-9820:
Hi [~jarcec]
Are you going to provide a patch for that? Else I can take a look. It's an easy
fix.
Also, do you have the stacktrace?
Thanks.
Method Replication.decorateMasterConfiguration can fail with NPE in case that
property HBASE_MASTER_LOGCLEANER_PLUGINS is not defined
-
Key: HBASE-9820
URL: https://issues.apache.org/jira/browse/HBASE-9820
Project: HBase
Issue Type: Bug
Affects Versions: 0.96.0
Reporter: Jarek Jarcec Cecho
While upgrading HBase dependency on Pig via PIG-3529, I've noticed that
method {{Replication.decorateMasterConfiguration()}} can throw
{{NullPointerException}}
([code|https://github.com/apache/hbase/blob/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java])
in case when property HBASE_MASTER_LOGCLEANER_PLUGINS won't be defined.
The issue was more on a pig side where we weren't propagating default HBase
configuration resources (such as {{hbase-default.xml}}), but I was curious if
it's expected that this property will be always defined? We might want to
tweak the code a bit if this property can be null.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9820) Method Replication.decorateMasterConfiguration can fail with NPE in case that property HBASE_MASTER_LOGCLEANER_PLUGINS is not defined
[
https://issues.apache.org/jira/browse/HBASE-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jean-Marc Spaggiari updated HBASE-9820:
---
Attachment: HBASE-9820-v0-trunk.patch
Very simple patch attached. Just setting a default value for it if not defines
in the hbase-default.xml file.
Method Replication.decorateMasterConfiguration can fail with NPE in case that
property HBASE_MASTER_LOGCLEANER_PLUGINS is not defined
-
Key: HBASE-9820
URL: https://issues.apache.org/jira/browse/HBASE-9820
Project: HBase
Issue Type: Bug
Affects Versions: 0.96.0
Reporter: Jarek Jarcec Cecho
Attachments: HBASE-9820-v0-trunk.patch
While upgrading HBase dependency on Pig via PIG-3529, I've noticed that
method {{Replication.decorateMasterConfiguration()}} can throw
{{NullPointerException}}
([code|https://github.com/apache/hbase/blob/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java])
in case when property HBASE_MASTER_LOGCLEANER_PLUGINS won't be defined.
The issue was more on a pig side where we weren't propagating default HBase
configuration resources (such as {{hbase-default.xml}}), but I was curious if
it's expected that this property will be always defined? We might want to
tweak the code a bit if this property can be null.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9820) Method Replication.decorateMasterConfiguration can fail with NPE in case that property HBASE_MASTER_LOGCLEANER_PLUGINS is not defined
[
https://issues.apache.org/jira/browse/HBASE-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jean-Marc Spaggiari updated HBASE-9820:
---
Assignee: Jean-Marc Spaggiari
Status: Patch Available (was: Open)
Method Replication.decorateMasterConfiguration can fail with NPE in case that
property HBASE_MASTER_LOGCLEANER_PLUGINS is not defined
-
Key: HBASE-9820
URL: https://issues.apache.org/jira/browse/HBASE-9820
Project: HBase
Issue Type: Bug
Affects Versions: 0.96.0
Reporter: Jarek Jarcec Cecho
Assignee: Jean-Marc Spaggiari
Attachments: HBASE-9820-v0-trunk.patch
While upgrading HBase dependency on Pig via PIG-3529, I've noticed that
method {{Replication.decorateMasterConfiguration()}} can throw
{{NullPointerException}}
([code|https://github.com/apache/hbase/blob/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java])
in case when property HBASE_MASTER_LOGCLEANER_PLUGINS won't be defined.
The issue was more on a pig side where we weren't propagating default HBase
configuration resources (such as {{hbase-default.xml}}), but I was curious if
it's expected that this property will be always defined? We might want to
tweak the code a bit if this property can be null.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-4811) Support reverse Scan
[
https://issues.apache.org/jira/browse/HBASE-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804255#comment-13804255
]
Jean-Marc Spaggiari commented on HBASE-4811:
You're welcome.
I can run it again with 32GB, or with a 4 node cluster, or with an 8 cores+32GB
server or... or ... or... ;)
The only limitation are the disks. I have only one SSD, and can not have more
than 6 disks per servers.
All my servers are not running anything now, so all the options are correct.
Just ask and I will run ;)
Support reverse Scan
Key: HBASE-4811
URL: https://issues.apache.org/jira/browse/HBASE-4811
Project: HBase
Issue Type: New Feature
Components: Client
Affects Versions: 0.20.6, 0.94.7
Reporter: John Carrino
Assignee: chunhui shen
Fix For: 0.98.0, 0.94.13, 0.96.1
Attachments: 4811-0.94-v22.txt, 4811-0.94-v3.txt, 4811-trunk-v10.txt,
4811-trunk-v5.patch, HBase-4811-0.94.3modified.txt, hbase-4811-0.94
v21.patch, HBase-4811-0.94-v2.txt, hbase-4811-trunkv11.patch,
hbase-4811-trunkv12.patch, hbase-4811-trunkv13.patch,
hbase-4811-trunkv14.patch, hbase-4811-trunkv15.patch,
hbase-4811-trunkv16.patch, hbase-4811-trunkv17.patch,
hbase-4811-trunkv18.patch, hbase-4811-trunkv19.patch,
hbase-4811-trunkv1.patch, hbase-4811-trunkv20.patch,
hbase-4811-trunkv21.patch, hbase-4811-trunkv4.patch,
hbase-4811-trunkv6.patch, hbase-4811-trunkv7.patch, hbase-4811-trunkv8.patch,
hbase-4811-trunkv9.patch
Reversed scan means scan the rows backward.
And StartRow bigger than StopRow in a reversed scan.
For example, for the following rows:
aaa/c1:q1/value1
aaa/c1:q2/value2
bbb/c1:q1/value1
bbb/c1:q2/value2
ccc/c1:q1/value1
ccc/c1:q2/value2
ddd/c1:q1/value1
ddd/c1:q2/value2
eee/c1:q1/value1
eee/c1:q2/value2
you could do a reversed scan from 'ddd' to 'bbb'(exclude) like this:
Scan scan = new Scan();
scan.setStartRow('ddd');
scan.setStopRow('bbb');
scan.setReversed(true);
for(Result result:htable.getScanner(scan)){
System.out.println(result);
}
Aslo you could do the reversed scan with shell like this:
hbase scan 'table',{REVERSED = true,STARTROW='ddd', STOPROW='bbb'}
And the output is:
ddd/c1:q1/value1
ddd/c1:q2/value2
ccc/c1:q1/value1
ccc/c1:q2/value2
NOTE: when setting reversed as true for a client scan, you must set the start
row, else will throw exception. Through {@link
Scan#createBiggestByteArray(int)},you could get a big enough byte array as
the start row
All the documentation I find about HBase says that if you want forward and
reverse scans you should just build 2 tables and one be ascending and one
descending. Is there a fundamental reason that HBase only supports forward
Scan? It seems like a lot of extra space overhead and coding overhead (to
keep them in sync) to support 2 tables.
I am assuming this has been discussed before, but I can't find the
discussions anywhere about it or why it would be infeasible.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9742) Add Documentation For Simple User Access
[ https://issues.apache.org/jira/browse/HBASE-9742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804314#comment-13804314 ] Jesse Anderson commented on HBASE-9742: --- No, simplewith0_94.diff is a patch for the 0.96 branch. It adds the 0.94 specific configuration docs to the 0.96 branch so everything is together. Add Documentation For Simple User Access Key: HBASE-9742 URL: https://issues.apache.org/jira/browse/HBASE-9742 Project: HBase Issue Type: Improvement Components: documentation Affects Versions: 0.94.12, 0.96.0 Reporter: Jesse Anderson Assignee: Jesse Anderson Fix For: 0.96.0 Attachments: simple0_94.diff, simple.diff, simplewith0_94.diff The [security section|http://hbase.apache.org/book/security.html] in the HBase book only talks about using Kerberos. There is a simple user access mode too that is not documented. This should be documented for development systems and HBase installs where security is not an issue, but they want to prevent user mistakes. I've added a section to the security chapter that talks about simple user access. The new section makes it very clear it is not secure. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-8859) truncate_preserve should get table split keys as it is instead of converting them to string type and then again to bytes
[
https://issues.apache.org/jira/browse/HBASE-8859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804319#comment-13804319
]
stack commented on HBASE-8859:
--
What APIs would we have to add to HTI? Is there enough to make a separate
Interface?
truncate_preserve should get table split keys as it is instead of converting
them to string type and then again to bytes
Key: HBASE-8859
URL: https://issues.apache.org/jira/browse/HBASE-8859
Project: HBase
Issue Type: Bug
Components: scripts
Affects Versions: 0.95.1
Reporter: rajeshbabu
Assignee: rajeshbabu
Fix For: 0.98.0, 0.96.1
Attachments: HBASE-8859-Test_to_reproduce.patch,
HBASE-8859_trunk_2.patch, HBASE-8859_trunk.patch
If we take int,long or double bytes as split keys then we are not creating
table with same split keys because converting them to strings directly and to
bytes is giving different split keys, sometimes getting IllegalArgument
exception because of same split keys(converted). Instead we can get split
keys directly from HTable and pass them while creating table.
{code}
h_table = org.apache.hadoop.hbase.client.HTable.new(conf, table_name)
splits = h_table.getRegionLocations().keys().map{|i| i.getStartKey}
:byte
splits = org.apache.hadoop.hbase.util.Bytes.toByteArrays(splits)
{code}
{code}
Truncating 'emp3' table (it may take a while):
- Disabling table...
- Dropping table...
- Creating table with region boundaries...
ERROR: java.lang.IllegalArgumentException: All split keys must be unique,
found duplicate: B\x11S\xEF\xBF\xBD\xEF\xBF\xBD\xEF\xBF\xBD\x00\x00,
B\x11S\xEF\xBF\xBD\xEF\xBF\xBD\xEF\xBF\xBD\x00\x00
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9831) 'hbasefsck.numthreads' property can not pass to hbck via generic option
[
https://issues.apache.org/jira/browse/HBASE-9831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804321#comment-13804321
]
stack commented on HBASE-9831:
--
What JMS said (down to his +1). When you set exec = 0, it makes me wonder if
it would ever be non-null and if so, should it be shutdown before you make a
new one? But maybe this is an impossible state.
'hbasefsck.numthreads' property can not pass to hbck via generic option
---
Key: HBASE-9831
URL: https://issues.apache.org/jira/browse/HBASE-9831
Project: HBase
Issue Type: Improvement
Components: hbck
Affects Versions: 0.94.12
Reporter: takeshi.miao
Priority: Minor
Labels: hbck
Fix For: 0.94.13
Attachments: HBASE-9831.v01.patch
We use generic option way to pass _'hbasefsck.numthreads'_ property to
_'hbase hbck'_, but it does not accept our new setting value
{code}
hbase hbck -D hbasefsck.numthreads=5
{code}
We can still find there are threads over than 5 we already set via generic
opttion
{code}
[2013-10-24
09:25:02,561][pool-2-thread-6][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,562][pool-2-thread-10][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,565][pool-2-thread-13][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,566][pool-2-thread-11][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,567][pool-2-thread-9][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,568][pool-2-thread-12][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,570][pool-2-thread-7][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,571][pool-2-thread-14][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9820) Method Replication.decorateMasterConfiguration can fail with NPE in case that property HBASE_MASTER_LOGCLEANER_PLUGINS is not defined
[
https://issues.apache.org/jira/browse/HBASE-9820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804338#comment-13804338
]
Hadoop QA commented on HBASE-9820:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12610091/HBASE-9820-v0-trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7615//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7615//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7615//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7615//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7615//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7615//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7615//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7615//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7615//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7615//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7615//console
This message is automatically generated.
Method Replication.decorateMasterConfiguration can fail with NPE in case that
property HBASE_MASTER_LOGCLEANER_PLUGINS is not defined
-
Key: HBASE-9820
URL: https://issues.apache.org/jira/browse/HBASE-9820
Project: HBase
Issue Type: Bug
Affects Versions: 0.96.0
Reporter: Jarek Jarcec Cecho
Assignee: Jean-Marc Spaggiari
Attachments: HBASE-9820-v0-trunk.patch
While upgrading HBase dependency on Pig via PIG-3529, I've noticed that
method {{Replication.decorateMasterConfiguration()}} can throw
{{NullPointerException}}
([code|https://github.com/apache/hbase/blob/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java])
in case when property HBASE_MASTER_LOGCLEANER_PLUGINS won't be defined.
The issue was more on a pig side where we weren't propagating default HBase
configuration resources (such as {{hbase-default.xml}}), but I was curious if
it's expected that this property will be always defined? We might want to
tweak the code a bit if this property can be null.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9830) Backport HBASE-9605 to 0.94
[ https://issues.apache.org/jira/browse/HBASE-9830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804357#comment-13804357 ] Ted Yu commented on HBASE-9830: --- TestAggregateProtocol passes with patch. +1 Backport HBASE-9605 to 0.94 --- Key: HBASE-9830 URL: https://issues.apache.org/jira/browse/HBASE-9830 Project: HBase Issue Type: Improvement Affects Versions: 0.94.3 Reporter: chendihao Priority: Minor Attachments: HBASE-9830-0.94-v1.patch Backport HBASE-9605 which is about Allow AggregationClient to skip specifying column family for row count aggregate -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9769) Improve performance of a Scanner with explicit column list when rows are small/medium size
[ https://issues.apache.org/jira/browse/HBASE-9769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804375#comment-13804375 ] Lars Hofhansl commented on HBASE-9769: -- Let's do a new issue. HBASE-5987 is for reseek only as we know we only scan forward in that case. So it looks like HBASE-5987 is working as expected for reseeks. Improve performance of a Scanner with explicit column list when rows are small/medium size -- Key: HBASE-9769 URL: https://issues.apache.org/jira/browse/HBASE-9769 Project: HBase Issue Type: Improvement Components: Scanners Affects Versions: 0.98.0, 0.94.12, 0.96.0 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Attachments: 9769-0.94-sample1.txt, 9769-0.94-sample2.txt, 9769-0.94-sample.txt, 9769-94.txt, 9769-94-v2.txt, 9769-trunk-v1.txt, 9769-trunk-v2.txt, 9769-trunk-v3.txt, 9769-trunk-v4.txt -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9272) A parallel, unordered scanner
[ https://issues.apache.org/jira/browse/HBASE-9272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804376#comment-13804376 ] Lars Hofhansl commented on HBASE-9272: -- Lemme look into this. I have been quite busy the past few days. A parallel, unordered scanner - Key: HBASE-9272 URL: https://issues.apache.org/jira/browse/HBASE-9272 Project: HBase Issue Type: New Feature Reporter: Lars Hofhansl Assignee: Lars Hofhansl Priority: Minor Fix For: 0.98.0, 0.94.13, 0.96.1 Attachments: 9272-0.94.txt, 9272-0.94-v2.txt, 9272-0.94-v3.txt, 9272-0.94-v4.txt, 9272-trunk.txt, 9272-trunk-v2.txt, 9272-trunk-v3.txt, 9272-trunk-v3.txt, ParallelClientScanner.java, ParallelClientScanner.java The contract of ClientScanner is to return rows in sort order. That limits the order in which region can be scanned. I propose a simple ParallelScanner that does not have this requirement and queries regions in parallel, return whatever gets returned first. This is generally useful for scans that filter a lot of data on the server, or in cases where the client can very quickly react to the returned data. I have a simple prototype (doesn't do error handling right, and might be a bit heavy on the synchronization side - it used a BlockingQueue to hand data between the client using the scanner and the threads doing the scanning, it also could potentially starve some scanners long enugh to time out at the server). On the plus side, it's only a 130 lines of code. :) -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9819) Backport HBASE-8372 'Provide mutability to CompoundConfiguration' to 0.94
[
https://issues.apache.org/jira/browse/HBASE-9819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804380#comment-13804380
]
Lars Hofhansl commented on HBASE-9819:
--
+1
Backport HBASE-8372 'Provide mutability to CompoundConfiguration' to 0.94
-
Key: HBASE-9819
URL: https://issues.apache.org/jira/browse/HBASE-9819
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 0.94.14
Attachments: 9819.txt
In the email thread: http://search-hadoop.com/m/dcqod1uy32h
yonghu encountered the following exception when he tried to retrieve
HTableInterface:
{code}
ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
Failed 1 action: org.apache.hadoop.hbase.DoNotRetryIOException:
Coprocessor:
'org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionEnvironment@9a99eb'
threw: 'java.lang.UnsupportedOperationException: Immutable Configuration'
and has been removedfrom the active coprocessor set.
at
org.apache.hadoop.hbase.coprocessor.CoprocessorHost.handleCoprocessorThrowable(CoprocessorHost.java:740)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.prePut(RegionCoprocessorHost.java:810)
at
org.apache.hadoop.hbase.regionserver.HRegion.doPreMutationHook(HRegion.java:2196)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2172)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3811)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
Caused by: java.lang.UnsupportedOperationException: Immutable Configuration
at
org.apache.hadoop.hbase.regionserver.CompoundConfiguration.set(CompoundConfiguration.java:484)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.ensureZookeeperTrackers(HConnectionManager.java:721)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:986)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:961)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:251)
at org.apache.hadoop.hbase.client.HTable.init(HTable.java:243)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getTable(HConnectionManager.java:671)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getTable(HConnectionManager.java:658)
at
CDCTrigger.TriggerForModification.prePut(TriggerForModification.java:61)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.prePut(RegionCoprocessorHost.java:808)
... 9 more
: 1 time, servers with issues: hans-laptop:60020
{code}
CompoundConfiguration is mutable in 0.96 and beyond.
This should be backported to 0.94
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-8741) Scope sequenceid to the region rather than regionserver (WAS: Mutations on Regions in recovery mode might have same sequenceIDs)
[ https://issues.apache.org/jira/browse/HBASE-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-8741: --- Attachment: (was: HBASE-8741-trunk-v6.1.patch) Scope sequenceid to the region rather than regionserver (WAS: Mutations on Regions in recovery mode might have same sequenceIDs) Key: HBASE-8741 URL: https://issues.apache.org/jira/browse/HBASE-8741 Project: HBase Issue Type: Bug Components: MTTR Affects Versions: 0.95.1 Reporter: Himanshu Vashishtha Assignee: Himanshu Vashishtha Fix For: 0.98.0 Attachments: HBASE-8741-trunk-v6.patch, HBASE-8741-v0.patch, HBASE-8741-v2.patch, HBASE-8741-v3.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4.patch, HBASE-8741-v5-again.patch, HBASE-8741-v5.patch Currently, when opening a region, we find the maximum sequence ID from all its HFiles and then set the LogSequenceId of the log (in case the later is at a small value). This works good in recovered.edits case as we are not writing to the region until we have replayed all of its previous edits. With distributed log replay, if we want to enable writes while a region is under recovery, we need to make sure that the logSequenceId maximum logSequenceId of the old regionserver. Otherwise, we might have a situation where new edits have same (or smaller) sequenceIds. We can store region level information in the WALTrailer, than this scenario could be avoided by: a) reading the trailer of the last completed file, i.e., last wal file which has a trailer and, b) completely reading the last wal file (this file would not have the trailer, so it needs to be read completely). In future, if we switch to multi wal file, we could read the trailer for all completed WAL files, and reading the remaining incomplete files. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-7404) Bucket Cache:A solution about CMS,Heap Fragment and Big Cache on HBASE
[ https://issues.apache.org/jira/browse/HBASE-7404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804386#comment-13804386 ] Lars Hofhansl commented on HBASE-7404: -- Was waiting for HBASE-8894, but it seems that has some ways to go. The problem with this patch is that when you enable the off-heap bucketcache the blockcache is disabled for all but the index blocks. There is no in-between. So from that angle it is not very useful, because most folks couldn't just switch it on, since it is quite a bit slower than the blockcache. If it would be configurable per table/cf that would be another story. One could then have smaller, hot table still use the blockcache and have larger not-so-tables use the offheap cache; and thus we'd be able to make use of RAM sizes of 128gb or more. At the same time, since this is all new code, I'm fine with reviewing Dave's backport a bit more and then committing it. Comments? Bucket Cache:A solution about CMS,Heap Fragment and Big Cache on HBASE -- Key: HBASE-7404 URL: https://issues.apache.org/jira/browse/HBASE-7404 Project: HBase Issue Type: New Feature Affects Versions: 0.94.3 Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.95.0 Attachments: 7404-0.94-fixed-lines.txt, 7404-trunk-v10.patch, 7404-trunk-v11.patch, 7404-trunk-v12.patch, 7404-trunk-v13.patch, 7404-trunk-v13.txt, 7404-trunk-v14.patch, BucketCache.pdf, hbase-7404-94v2.patch, HBASE-7404-backport-0.94.patch, hbase-7404-trunkv2.patch, hbase-7404-trunkv9.patch, Introduction of Bucket Cache.pdf First, thanks @neil from Fusion-IO share the source code. Usage: 1.Use bucket cache as main memory cache, configured as the following: –hbase.bucketcache.ioengine heap –hbase.bucketcache.size 0.4 (size for bucket cache, 0.4 is a percentage of max heap size) 2.Use bucket cache as a secondary cache, configured as the following: –hbase.bucketcache.ioengine file:/disk1/hbase/cache.data(The file path where to store the block data) –hbase.bucketcache.size 1024 (size for bucket cache, unit is MB, so 1024 means 1GB) –hbase.bucketcache.combinedcache.enabled false (default value being true) See more configurations from org.apache.hadoop.hbase.io.hfile.CacheConfig and org.apache.hadoop.hbase.io.hfile.bucket.BucketCache What's Bucket Cache? It could greatly decrease CMS and heap fragment by GC It support a large cache space for High Read Performance by using high speed disk like Fusion-io 1.An implementation of block cache like LruBlockCache 2.Self manage blocks' storage position through Bucket Allocator 3.The cached blocks could be stored in the memory or file system 4.Bucket Cache could be used as a mainly block cache(see CombinedBlockCache), combined with LruBlockCache to decrease CMS and fragment by GC. 5.BucketCache also could be used as a secondary cache(e.g. using Fusionio to store block) to enlarge cache space How about SlabCache? We have studied and test SlabCache first, but the result is bad, because: 1.SlabCache use SingleSizeCache, its use ratio of memory is low because kinds of block size, especially using DataBlockEncoding 2.SlabCache is uesd in DoubleBlockCache, block is cached both in SlabCache and LruBlockCache, put the block to LruBlockCache again if hit in SlabCache , it causes CMS and heap fragment don't get any better 3.Direct heap performance is not good as heap, and maybe cause OOM, so we recommend using heap engine See more in the attachment and in the patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-8741) Scope sequenceid to the region rather than regionserver (WAS: Mutations on Regions in recovery mode might have same sequenceIDs)
[ https://issues.apache.org/jira/browse/HBASE-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-8741: --- Attachment: HBASE-8741-trunk-v6.patch attaching previous patch as 9785 is in. Scope sequenceid to the region rather than regionserver (WAS: Mutations on Regions in recovery mode might have same sequenceIDs) Key: HBASE-8741 URL: https://issues.apache.org/jira/browse/HBASE-8741 Project: HBase Issue Type: Bug Components: MTTR Affects Versions: 0.95.1 Reporter: Himanshu Vashishtha Assignee: Himanshu Vashishtha Fix For: 0.98.0 Attachments: HBASE-8741-trunk-v6.patch, HBASE-8741-trunk-v6.patch, HBASE-8741-v0.patch, HBASE-8741-v2.patch, HBASE-8741-v3.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4.patch, HBASE-8741-v5-again.patch, HBASE-8741-v5.patch Currently, when opening a region, we find the maximum sequence ID from all its HFiles and then set the LogSequenceId of the log (in case the later is at a small value). This works good in recovered.edits case as we are not writing to the region until we have replayed all of its previous edits. With distributed log replay, if we want to enable writes while a region is under recovery, we need to make sure that the logSequenceId maximum logSequenceId of the old regionserver. Otherwise, we might have a situation where new edits have same (or smaller) sequenceIds. We can store region level information in the WALTrailer, than this scenario could be avoided by: a) reading the trailer of the last completed file, i.e., last wal file which has a trailer and, b) completely reading the last wal file (this file would not have the trailer, so it needs to be read completely). In future, if we switch to multi wal file, we could read the trailer for all completed WAL files, and reading the remaining incomplete files. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9808) org.apache.hadoop.hbase.rest.PerformanceEvaluation is out of sync with org.apache.hadoop.hbase.PerformanceEvaluation
[
https://issues.apache.org/jira/browse/HBASE-9808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804388#comment-13804388
]
Ted Yu commented on HBASE-9808:
---
Thanks a lot, Gustavo.
org.apache.hadoop.hbase.rest.PerformanceEvaluation is out of sync with
org.apache.hadoop.hbase.PerformanceEvaluation
Key: HBASE-9808
URL: https://issues.apache.org/jira/browse/HBASE-9808
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Gustavo Anatoly
Here is list of JIRAs whose fixes might have gone into
rest.PerformanceEvaluation :
{code}
r1527817 | mbertozzi | 2013-09-30 15:57:44 -0700 (Mon, 30 Sep 2013) | 1 line
HBASE-9663 PerformanceEvaluation does not properly honor specified table name
parameter
r1526452 | mbertozzi | 2013-09-26 04:58:50 -0700 (Thu, 26 Sep 2013) | 1 line
HBASE-9662 PerformanceEvaluation input do not handle tags properties
r1525269 | ramkrishna | 2013-09-21 11:01:32 -0700 (Sat, 21 Sep 2013) | 3 lines
HBASE-8496 - Implement tags and the internals of how a tag should look like
(Ram)
r1524985 | nkeywal | 2013-09-20 06:02:54 -0700 (Fri, 20 Sep 2013) | 1 line
HBASE-9558 PerformanceEvaluation is in hbase-server, and creates a
dependency to MiniDFSCluster
r1523782 | nkeywal | 2013-09-16 13:07:13 -0700 (Mon, 16 Sep 2013) | 1 line
HBASE-9521 clean clearBufferOnFail behavior and deprecate it
r1518341 | jdcryans | 2013-08-28 12:46:55 -0700 (Wed, 28 Aug 2013) | 2 lines
HBASE-9330 Refactor PE to create HTable the correct way
{code}
Long term, we may consider consolidating the two PerformanceEvaluation
classes so that such maintenance work can be reduced.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-8741) Scope sequenceid to the region rather than regionserver (WAS: Mutations on Regions in recovery mode might have same sequenceIDs)
[ https://issues.apache.org/jira/browse/HBASE-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-8741: --- Attachment: HBASE-8741-trunk-v6.1-rebased.patch rebased, as there were some changes in FSHLog in trunk Scope sequenceid to the region rather than regionserver (WAS: Mutations on Regions in recovery mode might have same sequenceIDs) Key: HBASE-8741 URL: https://issues.apache.org/jira/browse/HBASE-8741 Project: HBase Issue Type: Bug Components: MTTR Affects Versions: 0.95.1 Reporter: Himanshu Vashishtha Assignee: Himanshu Vashishtha Fix For: 0.98.0 Attachments: HBASE-8741-trunk-v6.1-rebased.patch, HBASE-8741-trunk-v6.patch, HBASE-8741-v0.patch, HBASE-8741-v2.patch, HBASE-8741-v3.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4.patch, HBASE-8741-v5-again.patch, HBASE-8741-v5.patch Currently, when opening a region, we find the maximum sequence ID from all its HFiles and then set the LogSequenceId of the log (in case the later is at a small value). This works good in recovered.edits case as we are not writing to the region until we have replayed all of its previous edits. With distributed log replay, if we want to enable writes while a region is under recovery, we need to make sure that the logSequenceId maximum logSequenceId of the old regionserver. Otherwise, we might have a situation where new edits have same (or smaller) sequenceIds. We can store region level information in the WALTrailer, than this scenario could be avoided by: a) reading the trailer of the last completed file, i.e., last wal file which has a trailer and, b) completely reading the last wal file (this file would not have the trailer, so it needs to be read completely). In future, if we switch to multi wal file, we could read the trailer for all completed WAL files, and reading the remaining incomplete files. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9272) A parallel, unordered scanner
[ https://issues.apache.org/jira/browse/HBASE-9272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804401#comment-13804401 ] Lars Hofhansl commented on HBASE-9272: -- bq. TestRegionObserverScannerOpenHook I've seen this in other test runs, so this is probably not related. A parallel, unordered scanner - Key: HBASE-9272 URL: https://issues.apache.org/jira/browse/HBASE-9272 Project: HBase Issue Type: New Feature Reporter: Lars Hofhansl Assignee: Lars Hofhansl Priority: Minor Fix For: 0.98.0, 0.94.13, 0.96.1 Attachments: 9272-0.94.txt, 9272-0.94-v2.txt, 9272-0.94-v3.txt, 9272-0.94-v4.txt, 9272-trunk.txt, 9272-trunk-v2.txt, 9272-trunk-v3.txt, 9272-trunk-v3.txt, ParallelClientScanner.java, ParallelClientScanner.java The contract of ClientScanner is to return rows in sort order. That limits the order in which region can be scanned. I propose a simple ParallelScanner that does not have this requirement and queries regions in parallel, return whatever gets returned first. This is generally useful for scans that filter a lot of data on the server, or in cases where the client can very quickly react to the returned data. I have a simple prototype (doesn't do error handling right, and might be a bit heavy on the synchronization side - it used a BlockingQueue to hand data between the client using the scanner and the threads doing the scanning, it also could potentially starve some scanners long enugh to time out at the server). On the plus side, it's only a 130 lines of code. :) -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-8741) Scope sequenceid to the region rather than regionserver (WAS: Mutations on Regions in recovery mode might have same sequenceIDs)
[ https://issues.apache.org/jira/browse/HBASE-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-8741: --- Attachment: (was: HBASE-8741-trunk-v6.patch) Scope sequenceid to the region rather than regionserver (WAS: Mutations on Regions in recovery mode might have same sequenceIDs) Key: HBASE-8741 URL: https://issues.apache.org/jira/browse/HBASE-8741 Project: HBase Issue Type: Bug Components: MTTR Affects Versions: 0.95.1 Reporter: Himanshu Vashishtha Assignee: Himanshu Vashishtha Fix For: 0.98.0 Attachments: HBASE-8741-trunk-v6.1-rebased.patch, HBASE-8741-trunk-v6.patch, HBASE-8741-v0.patch, HBASE-8741-v2.patch, HBASE-8741-v3.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4.patch, HBASE-8741-v5-again.patch, HBASE-8741-v5.patch Currently, when opening a region, we find the maximum sequence ID from all its HFiles and then set the LogSequenceId of the log (in case the later is at a small value). This works good in recovered.edits case as we are not writing to the region until we have replayed all of its previous edits. With distributed log replay, if we want to enable writes while a region is under recovery, we need to make sure that the logSequenceId maximum logSequenceId of the old regionserver. Otherwise, we might have a situation where new edits have same (or smaller) sequenceIds. We can store region level information in the WALTrailer, than this scenario could be avoided by: a) reading the trailer of the last completed file, i.e., last wal file which has a trailer and, b) completely reading the last wal file (this file would not have the trailer, so it needs to be read completely). In future, if we switch to multi wal file, we could read the trailer for all completed WAL files, and reading the remaining incomplete files. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9831) 'hbasefsck.numthreads' property can not pass to hbck via generic option
[
https://issues.apache.org/jira/browse/HBASE-9831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804403#comment-13804403
]
takeshi.miao commented on HBASE-9831:
-
@JMS @stack
Tks for your review and commtents, my reply as follows...
bq.Have you tried it? Does it works for you?
Yes, I've tested it in our env. and works well
{quote}
Is this required?
{code}
executor = null;
{code}
{quote}
My intention was to set _executor_ explicitly if the call path is _main -
HbaseFsck - initialPoolNumThreads - run - initialPoolNumThreads_, despite it
should be no hurt if I remove this statement.
{quote}
What JMS said (down to his +1). When you set exec = 0, it makes me wonder if it
would ever be non-null and if so, should it be shutdown before you make a new
one? But maybe this is an impossible state.
{quote}
Tks for stack's suggestion !! this way is more safer if any unexpected call
happens. So I changed the code snippet into following style. How do you think
about this ?
{code}
288 if (executor != null) {
289 executor.shutdown();
290 executor = null;
291 }
{code}
bq.Can you do a trunk version to run hadoopqa?
No problem, I can do it later
'hbasefsck.numthreads' property can not pass to hbck via generic option
---
Key: HBASE-9831
URL: https://issues.apache.org/jira/browse/HBASE-9831
Project: HBase
Issue Type: Improvement
Components: hbck
Affects Versions: 0.94.12
Reporter: takeshi.miao
Priority: Minor
Labels: hbck
Fix For: 0.94.13
Attachments: HBASE-9831.v01.patch
We use generic option way to pass _'hbasefsck.numthreads'_ property to
_'hbase hbck'_, but it does not accept our new setting value
{code}
hbase hbck -D hbasefsck.numthreads=5
{code}
We can still find there are threads over than 5 we already set via generic
opttion
{code}
[2013-10-24
09:25:02,561][pool-2-thread-6][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,562][pool-2-thread-10][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,565][pool-2-thread-13][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,566][pool-2-thread-11][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,567][pool-2-thread-9][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,568][pool-2-thread-12][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,570][pool-2-thread-7][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,571][pool-2-thread-14][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9831) 'hbasefsck.numthreads' property can not pass to hbck via generic option
[
https://issues.apache.org/jira/browse/HBASE-9831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
takeshi.miao updated HBASE-9831:
Status: Patch Available (was: Open)
'hbasefsck.numthreads' property can not pass to hbck via generic option
---
Key: HBASE-9831
URL: https://issues.apache.org/jira/browse/HBASE-9831
Project: HBase
Issue Type: Improvement
Components: hbck
Affects Versions: 0.94.12
Reporter: takeshi.miao
Priority: Minor
Labels: hbck
Fix For: 0.94.13
Attachments: HBASE-9831-0.94-v02.patch, HBASE-9831.v01.patch
We use generic option way to pass _'hbasefsck.numthreads'_ property to
_'hbase hbck'_, but it does not accept our new setting value
{code}
hbase hbck -D hbasefsck.numthreads=5
{code}
We can still find there are threads over than 5 we already set via generic
opttion
{code}
[2013-10-24
09:25:02,561][pool-2-thread-6][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,562][pool-2-thread-10][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,565][pool-2-thread-13][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,566][pool-2-thread-11][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,567][pool-2-thread-9][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,568][pool-2-thread-12][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,570][pool-2-thread-7][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,571][pool-2-thread-14][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9831) 'hbasefsck.numthreads' property can not pass to hbck via generic option
[
https://issues.apache.org/jira/browse/HBASE-9831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
takeshi.miao updated HBASE-9831:
Attachment: HBASE-9831-0.94-v02.patch
Added 0.94-v02.patch
'hbasefsck.numthreads' property can not pass to hbck via generic option
---
Key: HBASE-9831
URL: https://issues.apache.org/jira/browse/HBASE-9831
Project: HBase
Issue Type: Improvement
Components: hbck
Affects Versions: 0.94.12
Reporter: takeshi.miao
Priority: Minor
Labels: hbck
Fix For: 0.94.13
Attachments: HBASE-9831-0.94-v02.patch, HBASE-9831.v01.patch
We use generic option way to pass _'hbasefsck.numthreads'_ property to
_'hbase hbck'_, but it does not accept our new setting value
{code}
hbase hbck -D hbasefsck.numthreads=5
{code}
We can still find there are threads over than 5 we already set via generic
opttion
{code}
[2013-10-24
09:25:02,561][pool-2-thread-6][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,562][pool-2-thread-10][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,565][pool-2-thread-13][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,566][pool-2-thread-11][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,567][pool-2-thread-9][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,568][pool-2-thread-12][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,570][pool-2-thread-7][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,571][pool-2-thread-14][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9831) 'hbasefsck.numthreads' property can not pass to hbck via generic option
[
https://issues.apache.org/jira/browse/HBASE-9831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804410#comment-13804410
]
takeshi.miao commented on HBASE-9831:
-
I changed the status to 'patch available' for triggering the hadoopqa for 0.94,
and apply the trunk version later
'hbasefsck.numthreads' property can not pass to hbck via generic option
---
Key: HBASE-9831
URL: https://issues.apache.org/jira/browse/HBASE-9831
Project: HBase
Issue Type: Improvement
Components: hbck
Affects Versions: 0.94.12
Reporter: takeshi.miao
Priority: Minor
Labels: hbck
Fix For: 0.94.13
Attachments: HBASE-9831-0.94-v02.patch, HBASE-9831.v01.patch
We use generic option way to pass _'hbasefsck.numthreads'_ property to
_'hbase hbck'_, but it does not accept our new setting value
{code}
hbase hbck -D hbasefsck.numthreads=5
{code}
We can still find there are threads over than 5 we already set via generic
opttion
{code}
[2013-10-24
09:25:02,561][pool-2-thread-6][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,562][pool-2-thread-10][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,565][pool-2-thread-13][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,566][pool-2-thread-11][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,567][pool-2-thread-9][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,568][pool-2-thread-12][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,570][pool-2-thread-7][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,571][pool-2-thread-14][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9825) LoadIncrementalHFiles fails to load from remote cluster in hadoop 2
[
https://issues.apache.org/jira/browse/HBASE-9825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804415#comment-13804415
]
Ted Yu commented on HBASE-9825:
---
HBASE-9819 has been integrated to 0.94
Mind verifying whether this is fixed ?
LoadIncrementalHFiles fails to load from remote cluster in hadoop 2
---
Key: HBASE-9825
URL: https://issues.apache.org/jira/browse/HBASE-9825
Project: HBase
Issue Type: Bug
Components: hadoop2
Affects Versions: 0.94.12
Reporter: Jerry He
Attachments: HBASE-9825-0.94-only.patch
Running on hadoop 2, LoadIncrementalHFiles gives the following exception when
loading from a remote cluster.
{code}
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$3@455e455e,
java.io.IOException: j ava.io.IOException:
java.lang.UnsupportedOperationException: Immutable Configuration
at
org.apache.hadoop.hbase.regionserver.CompoundConfiguration.setClass(CompoundConfiguration.java:516)
at org.apache.hadoop.ipc.RPC.setProtocolEngine(RPC.java:195)
at
org.apache.hadoop.hdfs.NameNodeProxies.createNNProxyWithClientProtocol(NameNodeProxies.java:250)
at
org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:169)
at
org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:130)
at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:482)
at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:445)
at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:136)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2429)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2463)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2445)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:363)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:283)
at
org.apache.hadoop.hbase.regionserver.Store.assertBulkLoadHFileOk(Store.java:571)
at
org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:3689)
at
org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:3637)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.bulkLoadHFiles(HRegionServer.java:2939)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
at
org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:186)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.tryAtomicRegionLoad(LoadIncrementalHFiles.java:567)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$1.call(LoadIncrementalHFiles.java:317)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$1.call(LoadIncrementalHFiles.java:315)
{code}
This does not happen when loading from the same FileSystem.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9819) Backport HBASE-8372 'Provide mutability to CompoundConfiguration' to 0.94
[
https://issues.apache.org/jira/browse/HBASE-9819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804413#comment-13804413
]
Ted Yu commented on HBASE-9819:
---
Integrated to 0.94
Thanks for the review.
Backport HBASE-8372 'Provide mutability to CompoundConfiguration' to 0.94
-
Key: HBASE-9819
URL: https://issues.apache.org/jira/browse/HBASE-9819
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 0.94.14
Attachments: 9819.txt
In the email thread: http://search-hadoop.com/m/dcqod1uy32h
yonghu encountered the following exception when he tried to retrieve
HTableInterface:
{code}
ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
Failed 1 action: org.apache.hadoop.hbase.DoNotRetryIOException:
Coprocessor:
'org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionEnvironment@9a99eb'
threw: 'java.lang.UnsupportedOperationException: Immutable Configuration'
and has been removedfrom the active coprocessor set.
at
org.apache.hadoop.hbase.coprocessor.CoprocessorHost.handleCoprocessorThrowable(CoprocessorHost.java:740)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.prePut(RegionCoprocessorHost.java:810)
at
org.apache.hadoop.hbase.regionserver.HRegion.doPreMutationHook(HRegion.java:2196)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2172)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3811)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
Caused by: java.lang.UnsupportedOperationException: Immutable Configuration
at
org.apache.hadoop.hbase.regionserver.CompoundConfiguration.set(CompoundConfiguration.java:484)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.ensureZookeeperTrackers(HConnectionManager.java:721)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:986)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:961)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:251)
at org.apache.hadoop.hbase.client.HTable.init(HTable.java:243)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getTable(HConnectionManager.java:671)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getTable(HConnectionManager.java:658)
at
CDCTrigger.TriggerForModification.prePut(TriggerForModification.java:61)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.prePut(RegionCoprocessorHost.java:808)
... 9 more
: 1 time, servers with issues: hans-laptop:60020
{code}
CompoundConfiguration is mutable in 0.96 and beyond.
This should be backported to 0.94
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9831) 'hbasefsck.numthreads' property can not pass to hbck via generic option
[
https://issues.apache.org/jira/browse/HBASE-9831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804420#comment-13804420
]
Hadoop QA commented on HBASE-9831:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12610112/HBASE-9831-0.94-v02.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7617//console
This message is automatically generated.
'hbasefsck.numthreads' property can not pass to hbck via generic option
---
Key: HBASE-9831
URL: https://issues.apache.org/jira/browse/HBASE-9831
Project: HBase
Issue Type: Improvement
Components: hbck
Affects Versions: 0.94.12
Reporter: takeshi.miao
Priority: Minor
Labels: hbck
Fix For: 0.94.13
Attachments: HBASE-9831-0.94-v02.patch, HBASE-9831.v01.patch
We use generic option way to pass _'hbasefsck.numthreads'_ property to
_'hbase hbck'_, but it does not accept our new setting value
{code}
hbase hbck -D hbasefsck.numthreads=5
{code}
We can still find there are threads over than 5 we already set via generic
opttion
{code}
[2013-10-24
09:25:02,561][pool-2-thread-6][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,562][pool-2-thread-10][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,565][pool-2-thread-13][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,566][pool-2-thread-11][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,567][pool-2-thread-9][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,568][pool-2-thread-12][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,570][pool-2-thread-7][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,571][pool-2-thread-14][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9831) 'hbasefsck.numthreads' property can not pass to hbck via generic option
[
https://issues.apache.org/jira/browse/HBASE-9831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804422#comment-13804422
]
Jean-Marc Spaggiari commented on HBASE-9831:
HadooQA doesn't work with 0.94 ;)
{code}
if (executor != null) {
executor.shutdown();
executor = null;
}
{code}
I'm still not sure about the executor = null; ;) You set it to null then
right after instantiate it. So might not be required?
'hbasefsck.numthreads' property can not pass to hbck via generic option
---
Key: HBASE-9831
URL: https://issues.apache.org/jira/browse/HBASE-9831
Project: HBase
Issue Type: Improvement
Components: hbck
Affects Versions: 0.94.12
Reporter: takeshi.miao
Priority: Minor
Labels: hbck
Fix For: 0.94.13
Attachments: HBASE-9831-0.94-v02.patch, HBASE-9831.v01.patch
We use generic option way to pass _'hbasefsck.numthreads'_ property to
_'hbase hbck'_, but it does not accept our new setting value
{code}
hbase hbck -D hbasefsck.numthreads=5
{code}
We can still find there are threads over than 5 we already set via generic
opttion
{code}
[2013-10-24
09:25:02,561][pool-2-thread-6][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,562][pool-2-thread-10][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,565][pool-2-thread-13][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,566][pool-2-thread-11][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,567][pool-2-thread-9][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,568][pool-2-thread-12][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,570][pool-2-thread-7][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,571][pool-2-thread-14][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Assigned] (HBASE-8691) High-Throughput Streaming Scan API
[ https://issues.apache.org/jira/browse/HBASE-8691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reassigned HBASE-8691: - Assignee: Sandy Pratt High-Throughput Streaming Scan API -- Key: HBASE-8691 URL: https://issues.apache.org/jira/browse/HBASE-8691 Project: HBase Issue Type: Improvement Components: Scanners Affects Versions: 0.95.0 Reporter: Sandy Pratt Assignee: Sandy Pratt Labels: perfomance, scan Attachments: HRegionServlet.java, README.txt, RecordReceiver.java, ScannerTest.java, StreamHRegionServer.java, StreamReceiverDirect.java, StreamServletDirect.java I've done some working testing various ways to refactor and optimize Scans in HBase, and have found that performance can be dramatically increased by the addition of a streaming scan API. The attached code constitutes a proof of concept that shows performance increases of almost 4x in some workloads. I'd appreciate testing, replication, and comments. If the approach seems viable, I think such an API should be built into some future version of HBase. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9742) Add Documentation For Simple User Access
[ https://issues.apache.org/jira/browse/HBASE-9742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804428#comment-13804428 ] stack commented on HBASE-9742: -- Applied Jesse's 0.94 patch. Thanks Jesse. Add Documentation For Simple User Access Key: HBASE-9742 URL: https://issues.apache.org/jira/browse/HBASE-9742 Project: HBase Issue Type: Improvement Components: documentation Affects Versions: 0.94.12, 0.96.0 Reporter: Jesse Anderson Assignee: Jesse Anderson Fix For: 0.96.0 Attachments: simple0_94.diff, simple.diff, simplewith0_94.diff The [security section|http://hbase.apache.org/book/security.html] in the HBase book only talks about using Kerberos. There is a simple user access mode too that is not documented. This should be documented for development systems and HBase installs where security is not an issue, but they want to prevent user mistakes. I've added a section to the security chapter that talks about simple user access. The new section makes it very clear it is not secure. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-8691) High-Throughput Streaming Scan API
[ https://issues.apache.org/jira/browse/HBASE-8691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804430#comment-13804430 ] Ted Yu commented on HBASE-8691: --- @Sandy: Were there any issues uncovered from the deployment with streaming API ? Are you able to produce patch for trunk ? Cheers High-Throughput Streaming Scan API -- Key: HBASE-8691 URL: https://issues.apache.org/jira/browse/HBASE-8691 Project: HBase Issue Type: Improvement Components: Scanners Affects Versions: 0.95.0 Reporter: Sandy Pratt Assignee: Sandy Pratt Labels: perfomance, scan Attachments: HRegionServlet.java, README.txt, RecordReceiver.java, ScannerTest.java, StreamHRegionServer.java, StreamReceiverDirect.java, StreamServletDirect.java I've done some working testing various ways to refactor and optimize Scans in HBase, and have found that performance can be dramatically increased by the addition of a streaming scan API. The attached code constitutes a proof of concept that shows performance increases of almost 4x in some workloads. I'd appreciate testing, replication, and comments. If the approach seems viable, I think such an API should be built into some future version of HBase. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9831) 'hbasefsck.numthreads' property can not pass to hbck via generic option
[
https://issues.apache.org/jira/browse/HBASE-9831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804437#comment-13804437
]
takeshi.miao commented on HBASE-9831:
-
@JMS, yes, it is not required. I afraid that it more tends to my personal
coding habit :P, I can remove it and apply both 0.94 and trunk patch later. tks
'hbasefsck.numthreads' property can not pass to hbck via generic option
---
Key: HBASE-9831
URL: https://issues.apache.org/jira/browse/HBASE-9831
Project: HBase
Issue Type: Improvement
Components: hbck
Affects Versions: 0.94.12
Reporter: takeshi.miao
Priority: Minor
Labels: hbck
Fix For: 0.94.13
Attachments: HBASE-9831-0.94-v02.patch, HBASE-9831.v01.patch
We use generic option way to pass _'hbasefsck.numthreads'_ property to
_'hbase hbck'_, but it does not accept our new setting value
{code}
hbase hbck -D hbasefsck.numthreads=5
{code}
We can still find there are threads over than 5 we already set via generic
opttion
{code}
[2013-10-24
09:25:02,561][pool-2-thread-6][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,562][pool-2-thread-10][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,565][pool-2-thread-13][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,566][pool-2-thread-11][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,567][pool-2-thread-9][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,568][pool-2-thread-12][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,570][pool-2-thread-7][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,571][pool-2-thread-14][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-8397) improve unit-test coverage of package org.apache.hadoop.hbase.master.metrics (0.94)
[ https://issues.apache.org/jira/browse/HBASE-8397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804442#comment-13804442 ] Ted Yu commented on HBASE-8397: --- TestMasterStatistics passed. +1 improve unit-test coverage of package org.apache.hadoop.hbase.master.metrics (0.94) --- Key: HBASE-8397 URL: https://issues.apache.org/jira/browse/HBASE-8397 Project: HBase Issue Type: Test Affects Versions: 0.94.8 Reporter: Ivan A. Veselovsky Attachments: HBASE-8397-0.94--N3.patch, HBASE-8397-0.94--N5.patch, HBASE-8397-0.94--N6.patch The patch suggests new test that elevates unit-test coverage of package org.apache.hadoop.hbase.master.metrics . The patch is applicable to branch 0.94 only. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-8397) improve unit-test coverage of package org.apache.hadoop.hbase.master.metrics (0.94)
[ https://issues.apache.org/jira/browse/HBASE-8397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-8397: -- Assignee: Ivan A. Veselovsky improve unit-test coverage of package org.apache.hadoop.hbase.master.metrics (0.94) --- Key: HBASE-8397 URL: https://issues.apache.org/jira/browse/HBASE-8397 Project: HBase Issue Type: Test Affects Versions: 0.94.8 Reporter: Ivan A. Veselovsky Assignee: Ivan A. Veselovsky Attachments: HBASE-8397-0.94--N3.patch, HBASE-8397-0.94--N5.patch, HBASE-8397-0.94--N6.patch The patch suggests new test that elevates unit-test coverage of package org.apache.hadoop.hbase.master.metrics . The patch is applicable to branch 0.94 only. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-8397) improve unit-test coverage of package org.apache.hadoop.hbase.master.metrics (0.94)
[ https://issues.apache.org/jira/browse/HBASE-8397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-8397: -- Fix Version/s: 0.94.13 Hadoop Flags: Reviewed Integrated to 0.94 Thanks for the review, Stack. improve unit-test coverage of package org.apache.hadoop.hbase.master.metrics (0.94) --- Key: HBASE-8397 URL: https://issues.apache.org/jira/browse/HBASE-8397 Project: HBase Issue Type: Test Affects Versions: 0.94.8 Reporter: Ivan A. Veselovsky Assignee: Ivan A. Veselovsky Fix For: 0.94.13 Attachments: HBASE-8397-0.94--N3.patch, HBASE-8397-0.94--N5.patch, HBASE-8397-0.94--N6.patch The patch suggests new test that elevates unit-test coverage of package org.apache.hadoop.hbase.master.metrics . The patch is applicable to branch 0.94 only. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-8397) improve unit-test coverage of package org.apache.hadoop.hbase.master.metrics (0.94)
[ https://issues.apache.org/jira/browse/HBASE-8397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-8397: -- Resolution: Fixed Status: Resolved (was: Patch Available) improve unit-test coverage of package org.apache.hadoop.hbase.master.metrics (0.94) --- Key: HBASE-8397 URL: https://issues.apache.org/jira/browse/HBASE-8397 Project: HBase Issue Type: Test Affects Versions: 0.94.8 Reporter: Ivan A. Veselovsky Assignee: Ivan A. Veselovsky Fix For: 0.94.13 Attachments: HBASE-8397-0.94--N3.patch, HBASE-8397-0.94--N5.patch, HBASE-8397-0.94--N6.patch The patch suggests new test that elevates unit-test coverage of package org.apache.hadoop.hbase.master.metrics . The patch is applicable to branch 0.94 only. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9819) Backport HBASE-8372 'Provide mutability to CompoundConfiguration' to 0.94
[
https://issues.apache.org/jira/browse/HBASE-9819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804482#comment-13804482
]
Hudson commented on HBASE-9819:
---
SUCCESS: Integrated in HBase-0.94-security #322 (See
[https://builds.apache.org/job/HBase-0.94-security/322/])
HBASE-9819 Backport HBASE-8372 'Provide mutability to CompoundConfiguration' to
0.94 (Ted Yu) (tedyu: rev 1535442)
*
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/CompoundConfiguration.java
*
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundConfiguration.java
Backport HBASE-8372 'Provide mutability to CompoundConfiguration' to 0.94
-
Key: HBASE-9819
URL: https://issues.apache.org/jira/browse/HBASE-9819
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 0.94.14
Attachments: 9819.txt
In the email thread: http://search-hadoop.com/m/dcqod1uy32h
yonghu encountered the following exception when he tried to retrieve
HTableInterface:
{code}
ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
Failed 1 action: org.apache.hadoop.hbase.DoNotRetryIOException:
Coprocessor:
'org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionEnvironment@9a99eb'
threw: 'java.lang.UnsupportedOperationException: Immutable Configuration'
and has been removedfrom the active coprocessor set.
at
org.apache.hadoop.hbase.coprocessor.CoprocessorHost.handleCoprocessorThrowable(CoprocessorHost.java:740)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.prePut(RegionCoprocessorHost.java:810)
at
org.apache.hadoop.hbase.regionserver.HRegion.doPreMutationHook(HRegion.java:2196)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2172)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3811)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
Caused by: java.lang.UnsupportedOperationException: Immutable Configuration
at
org.apache.hadoop.hbase.regionserver.CompoundConfiguration.set(CompoundConfiguration.java:484)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.ensureZookeeperTrackers(HConnectionManager.java:721)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:986)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:961)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:251)
at org.apache.hadoop.hbase.client.HTable.init(HTable.java:243)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getTable(HConnectionManager.java:671)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getTable(HConnectionManager.java:658)
at
CDCTrigger.TriggerForModification.prePut(TriggerForModification.java:61)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.prePut(RegionCoprocessorHost.java:808)
... 9 more
: 1 time, servers with issues: hans-laptop:60020
{code}
CompoundConfiguration is mutable in 0.96 and beyond.
This should be backported to 0.94
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-7544) Transparent table/CF encryption
[
https://issues.apache.org/jira/browse/HBASE-7544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-7544:
--
Attachment: 7544.patch
Posted updated patch to RB.
Adds a missing license header.
Adds shell support for testing this feature, new CF attributes 'ENCRYPTION'
(the algorithm name, as a string) and 'ENCRYPTION_KEY' (a string that will be
hashed into a 128 bit key).
Adds caching of instantiated key providers.
Adds LoadTestTool support for enabling transparent encryption on a CF.
Transparent table/CF encryption
---
Key: HBASE-7544
URL: https://issues.apache.org/jira/browse/HBASE-7544
Project: HBase
Issue Type: New Feature
Components: HFile, io
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Fix For: 0.98.0
Attachments: 7544p1.patch, 7544p1.patch, 7544p2.patch, 7544p2.patch,
7544p3.patch, 7544p3.patch, 7544p4.patch, 7544.patch, 7544.patch,
historical-7544.patch, historical-7544.pdf, historical-shell.patch
Introduce transparent encryption of HBase on disk data.
Depends on a separate contribution of an encryption codec framework to Hadoop
core and an AES-NI (native code) codec. This is work done in the context of
MAPREDUCE-4491 but I'd gather there will be additional JIRAs for common and
HDFS parts of it.
Requirements:
- Transparent encryption at the CF or table level
- Protect against all data leakage from files at rest
- Two-tier key architecture for consistency with best practices for this
feature in the RDBMS world
- Built-in key management
- Flexible and non-intrusive key rotation
- Mechanisms not exposed to or modifiable by users
- Hardware security module integration (via Java KeyStore)
- HBCK support for transparently encrypted files (+ plugin architecture for
HBCK)
Additional goals:
- Shell support for administrative functions
- Avoid performance impact for the null crypto codec case
- Play nicely with other changes underway: in HFile, block coding, etc.
We're aiming for rough parity with Oracle's transparent tablespace encryption
feature, described in
http://www.oracle.com/technetwork/database/owp-security-advanced-security-11gr-133411.pdf
as
{quote}
“Transparent Data Encryption uses a 2-tier key architecture for flexible and
non-intrusive key rotation and least operational and performance impact: Each
application table with at least one encrypted column has its own table key,
which is applied to all encrypted columns in that table. Equally, each
encrypted tablespace has its own tablespace key. Table keys are stored in the
data dictionary of the database, while tablespace keys are stored in the
header of the tablespace and additionally, the header of each underlying OS
file that makes up the tablespace. Each of these keys is encrypted with the
TDE master encryption key, which is stored outside of the database in an
external security module: either the Oracle Wallet (a PKCS#12 formatted file
that is encrypted using a passphrase supplied either by the designated
security administrator or DBA during setup), or a Hardware Security Module
(HSM) device for higher assurance […]”
{quote}
Further design details forthcoming in a design document and patch as soon as
we have all of the clearances in place.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-8372) Provide mutability to CompoundConfiguration
[ https://issues.apache.org/jira/browse/HBASE-8372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804483#comment-13804483 ] Hudson commented on HBASE-8372: --- SUCCESS: Integrated in HBase-0.94-security #322 (See [https://builds.apache.org/job/HBase-0.94-security/322/]) HBASE-9819 Backport HBASE-8372 'Provide mutability to CompoundConfiguration' to 0.94 (Ted Yu) (tedyu: rev 1535442) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/CompoundConfiguration.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundConfiguration.java Provide mutability to CompoundConfiguration --- Key: HBASE-8372 URL: https://issues.apache.org/jira/browse/HBASE-8372 Project: HBase Issue Type: New Feature Reporter: Ted Yu Assignee: Jonathan Hsieh Fix For: 0.98.0, 0.95.1 Attachments: hbase-8372.patch, hbase-8372.v2.patch, hbase-8372.v3.patch, hbase-8372.v4.patch, hbase-8372.v5.patch In discussion of HBASE-8347, it was proposed that CompoundConfiguration should support mutability. This can be done by consolidating ImmutableConfigMap's on first modification to CompoundConfiguration. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Assigned] (HBASE-8552) fix coverage org.apache.hadoop.hbase.rest.filter
[ https://issues.apache.org/jira/browse/HBASE-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov reassigned HBASE-8552: -- Assignee: Andrey Klochkov fix coverage org.apache.hadoop.hbase.rest.filter - Key: HBASE-8552 URL: https://issues.apache.org/jira/browse/HBASE-8552 Project: HBase Issue Type: Test Reporter: Aleksey Gorshkov Assignee: Andrey Klochkov Attachments: HBASE-8552-0.94.patch, HBASE-8552-trunk--n2.patch, HBASE-8552-trunk.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-8553) improve unit-test coverage of package org.apache.hadoop.hbase.mapreduce.hadoopbackport
[ https://issues.apache.org/jira/browse/HBASE-8553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804498#comment-13804498 ] Devaraj Das commented on HBASE-8553: LGTM. [~lhofhansl], can I integrate this (harmless, mostly Test fixes patch) in 0.94. improve unit-test coverage of package org.apache.hadoop.hbase.mapreduce.hadoopbackport -- Key: HBASE-8553 URL: https://issues.apache.org/jira/browse/HBASE-8553 Project: HBase Issue Type: Test Affects Versions: 0.94.9 Reporter: Ivan A. Veselovsky Attachments: HBASE-8553-0.94--N2.patch The patch is for branch 0.94 only. The class InputSampler is modified because need to fix bug there: in method run(String[] args) should be TotalOrderPartitioner.setPartitionFile(job.getConfiguration(), outf); instead of TotalOrderPartitioner.setPartitionFile(getConf(), outf);. Otherwise it is impossible to set output file correctly. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-8741) Scope sequenceid to the region rather than regionserver (WAS: Mutations on Regions in recovery mode might have same sequenceIDs)
[
https://issues.apache.org/jira/browse/HBASE-8741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804499#comment-13804499
]
Hadoop QA commented on HBASE-8741:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12610111/HBASE-8741-trunk-v6.1-rebased.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 36 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.coprocessor.TestRegionObserverScannerOpenHook
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7616//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7616//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7616//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7616//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7616//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7616//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7616//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7616//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7616//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7616//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7616//console
This message is automatically generated.
Scope sequenceid to the region rather than regionserver (WAS: Mutations on
Regions in recovery mode might have same sequenceIDs)
Key: HBASE-8741
URL: https://issues.apache.org/jira/browse/HBASE-8741
Project: HBase
Issue Type: Bug
Components: MTTR
Affects Versions: 0.95.1
Reporter: Himanshu Vashishtha
Assignee: Himanshu Vashishtha
Fix For: 0.98.0
Attachments: HBASE-8741-trunk-v6.1-rebased.patch,
HBASE-8741-trunk-v6.patch, HBASE-8741-v0.patch, HBASE-8741-v2.patch,
HBASE-8741-v3.patch, HBASE-8741-v4-again.patch, HBASE-8741-v4-again.patch,
HBASE-8741-v4.patch, HBASE-8741-v5-again.patch, HBASE-8741-v5.patch
Currently, when opening a region, we find the maximum sequence ID from all
its HFiles and then set the LogSequenceId of the log (in case the later is at
a small value). This works good in recovered.edits case as we are not writing
to the region until we have replayed all of its previous edits.
With distributed log replay, if we want to enable writes while a region is
under recovery, we need to make sure that the logSequenceId maximum
logSequenceId of the old regionserver. Otherwise, we might have a situation
where new edits have same (or smaller) sequenceIds.
We can store region level information in the WALTrailer, than this scenario
could be avoided by:
a) reading the trailer of the last completed file, i.e., last wal file
which has a trailer and,
b) completely reading the last wal file (this file would not have the
trailer, so it needs to be read completely).
In future, if we switch to multi wal
[jira] [Commented] (HBASE-9831) 'hbasefsck.numthreads' property can not pass to hbck via generic option
[
https://issues.apache.org/jira/browse/HBASE-9831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804504#comment-13804504
]
Jean-Marc Spaggiari commented on HBASE-9831:
Thanks [~takeshi.miao].
So the right process now is:
1) Cancel patch.
2) Upload trunk version
3) Apply patch
4) Wait for hadoopQA to test it
5) Upload 0.94 version.
JM
'hbasefsck.numthreads' property can not pass to hbck via generic option
---
Key: HBASE-9831
URL: https://issues.apache.org/jira/browse/HBASE-9831
Project: HBase
Issue Type: Improvement
Components: hbck
Affects Versions: 0.94.12
Reporter: takeshi.miao
Priority: Minor
Labels: hbck
Fix For: 0.94.13
Attachments: HBASE-9831-0.94-v02.patch, HBASE-9831.v01.patch
We use generic option way to pass _'hbasefsck.numthreads'_ property to
_'hbase hbck'_, but it does not accept our new setting value
{code}
hbase hbck -D hbasefsck.numthreads=5
{code}
We can still find there are threads over than 5 we already set via generic
opttion
{code}
[2013-10-24
09:25:02,561][pool-2-thread-6][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,562][pool-2-thread-10][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,565][pool-2-thread-13][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,566][pool-2-thread-11][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,567][pool-2-thread-9][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,568][pool-2-thread-12][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,570][pool-2-thread-7][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,571][pool-2-thread-14][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9767) Support OperationAttributes in ImportTSV Parser
[ https://issues.apache.org/jira/browse/HBASE-9767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-9767: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to 0.98. Thanks for the review Anoop and Andrew. Support OperationAttributes in ImportTSV Parser --- Key: HBASE-9767 URL: https://issues.apache.org/jira/browse/HBASE-9767 Project: HBase Issue Type: Improvement Affects Versions: 0.98.0 Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Attachments: HBASE-9767_1.patch, HBASE-9767.patch This JIRA aims at supporting the operation attributes in bulk loads. Ideally this operation attributes once extracted has to be set in the mappers/reducers. In case of mappers using TableOutPutFormat this would be very helpful when the puts are done through HTable. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9831) 'hbasefsck.numthreads' property can not pass to hbck via generic option
[
https://issues.apache.org/jira/browse/HBASE-9831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804514#comment-13804514
]
takeshi.miao commented on HBASE-9831:
-
@JM Got it, do it now
'hbasefsck.numthreads' property can not pass to hbck via generic option
---
Key: HBASE-9831
URL: https://issues.apache.org/jira/browse/HBASE-9831
Project: HBase
Issue Type: Improvement
Components: hbck
Affects Versions: 0.94.12
Reporter: takeshi.miao
Priority: Minor
Labels: hbck
Fix For: 0.94.13
Attachments: HBASE-9831-0.94-v02.patch, HBASE-9831.v01.patch
We use generic option way to pass _'hbasefsck.numthreads'_ property to
_'hbase hbck'_, but it does not accept our new setting value
{code}
hbase hbck -D hbasefsck.numthreads=5
{code}
We can still find there are threads over than 5 we already set via generic
opttion
{code}
[2013-10-24
09:25:02,561][pool-2-thread-6][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,562][pool-2-thread-10][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,565][pool-2-thread-13][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,566][pool-2-thread-11][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,567][pool-2-thread-9][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,568][pool-2-thread-12][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,570][pool-2-thread-7][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,571][pool-2-thread-14][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-3149) Make flush decisions per column family
[ https://issues.apache.org/jira/browse/HBASE-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gaurav Menghani updated HBASE-3149: --- Fix Version/s: 0.89-fb Status: Patch Available (was: Open) Patch for the 0.89-fb version. Make flush decisions per column family -- Key: HBASE-3149 URL: https://issues.apache.org/jira/browse/HBASE-3149 Project: HBase Issue Type: Improvement Components: regionserver Reporter: Karthik Ranganathan Assignee: Gaurav Menghani Priority: Critical Fix For: 0.89-fb Today, the flush decision is made using the aggregate size of all column families. When large and small column families co-exist, this causes many small flushes of the smaller CF. We need to make per-CF flush decisions. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9831) 'hbasefsck.numthreads' property can not pass to hbck via generic option
[
https://issues.apache.org/jira/browse/HBASE-9831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
takeshi.miao updated HBASE-9831:
Status: Open (was: Patch Available)
Cancel patch for re-applying the patch for trunk
'hbasefsck.numthreads' property can not pass to hbck via generic option
---
Key: HBASE-9831
URL: https://issues.apache.org/jira/browse/HBASE-9831
Project: HBase
Issue Type: Improvement
Components: hbck
Affects Versions: 0.94.12
Reporter: takeshi.miao
Priority: Minor
Labels: hbck
Fix For: 0.94.13
Attachments: HBASE-9831-0.94-v02.patch, HBASE-9831-trunk-v01.patch,
HBASE-9831.v01.patch
We use generic option way to pass _'hbasefsck.numthreads'_ property to
_'hbase hbck'_, but it does not accept our new setting value
{code}
hbase hbck -D hbasefsck.numthreads=5
{code}
We can still find there are threads over than 5 we already set via generic
opttion
{code}
[2013-10-24
09:25:02,561][pool-2-thread-6][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,562][pool-2-thread-10][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,565][pool-2-thread-13][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,566][pool-2-thread-11][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,567][pool-2-thread-9][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,568][pool-2-thread-12][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,570][pool-2-thread-7][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,571][pool-2-thread-14][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9831) 'hbasefsck.numthreads' property can not pass to hbck via generic option
[
https://issues.apache.org/jira/browse/HBASE-9831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
takeshi.miao updated HBASE-9831:
Status: Patch Available (was: Open)
'hbasefsck.numthreads' property can not pass to hbck via generic option
---
Key: HBASE-9831
URL: https://issues.apache.org/jira/browse/HBASE-9831
Project: HBase
Issue Type: Improvement
Components: hbck
Affects Versions: 0.94.12
Reporter: takeshi.miao
Priority: Minor
Labels: hbck
Fix For: 0.94.13
Attachments: HBASE-9831-0.94-v02.patch, HBASE-9831-trunk-v01.patch,
HBASE-9831.v01.patch
We use generic option way to pass _'hbasefsck.numthreads'_ property to
_'hbase hbck'_, but it does not accept our new setting value
{code}
hbase hbck -D hbasefsck.numthreads=5
{code}
We can still find there are threads over than 5 we already set via generic
opttion
{code}
[2013-10-24
09:25:02,561][pool-2-thread-6][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,562][pool-2-thread-10][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,565][pool-2-thread-13][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,566][pool-2-thread-11][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,567][pool-2-thread-9][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,568][pool-2-thread-12][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,570][pool-2-thread-7][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,571][pool-2-thread-14][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9831) 'hbasefsck.numthreads' property can not pass to hbck via generic option
[
https://issues.apache.org/jira/browse/HBASE-9831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804521#comment-13804521
]
takeshi.miao commented on HBASE-9831:
-
submit patch again
'hbasefsck.numthreads' property can not pass to hbck via generic option
---
Key: HBASE-9831
URL: https://issues.apache.org/jira/browse/HBASE-9831
Project: HBase
Issue Type: Improvement
Components: hbck
Affects Versions: 0.94.12
Reporter: takeshi.miao
Priority: Minor
Labels: hbck
Fix For: 0.94.13
Attachments: HBASE-9831-0.94-v02.patch, HBASE-9831-trunk-v01.patch,
HBASE-9831.v01.patch
We use generic option way to pass _'hbasefsck.numthreads'_ property to
_'hbase hbck'_, but it does not accept our new setting value
{code}
hbase hbck -D hbasefsck.numthreads=5
{code}
We can still find there are threads over than 5 we already set via generic
opttion
{code}
[2013-10-24
09:25:02,561][pool-2-thread-6][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,562][pool-2-thread-10][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,565][pool-2-thread-13][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,566][pool-2-thread-11][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,567][pool-2-thread-9][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,568][pool-2-thread-12][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,570][pool-2-thread-7][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,571][pool-2-thread-14][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9831) 'hbasefsck.numthreads' property can not pass to hbck via generic option
[
https://issues.apache.org/jira/browse/HBASE-9831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
takeshi.miao updated HBASE-9831:
Attachment: HBASE-9831-trunk-v01.patch
add trunk-v01.patch
'hbasefsck.numthreads' property can not pass to hbck via generic option
---
Key: HBASE-9831
URL: https://issues.apache.org/jira/browse/HBASE-9831
Project: HBase
Issue Type: Improvement
Components: hbck
Affects Versions: 0.94.12
Reporter: takeshi.miao
Priority: Minor
Labels: hbck
Fix For: 0.94.13
Attachments: HBASE-9831-0.94-v02.patch, HBASE-9831-trunk-v01.patch,
HBASE-9831.v01.patch
We use generic option way to pass _'hbasefsck.numthreads'_ property to
_'hbase hbck'_, but it does not accept our new setting value
{code}
hbase hbck -D hbasefsck.numthreads=5
{code}
We can still find there are threads over than 5 we already set via generic
opttion
{code}
[2013-10-24
09:25:02,561][pool-2-thread-6][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,562][pool-2-thread-10][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,565][pool-2-thread-13][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,566][pool-2-thread-11][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,567][pool-2-thread-9][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,568][pool-2-thread-12][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,570][pool-2-thread-7][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
[2013-10-24
09:25:02,571][pool-2-thread-14][DEBUG][org.apache.hadoop.security.UserGroupInformation]:
PrivilegedAction as:hbase/spn-d-hdn1.s...@ispn.trendmicro.com
(auth:KERBEROS) from:sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) (UserGroupInformation.java:1430)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-3149) Make flush decisions per column family
[ https://issues.apache.org/jira/browse/HBASE-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gaurav Menghani updated HBASE-3149: --- Attachment: Per-CF-Memstore-Flush.diff This diff is only for the 0.89-fb branch, I will port this to master soon. Make flush decisions per column family -- Key: HBASE-3149 URL: https://issues.apache.org/jira/browse/HBASE-3149 Project: HBase Issue Type: Improvement Components: regionserver Reporter: Karthik Ranganathan Assignee: Gaurav Menghani Priority: Critical Fix For: 0.89-fb Attachments: Per-CF-Memstore-Flush.diff Today, the flush decision is made using the aggregate size of all column families. When large and small column families co-exist, this causes many small flushes of the smaller CF. We need to make per-CF flush decisions. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-3149) Make flush decisions per column family
[
https://issues.apache.org/jira/browse/HBASE-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804528#comment-13804528
]
Hadoop QA commented on HBASE-3149:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12610129/Per-CF-Memstore-Flush.diff
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7618//console
This message is automatically generated.
Make flush decisions per column family
--
Key: HBASE-3149
URL: https://issues.apache.org/jira/browse/HBASE-3149
Project: HBase
Issue Type: Improvement
Components: regionserver
Reporter: Karthik Ranganathan
Assignee: Gaurav Menghani
Priority: Critical
Fix For: 0.89-fb
Attachments: Per-CF-Memstore-Flush.diff
Today, the flush decision is made using the aggregate size of all column
families. When large and small column families co-exist, this causes many
small flushes of the smaller CF. We need to make per-CF flush decisions.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9825) LoadIncrementalHFiles fails to load from remote cluster in hadoop 2
[
https://issues.apache.org/jira/browse/HBASE-9825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804535#comment-13804535
]
Jerry He commented on HBASE-9825:
-
Verified the problem has been fixed with HBASE-9819.
Thanks, Ted.
LoadIncrementalHFiles fails to load from remote cluster in hadoop 2
---
Key: HBASE-9825
URL: https://issues.apache.org/jira/browse/HBASE-9825
Project: HBase
Issue Type: Bug
Components: hadoop2
Affects Versions: 0.94.12
Reporter: Jerry He
Attachments: HBASE-9825-0.94-only.patch
Running on hadoop 2, LoadIncrementalHFiles gives the following exception when
loading from a remote cluster.
{code}
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$3@455e455e,
java.io.IOException: j ava.io.IOException:
java.lang.UnsupportedOperationException: Immutable Configuration
at
org.apache.hadoop.hbase.regionserver.CompoundConfiguration.setClass(CompoundConfiguration.java:516)
at org.apache.hadoop.ipc.RPC.setProtocolEngine(RPC.java:195)
at
org.apache.hadoop.hdfs.NameNodeProxies.createNNProxyWithClientProtocol(NameNodeProxies.java:250)
at
org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:169)
at
org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:130)
at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:482)
at org.apache.hadoop.hdfs.DFSClient.init(DFSClient.java:445)
at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:136)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2429)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2463)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2445)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:363)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:283)
at
org.apache.hadoop.hbase.regionserver.Store.assertBulkLoadHFileOk(Store.java:571)
at
org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:3689)
at
org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:3637)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.bulkLoadHFiles(HRegionServer.java:2939)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
at
org.apache.hadoop.hbase.client.ServerCallable.withRetries(ServerCallable.java:186)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.tryAtomicRegionLoad(LoadIncrementalHFiles.java:567)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$1.call(LoadIncrementalHFiles.java:317)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$1.call(LoadIncrementalHFiles.java:315)
{code}
This does not happen when loading from the same FileSystem.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-3149) Make flush decisions per column family
[ https://issues.apache.org/jira/browse/HBASE-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804537#comment-13804537 ] Gaurav Menghani commented on HBASE-3149: The basic idea is to be able to maintain the smallest LSN amongst the edits present in a particular memstore for a column family. When we decide to flush a set of memstores, we find the smallest LSN id amongst the memstores that we are not flushing, say X, and say that we can remove the logs for any edits with LSN less than X. We choose a particular memstore to be flushed, if it occupies more than 't' bytes, when the global memstore size threshold is 'T' (and t/T = 1/4 for our configuration). If there is no memstore with = t bytes but the total size of all the memstores is above T, we flush all the memstores. Make flush decisions per column family -- Key: HBASE-3149 URL: https://issues.apache.org/jira/browse/HBASE-3149 Project: HBase Issue Type: Improvement Components: regionserver Reporter: Karthik Ranganathan Assignee: Gaurav Menghani Priority: Critical Fix For: 0.89-fb Attachments: Per-CF-Memstore-Flush.diff Today, the flush decision is made using the aggregate size of all column families. When large and small column families co-exist, this causes many small flushes of the smaller CF. We need to make per-CF flush decisions. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-8859) truncate_preserve should get table split keys as it is instead of converting them to string type and then again to bytes
[
https://issues.apache.org/jira/browse/HBASE-8859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804542#comment-13804542
]
rajeshbabu commented on HBASE-8859:
---
These are APIs related to split keys need to be added in HTI
getStartKeys()
getEndKeys()
getStartEndKeys()
getSplitKeys() - New
and 3 more APIs related to find region location when we specify row also can be
added to HTI
getRegionLocation(final String row)
getRegionLocation(final byte [] row)
getRegionLocation(final byte [] row, boolean reload)
truncate_preserve should get table split keys as it is instead of converting
them to string type and then again to bytes
Key: HBASE-8859
URL: https://issues.apache.org/jira/browse/HBASE-8859
Project: HBase
Issue Type: Bug
Components: scripts
Affects Versions: 0.95.1
Reporter: rajeshbabu
Assignee: rajeshbabu
Fix For: 0.98.0, 0.96.1
Attachments: HBASE-8859-Test_to_reproduce.patch,
HBASE-8859_trunk_2.patch, HBASE-8859_trunk.patch
If we take int,long or double bytes as split keys then we are not creating
table with same split keys because converting them to strings directly and to
bytes is giving different split keys, sometimes getting IllegalArgument
exception because of same split keys(converted). Instead we can get split
keys directly from HTable and pass them while creating table.
{code}
h_table = org.apache.hadoop.hbase.client.HTable.new(conf, table_name)
splits = h_table.getRegionLocations().keys().map{|i| i.getStartKey}
:byte
splits = org.apache.hadoop.hbase.util.Bytes.toByteArrays(splits)
{code}
{code}
Truncating 'emp3' table (it may take a while):
- Disabling table...
- Dropping table...
- Creating table with region boundaries...
ERROR: java.lang.IllegalArgumentException: All split keys must be unique,
found duplicate: B\x11S\xEF\xBF\xBD\xEF\xBF\xBD\xEF\xBF\xBD\x00\x00,
B\x11S\xEF\xBF\xBD\xEF\xBF\xBD\xEF\xBF\xBD\x00\x00
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-8611) Improve test coverage in pkg org.apache.hadoop.hbase.mapred
[ https://issues.apache.org/jira/browse/HBASE-8611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HBASE-8611: --- Attachment: driver.patch [~vbondarev], tried applying the patch on trunk and it failed - was a simple fix and the fixed Driver.java changes are attached. I tried to run the tests outlined by Nick. It seems to be failing. Can you please take a look. The command I ran: mvn clean -Dhadoop.profile=2.0 test -Dtest=org.apache.hadoop.hbase.mapred.TestDriver,org.apache.hadoop.hbase.mapred.TestGroupingTableMap,org.apache.hadoop.hbase.mapred.TestIdentityTableMap,org.apache.hadoop.hbase.mapred.TestRowCounter,org.apache.hadoop.hbase.mapred.TestSplitTable,org.apache.hadoop.hbase.mapred.TestTableMapReduceUtil Improve test coverage in pkg org.apache.hadoop.hbase.mapred --- Key: HBASE-8611 URL: https://issues.apache.org/jira/browse/HBASE-8611 Project: HBase Issue Type: Test Reporter: Vadim Bondarev Assignee: Vadim Bondarev Attachments: driver.patch, HBASE-8611-0.94-a.patch, HBASE-8611-0.94-c.patch, HBASE-8611-trunk-a.patch, HBASE-8611-trunk-b.patch, HBASE-8611-trunk-c.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9593) Region server left in online servers list forever if it went down after registering to master and before creating ephemeral node
[
https://issues.apache.org/jira/browse/HBASE-9593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804549#comment-13804549
]
rajeshbabu commented on HBASE-9593:
---
Committed to trunk and 0.96.1
Thanks for review Ted and Lars.
Region server left in online servers list forever if it went down after
registering to master and before creating ephemeral node
Key: HBASE-9593
URL: https://issues.apache.org/jira/browse/HBASE-9593
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.94.11
Reporter: rajeshbabu
Assignee: rajeshbabu
Fix For: 0.98.0, 0.94.13, 0.96.1
Attachments: HBASE-9593.patch, HBASE-9593_v2.patch,
HBASE-9593_v3.patch
In some of our tests we found that regionserer always showing online in
master UI but its actually dead.
If region server went down in the middle following steps then the region
server always showing in master online servers list.
1) register to master
2) create ephemeral znode
Since no notification from zookeeper, master is not removing the expired
server from online servers list.
Assignments will fail if the RS is selected as destination server.
Some cases ROOT or META also wont be assigned if the RS is randomly selected
every time need to wait for timeout.
Here are the logs:
1) HOST-10-18-40-153 is registered to master
{code}
2013-09-19 19:47:41,123 DEBUG org.apache.hadoop.hbase.master.ServerManager:
STARTUP: Server HOST-10-18-40-153,61020,1379600260255 came back up, removed
it from the dead servers list
2013-09-19 19:47:41,123 INFO org.apache.hadoop.hbase.master.ServerManager:
Registering server=HOST-10-18-40-153,61020,1379600260255
{code}
{code}
2013-09-19 19:47:41,119 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Connected to master at
HOST-10-18-40-153/10.18.40.153:61000
2013-09-19 19:47:41,119 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Telling master at
HOST-10-18-40-153,61000,1379600055284 that we are up with port=61020,
startcode=1379600260255
{code}
2) Terminated before creating ephemeral node.
{code}
Thu Sep 19 19:47:41 IST 2013 Terminating regionserver
{code}
3) The RS can be selected for assignment and they will fail.
{code}
2013-09-19 19:47:54,049 WARN
org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment of
-ROOT-,,0.70236052 to HOST-10-18-40-153,61020,1379600260255, trying to assign
elsewhere instead; retry=0
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupConnection(HBaseClient.java:390)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:436)
at
org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1127)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:974)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:86)
at $Proxy15.openRegion(Unknown Source)
at
org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:533)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1734)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1431)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1406)
at
org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1401)
at
org.apache.hadoop.hbase.master.AssignmentManager.assignRoot(AssignmentManager.java:2374)
at
org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.verifyAndAssignRoot(MetaServerShutdownHandler.java:136)
at
org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.verifyAndAssignRootWithRetries(MetaServerShutdownHandler.java:160)
at
org.apache.hadoop.hbase.master.handler.MetaServerShutdownHandler.process(MetaServerShutdownHandler.java:82)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
2013-09-19
[jira] [Commented] (HBASE-8397) improve unit-test coverage of package org.apache.hadoop.hbase.master.metrics (0.94)
[ https://issues.apache.org/jira/browse/HBASE-8397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804552#comment-13804552 ] Hudson commented on HBASE-8397: --- FAILURE: Integrated in HBase-0.94-security #323 (See [https://builds.apache.org/job/HBase-0.94-security/323/]) HBASE-8397 improve unit-test coverage of package org.apache.hadoop.hbase.master.metrics (0.94) (tedyu: rev 1535452) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/metrics/MasterMetrics.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/metrics * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/metrics/TestMasterStatistics.java improve unit-test coverage of package org.apache.hadoop.hbase.master.metrics (0.94) --- Key: HBASE-8397 URL: https://issues.apache.org/jira/browse/HBASE-8397 Project: HBase Issue Type: Test Affects Versions: 0.94.8 Reporter: Ivan A. Veselovsky Assignee: Ivan A. Veselovsky Fix For: 0.94.13 Attachments: HBASE-8397-0.94--N3.patch, HBASE-8397-0.94--N5.patch, HBASE-8397-0.94--N6.patch The patch suggests new test that elevates unit-test coverage of package org.apache.hadoop.hbase.master.metrics . The patch is applicable to branch 0.94 only. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HBASE-9833) Be able to block on HTableMultiplexer puts when the buffer is full
Gaurav Menghani created HBASE-9833: -- Summary: Be able to block on HTableMultiplexer puts when the buffer is full Key: HBASE-9833 URL: https://issues.apache.org/jira/browse/HBASE-9833 Project: HBase Issue Type: Improvement Affects Versions: 0.89-fb Reporter: Gaurav Menghani Priority: Minor Fix For: 0.89-fb So far, when we try to insert to HTableMultiplexer, it returns false if the buffer is full. This forces one to write wrapper code to retry inserting into the buffer. If someone is okay with the put getting blocked while the buffer is flushed, we should block. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HBASE-9834) Minimize byte[] copies for 'smart' clients
Jesse Yates created HBASE-9834: -- Summary: Minimize byte[] copies for 'smart' clients Key: HBASE-9834 URL: https://issues.apache.org/jira/browse/HBASE-9834 Project: HBase Issue Type: Bug Components: Client Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.98.0, 0.94.13, 0.96.1 'Smart' clients (e.g. phoenix) that have in-depth knowledge of HBase often bemoan the extra byte[] copies that must be done when building multiple puts/deletes. We should provide a mechanism by which they can minimize these copies, but still remain wire compatible. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9834) Minimize byte[] copies for 'smart' clients
[ https://issues.apache.org/jira/browse/HBASE-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804559#comment-13804559 ] Jesse Yates commented on HBASE-9834: Patch coming momentarily for 0.94 (easier to grok for the moment). 0.96/98 shouldn't be too bad, though haven't looked yet :). Minimize byte[] copies for 'smart' clients -- Key: HBASE-9834 URL: https://issues.apache.org/jira/browse/HBASE-9834 Project: HBase Issue Type: Bug Components: Client Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.98.0, 0.94.13, 0.96.1 'Smart' clients (e.g. phoenix) that have in-depth knowledge of HBase often bemoan the extra byte[] copies that must be done when building multiple puts/deletes. We should provide a mechanism by which they can minimize these copies, but still remain wire compatible. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-8397) improve unit-test coverage of package org.apache.hadoop.hbase.master.metrics (0.94)
[ https://issues.apache.org/jira/browse/HBASE-8397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804564#comment-13804564 ] Ted Yu commented on HBASE-8397: --- Looks like this failure was related to the patch: https://builds.apache.org/job/HBase-0.94-security/323/testReport/junit/org.apache.hadoop.hbase.master.metrics/TestMasterStatistics/testMasterStatistics/ @Ivan: Mind taking a look ? improve unit-test coverage of package org.apache.hadoop.hbase.master.metrics (0.94) --- Key: HBASE-8397 URL: https://issues.apache.org/jira/browse/HBASE-8397 Project: HBase Issue Type: Test Affects Versions: 0.94.8 Reporter: Ivan A. Veselovsky Assignee: Ivan A. Veselovsky Fix For: 0.94.13 Attachments: HBASE-8397-0.94--N3.patch, HBASE-8397-0.94--N5.patch, HBASE-8397-0.94--N6.patch The patch suggests new test that elevates unit-test coverage of package org.apache.hadoop.hbase.master.metrics . The patch is applicable to branch 0.94 only. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9833) Be able to block on HTableMultiplexer puts when the buffer is full
[ https://issues.apache.org/jira/browse/HBASE-9833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated HBASE-9833: Tags: Phoenix Be able to block on HTableMultiplexer puts when the buffer is full -- Key: HBASE-9833 URL: https://issues.apache.org/jira/browse/HBASE-9833 Project: HBase Issue Type: Improvement Affects Versions: 0.89-fb Reporter: Gaurav Menghani Priority: Minor Fix For: 0.89-fb So far, when we try to insert to HTableMultiplexer, it returns false if the buffer is full. This forces one to write wrapper code to retry inserting into the buffer. If someone is okay with the put getting blocked while the buffer is flushed, we should block. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9834) Minimize byte[] copies for 'smart' clients
[ https://issues.apache.org/jira/browse/HBASE-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-9834: --- Attachment: hbase-9834-0.94-v0.patch Attaching 0.94 version. Turns out, we can also re-use this for our own clients. Should help cut down some of the time it takes to build Put/Deletes since we don't need to copy all the bytes in each time we create a new kv. Minimize byte[] copies for 'smart' clients -- Key: HBASE-9834 URL: https://issues.apache.org/jira/browse/HBASE-9834 Project: HBase Issue Type: Bug Components: Client Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.98.0, 0.94.13, 0.96.1 Attachments: hbase-9834-0.94-v0.patch 'Smart' clients (e.g. phoenix) that have in-depth knowledge of HBase often bemoan the extra byte[] copies that must be done when building multiple puts/deletes. We should provide a mechanism by which they can minimize these copies, but still remain wire compatible. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9834) Minimize byte[] copies for 'smart' clients
[ https://issues.apache.org/jira/browse/HBASE-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Taylor updated HBASE-9834: Tags: Phoenix Minimize byte[] copies for 'smart' clients -- Key: HBASE-9834 URL: https://issues.apache.org/jira/browse/HBASE-9834 Project: HBase Issue Type: Bug Components: Client Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.98.0, 0.94.13, 0.96.1 Attachments: hbase-9834-0.94-v0.patch 'Smart' clients (e.g. phoenix) that have in-depth knowledge of HBase often bemoan the extra byte[] copies that must be done when building multiple puts/deletes. We should provide a mechanism by which they can minimize these copies, but still remain wire compatible. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9834) Minimize byte[] copies for 'smart' clients
[ https://issues.apache.org/jira/browse/HBASE-9834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804567#comment-13804567 ] Jesse Yates commented on HBASE-9834: Also, shout out to [~lhofhansl] and [~giacomotaylor] for inspiration. Minimize byte[] copies for 'smart' clients -- Key: HBASE-9834 URL: https://issues.apache.org/jira/browse/HBASE-9834 Project: HBase Issue Type: Bug Components: Client Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.98.0, 0.94.13, 0.96.1 Attachments: hbase-9834-0.94-v0.patch 'Smart' clients (e.g. phoenix) that have in-depth knowledge of HBase often bemoan the extra byte[] copies that must be done when building multiple puts/deletes. We should provide a mechanism by which they can minimize these copies, but still remain wire compatible. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-8397) improve unit-test coverage of package org.apache.hadoop.hbase.master.metrics (0.94)
[ https://issues.apache.org/jira/browse/HBASE-8397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804568#comment-13804568 ] Ted Yu commented on HBASE-8397: --- Reverted for now. improve unit-test coverage of package org.apache.hadoop.hbase.master.metrics (0.94) --- Key: HBASE-8397 URL: https://issues.apache.org/jira/browse/HBASE-8397 Project: HBase Issue Type: Test Affects Versions: 0.94.8 Reporter: Ivan A. Veselovsky Assignee: Ivan A. Veselovsky Fix For: 0.94.13 Attachments: HBASE-8397-0.94--N3.patch, HBASE-8397-0.94--N5.patch, HBASE-8397-0.94--N6.patch The patch suggests new test that elevates unit-test coverage of package org.apache.hadoop.hbase.master.metrics . The patch is applicable to branch 0.94 only. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-8372) Provide mutability to CompoundConfiguration
[ https://issues.apache.org/jira/browse/HBASE-8372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804582#comment-13804582 ] Hudson commented on HBASE-8372: --- FAILURE: Integrated in HBase-0.94 #1181 (See [https://builds.apache.org/job/HBase-0.94/1181/]) HBASE-9819 Backport HBASE-8372 'Provide mutability to CompoundConfiguration' to 0.94 (Ted Yu) (tedyu: rev 1535442) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/CompoundConfiguration.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundConfiguration.java Provide mutability to CompoundConfiguration --- Key: HBASE-8372 URL: https://issues.apache.org/jira/browse/HBASE-8372 Project: HBase Issue Type: New Feature Reporter: Ted Yu Assignee: Jonathan Hsieh Fix For: 0.98.0, 0.95.1 Attachments: hbase-8372.patch, hbase-8372.v2.patch, hbase-8372.v3.patch, hbase-8372.v4.patch, hbase-8372.v5.patch In discussion of HBASE-8347, it was proposed that CompoundConfiguration should support mutability. This can be done by consolidating ImmutableConfigMap's on first modification to CompoundConfiguration. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-8397) improve unit-test coverage of package org.apache.hadoop.hbase.master.metrics (0.94)
[ https://issues.apache.org/jira/browse/HBASE-8397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804581#comment-13804581 ] Hudson commented on HBASE-8397: --- FAILURE: Integrated in HBase-0.94 #1181 (See [https://builds.apache.org/job/HBase-0.94/1181/]) HBASE-8397 improve unit-test coverage of package org.apache.hadoop.hbase.master.metrics (0.94) (tedyu: rev 1535452) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/metrics/MasterMetrics.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/metrics * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/master/metrics/TestMasterStatistics.java improve unit-test coverage of package org.apache.hadoop.hbase.master.metrics (0.94) --- Key: HBASE-8397 URL: https://issues.apache.org/jira/browse/HBASE-8397 Project: HBase Issue Type: Test Affects Versions: 0.94.8 Reporter: Ivan A. Veselovsky Assignee: Ivan A. Veselovsky Fix For: 0.94.13 Attachments: HBASE-8397-0.94--N3.patch, HBASE-8397-0.94--N5.patch, HBASE-8397-0.94--N6.patch The patch suggests new test that elevates unit-test coverage of package org.apache.hadoop.hbase.master.metrics . The patch is applicable to branch 0.94 only. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9819) Backport HBASE-8372 'Provide mutability to CompoundConfiguration' to 0.94
[
https://issues.apache.org/jira/browse/HBASE-9819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804580#comment-13804580
]
Hudson commented on HBASE-9819:
---
FAILURE: Integrated in HBase-0.94 #1181 (See
[https://builds.apache.org/job/HBase-0.94/1181/])
HBASE-9819 Backport HBASE-8372 'Provide mutability to CompoundConfiguration' to
0.94 (Ted Yu) (tedyu: rev 1535442)
*
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/CompoundConfiguration.java
*
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompoundConfiguration.java
Backport HBASE-8372 'Provide mutability to CompoundConfiguration' to 0.94
-
Key: HBASE-9819
URL: https://issues.apache.org/jira/browse/HBASE-9819
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 0.94.14
Attachments: 9819.txt
In the email thread: http://search-hadoop.com/m/dcqod1uy32h
yonghu encountered the following exception when he tried to retrieve
HTableInterface:
{code}
ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException:
Failed 1 action: org.apache.hadoop.hbase.DoNotRetryIOException:
Coprocessor:
'org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionEnvironment@9a99eb'
threw: 'java.lang.UnsupportedOperationException: Immutable Configuration'
and has been removedfrom the active coprocessor set.
at
org.apache.hadoop.hbase.coprocessor.CoprocessorHost.handleCoprocessorThrowable(CoprocessorHost.java:740)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.prePut(RegionCoprocessorHost.java:810)
at
org.apache.hadoop.hbase.regionserver.HRegion.doPreMutationHook(HRegion.java:2196)
at
org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2172)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3811)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426)
Caused by: java.lang.UnsupportedOperationException: Immutable Configuration
at
org.apache.hadoop.hbase.regionserver.CompoundConfiguration.set(CompoundConfiguration.java:484)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.ensureZookeeperTrackers(HConnectionManager.java:721)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:986)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:961)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:251)
at org.apache.hadoop.hbase.client.HTable.init(HTable.java:243)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getTable(HConnectionManager.java:671)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getTable(HConnectionManager.java:658)
at
CDCTrigger.TriggerForModification.prePut(TriggerForModification.java:61)
at
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.prePut(RegionCoprocessorHost.java:808)
... 9 more
: 1 time, servers with issues: hans-laptop:60020
{code}
CompoundConfiguration is mutable in 0.96 and beyond.
This should be backported to 0.94
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Comment Edited] (HBASE-8397) improve unit-test coverage of package org.apache.hadoop.hbase.master.metrics (0.94)
[
https://issues.apache.org/jira/browse/HBASE-8397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13804564#comment-13804564
]
Ted Yu edited comment on HBASE-8397 at 10/24/13 7:22 PM:
-
Looks like this failure was related to the patch:
{code}
https://builds.apache.org/job/HBase-0.94-security/323/testReport/junit/org.apache.hadoop.hbase.master.metrics/TestMasterStatistics/testMasterStatistics/
java.lang.AssertionError: The value obtained through bean server should be
equal to the one obtained directly. expected:10.0 but was:0.0
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:443)
at
org.apache.hadoop.hbase.master.metrics.TestMasterStatistics.testMasterStatistics(TestMasterStatistics.java:74)
{code}
@Ivan:
Mind taking a look ?
was (Author: yuzhih...@gmail.com):
Looks like this failure was related to the patch:
https://builds.apache.org/job/HBase-0.94-security/323/testReport/junit/org.apache.hadoop.hbase.master.metrics/TestMasterStatistics/testMasterStatistics/
@Ivan:
Mind taking a look ?
improve unit-test coverage of package org.apache.hadoop.hbase.master.metrics
(0.94)
---
Key: HBASE-8397
URL: https://issues.apache.org/jira/browse/HBASE-8397
Project: HBase
Issue Type: Test
Affects Versions: 0.94.8
Reporter: Ivan A. Veselovsky
Assignee: Ivan A. Veselovsky
Fix For: 0.94.13
Attachments: HBASE-8397-0.94--N3.patch, HBASE-8397-0.94--N5.patch,
HBASE-8397-0.94--N6.patch
The patch suggests new test that elevates unit-test coverage of package
org.apache.hadoop.hbase.master.metrics .
The patch is applicable to branch 0.94 only.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
