[jira] [Commented] (HBASE-17234) Allow alternate Readers/Writers; currently hardcoded

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15717631#comment-15717631
 ] 

Anoop Sam John commented on HBASE-17234:


So there should be 1-1 mapping btw writer and reader.. Ya this is any way 
there.. We need to make it more formalized?  Like how we have Encoder and 
Decoder tied together in one Codec

> Allow alternate Readers/Writers; currently hardcoded
> 
>
> Key: HBASE-17234
> URL: https://issues.apache.org/jira/browse/HBASE-17234
> Project: HBase
>  Issue Type: Task
>  Components: io
>Reporter: stack
> Attachments: HBASE-17234.master.001.patch
>
>
> Allow alternate HFile Reader and Writers. For Writers, we have WriterFactory 
> so you'd think it possible to supply a different Writer but in actuality, 
> WriterFactory is hardcoded.
> Read side does something else altogether complicated by fact that Reader 
> presumes trailer and that it has to take a Stream.
> Yeah, expecting someone would provide their own Reader/Writer is a little 
> unexpected but



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (HBASE-17231) Region#getCellCompartor sp?

[Anoop Sam John (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John updated HBASE-17231:
---
Comment: was deleted

(was: HBASE-10800 is committed only in 2.0 so we can correct the method name 
with out deprecation?  Seems branch-1 not having this change?)

> Region#getCellCompartor sp?
> ---
>
> Key: HBASE-17231
> URL: https://issues.apache.org/jira/browse/HBASE-17231
> Project: HBase
>  Issue Type: Bug
>Reporter: John Leach
>Assignee: John Leach
>Priority: Trivial
> Attachments: HBASE-17231.patch
>
>
> Region#getCellCompartor -> Region#getCellComparator



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17231) Region#getCellCompartor sp?

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15717624#comment-15717624
 ] 

Anoop Sam John commented on HBASE-17231:


HBASE-10800 is committed only in 2.0 so we can correct the method name with out 
deprecation?  Seems branch-1 not having this change?

> Region#getCellCompartor sp?
> ---
>
> Key: HBASE-17231
> URL: https://issues.apache.org/jira/browse/HBASE-17231
> Project: HBase
>  Issue Type: Bug
>Reporter: John Leach
>Assignee: John Leach
>Priority: Trivial
> Attachments: HBASE-17231.patch
>
>
> Region#getCellCompartor -> Region#getCellComparator



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17231) Region#getCellCompartor sp?

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15717625#comment-15717625
 ] 

Anoop Sam John commented on HBASE-17231:


HBASE-10800 is committed only in 2.0 so we can correct the method name with out 
deprecation?  Seems branch-1 not having this change?

> Region#getCellCompartor sp?
> ---
>
> Key: HBASE-17231
> URL: https://issues.apache.org/jira/browse/HBASE-17231
> Project: HBase
>  Issue Type: Bug
>Reporter: John Leach
>Assignee: John Leach
>Priority: Trivial
> Attachments: HBASE-17231.patch
>
>
> Region#getCellCompartor -> Region#getCellComparator



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17128) Find Cause of a Write Perf Regression in branch-1.2

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15717532#comment-15717532
 ] 

stack commented on HBASE-17128:
---

[~gbaecher] HBASE-17072 is pretty small change. This is on top of a 5.8/5.9? 
Let me know and I'll have a go at it...

> Find Cause of a Write Perf Regression in branch-1.2
> ---
>
> Key: HBASE-17128
> URL: https://issues.apache.org/jira/browse/HBASE-17128
> Project: HBase
>  Issue Type: Task
>Reporter: stack
>
> As reported by [~gbaecher] up on the mailing list, there is a regression in 
> 1.2. The regression is in a CDH version of 1.2 actually but the CDH hbase is 
> a near pure 1.2. This is a working issue to figure which of the below changes 
> brought on slower writes (The list comes from doing the following...git log 
> --oneline  
> remotes/origin/cdh5-1.2.0_5.8.0_dev..remotes/origin/cdh5-1.2.0_5.9.0_dev ... 
> I stripped the few CDH specific changes, packaging and tagging only, and then 
> made two groupings; candidates and the unlikelies):
> {code}
>   1 bbc6762 HBASE-16023 Fastpath for the FIFO rpcscheduler Adds an executor 
> that does balanced queue and fast path handing off requests directly to 
> waiting handlers if any present. Idea taken from Apace Kudu (incubating). See 
> https://gerr#
>   2 a260917 HBASE-16288 HFile intermediate block level indexes might recurse 
> forever creating multi TB files
>   3 5633281 HBASE-15811 Batch Get after batch Put does not fetch all Cells We 
> were not waiting on all executors in a batch to complete. The test for 
> no-more-executors was damaged by the 0.99/0.98.4 fix "HBASE-11403 Fix race 
> conditions aro#
>   4 780f720 HBASE-11625 - Verifies data before building HFileBlock. - Adds 
> HFileBlock.Header class which contains information about location of fields. 
> Testing: Adds CorruptedFSReaderImpl to TestChecksum. (Apekshit)
>   5 d735680 HBASE-12133 Add FastLongHistogram for metric computation (Yi Deng)
>   6 c4ee832 HBASE-15222 Use less contended classes for metrics
>   7
>   8 17320a4 HBASE-15683 Min latency in latency histograms are emitted as 
> Long.MAX_VALUE
>   9 283b39f HBASE-15396 Enhance mapreduce.TableSplit to add encoded region 
> name
>  10 39db592 HBASE-16195 Should not add chunk into chunkQueue if not using 
> chunk pool in HeapMemStoreLAB
>  11 5ff28b7 HBASE-16194 Should count in MSLAB chunk allocation into heap size 
> change when adding duplicate cells
>  12 5e3e0d2 HBASE-16318 fail build while rendering velocity template if 
> dependency license isn't in whitelist.
>  13 3ed66e3 HBASE-16318 consistently use the correct name for 'Apache 
> License, Version 2.0'
>  14 351832d HBASE-16340 exclude Xerces iplementation jars from coming in 
> transitively.
>  15 b6aa4be HBASE-16321 ensure no findbugs-jsr305
>  16 4f9dde7 HBASE-16317 revert all ESAPI changes
>  17 71b6a8a HBASE-16284 Unauthorized client can shutdown the cluster (Deokwoo 
> Han)
>  18 523753f HBASE-16450 Shell tool to dump replication queues
>  19 ca5f2ee HBASE-16379 [replication] Minor improvement to 
> replication/copy_tables_desc.rb
>  20 effd105 HBASE-16135 PeerClusterZnode under rs of removed peer may never 
> be deleted
>  21 a5c6610 HBASE-16319 Fix TestCacheOnWrite after HBASE-16288
>  22 1956bb0 HBASE-15808 Reduce potential bulk load intermediate space usage 
> and waste
>  23 031c54e HBASE-16096 Backport. Cleanly remove replication peers from 
> ZooKeeper.
>  24 60a3b12 HBASE-14963 Remove use of Guava Stopwatch from HBase client code 
> (Devaraj Das)
>  25 c7724fc HBASE-16207 can't restore snapshot without "Admin" permission
>  26 8322a0b HBASE-16227 [Shell] Column value formatter not working in scans. 
> Tested : manually using shell.
>  27 8f86658 HBASE-14818 user_permission does not list namespace permissions 
> (li xiang)
>  28 775cd21 HBASE-15465 userPermission returned by getUserPermission() for 
> the selected namespace does not have namespace set (li xiang)
>  29 8d85aff HBASE-16093 Fix splits failed before creating daughter regions 
> leave meta inconsistent
>  30 bc41317 HBASE-16140 bump owasp.esapi from 2.1.0 to 2.1.0.1
>  31 6fc70cd HBASE-16035 Nested AutoCloseables might not all get closed (Sean 
> Mackrory)
>  32 fe28fe84 HBASE-15891. Closeable resources potentially not getting closed 
> if exception is thrown.
>  33 1d2bf3c HBASE-14644 Region in transition metric is broken -- addendum 
> (Huaxiang Sun)
>  34 fd5f56c HBASE-16056 Procedure v2 - fix master crash for FileNotFound
>  35 10cd038 HBASE-16034 Fix ProcedureTestingUtility#LoadCounter.setMaxProcId()
>  36 dae4db4 HBASE-15872 Split TestWALProcedureStore
>  37 e638d86 HBASE-14644 Region in transition metric is broken (Huaxiang Sun)
>  38 f01b01d HBASE-15496 Throw RowTooBigException only for user scan/get 
> (Guanghao Zhang)
>  39 cc0ce66 HBASE-15746 Remove extra 

[jira] [Commented] (HBASE-17177) Compaction can break the region/row level atomic when scan even if we pass mvcc to client

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15717525#comment-15717525
 ] 

stack commented on HBASE-17177:
---

I like your idea that we do keep deleted cells if a compaction runs inside the 
scanner timeout after open (and a minor cannot graduate to major).

How we make this cryptic behavior 'obvious' to the operator?

> Compaction can break the region/row level atomic when scan even if we pass 
> mvcc to client
> -
>
> Key: HBASE-17177
> URL: https://issues.apache.org/jira/browse/HBASE-17177
> Project: HBase
>  Issue Type: Sub-task
>  Components: scan
>Reporter: Duo Zhang
> Fix For: 2.0.0, 1.4.0
>
>
> We know that major compaction will actually delete the cells which are 
> deleted by a delete marker. In order to give a consistent view for a scan, we 
> need to use a map to track the read points for all scanners for a region, and 
> the smallest one will be used for a compaction. For all delete markers whose 
> mvcc is greater than this value, we will not use it to delete other cells.
> And the problem for a scan restart after region move is that, the new RS does 
> not have the information of the scanners opened at the old RS before the 
> client sends scan requests to the new RS which means the read points map is 
> incomplete and the smallest read point maybe greater than the correct value. 
> So if a major compaction happens at that time, it may delete some cells which 
> should be kept.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17177) Compaction can break the region/row level atomic when scan even if we pass mvcc to client

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15717522#comment-15717522
 ] 

stack commented on HBASE-17177:
---

We already note if a file is product of a major compaction. As you suggest, we 
could add readpoint to the hfile metadata.

> Compaction can break the region/row level atomic when scan even if we pass 
> mvcc to client
> -
>
> Key: HBASE-17177
> URL: https://issues.apache.org/jira/browse/HBASE-17177
> Project: HBase
>  Issue Type: Sub-task
>  Components: scan
>Reporter: Duo Zhang
> Fix For: 2.0.0, 1.4.0
>
>
> We know that major compaction will actually delete the cells which are 
> deleted by a delete marker. In order to give a consistent view for a scan, we 
> need to use a map to track the read points for all scanners for a region, and 
> the smallest one will be used for a compaction. For all delete markers whose 
> mvcc is greater than this value, we will not use it to delete other cells.
> And the problem for a scan restart after region move is that, the new RS does 
> not have the information of the scanners opened at the old RS before the 
> client sends scan requests to the new RS which means the read points map is 
> incomplete and the smallest read point maybe greater than the correct value. 
> So if a major compaction happens at that time, it may delete some cells which 
> should be kept.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17069) RegionServer writes invalid META entries for split daughters in some circumstances

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15717492#comment-15717492
 ] 

stack commented on HBASE-17069:
---

On real hw w/ ubuntu 14, I've looped 1B three times now and have not seen this 
issue. Let me up the size for the w/e.

> RegionServer writes invalid META entries for split daughters in some 
> circumstances
> --
>
> Key: HBASE-17069
> URL: https://issues.apache.org/jira/browse/HBASE-17069
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.2.4
>Reporter: Andrew Purtell
>Priority: Critical
> Attachments: daughter_1_d55ef81c2f8299abbddfce0445067830.log, 
> daughter_2_08629d59564726da2497f70451aafcdb.log, logs.tar.gz, 
> parent-393d2bfd8b1c52ce08540306659624f2.log
>
>
> I have been seeing frequent ITBLL failures testing various versions of 1.2.x. 
> Over the lifetime of 1.2.x the following issues have been fixed:
> - HBASE-15315 (Remove always set super user call as high priority)
> - HBASE-16093 (Fix splits failed before creating daughter regions leave meta 
> inconsistent)
> And this one is pending:
> - HBASE-17044 (Fix merge failed before creating merged region leaves meta 
> inconsistent)
> I can apply all of the above to branch-1.2 and still see this failure: 
> *The life of stillborn region d55ef81c2f8299abbddfce0445067830*
> *Master sees SPLITTING_NEW*
> {noformat}
> 2016-11-08 04:23:21,186 INFO  [AM.ZK.Worker-pool2-t82] master.RegionStates: 
> Transition null to {d55ef81c2f8299abbddfce0445067830 state=SPLITTING_NEW, 
> ts=1478579001186, server=node-3.cluster,16020,1478578389506}
> {noformat}
> *The RegionServer creates it*
> {noformat}
> 2016-11-08 04:23:26,035 INFO  
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created 
> cacheConfig for GomnU: blockCache=LruBlockCache{blockCount=34, 
> currentSize=14996112, freeSize=12823716208, maxSize=12838712320, 
> heapSize=14996112, minSize=12196776960, minFactor=0.95, multiSize=6098388480, 
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25}, 
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, 
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, 
> prefetchOnOpen=false
> 2016-11-08 04:23:26,038 INFO  
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created 
> cacheConfig for big: blockCache=LruBlockCache{blockCount=34, 
> currentSize=14996112, freeSize=12823716208, maxSize=12838712320, 
> heapSize=14996112, minSize=12196776960, minFactor=0.95, multiSize=6098388480, 
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25}, 
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, 
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, 
> prefetchOnOpen=false
> 2016-11-08 04:23:26,442 INFO  
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created 
> cacheConfig for meta: blockCache=LruBlockCache{blockCount=63, 
> currentSize=17187656, freeSize=12821524664, maxSize=12838712320, 
> heapSize=17187656, minSize=12196776960, minFactor=0.95, multiSize=6098388480, 
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25}, 
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, 
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, 
> prefetchOnOpen=false
> 2016-11-08 04:23:26,713 INFO  
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created 
> cacheConfig for nwmrW: blockCache=LruBlockCache{blockCount=96, 
> currentSize=19178440, freeSize=12819533880, maxSize=12838712320, 
> heapSize=19178440, minSize=12196776960, minFactor=0.95, multiSize=6098388480, 
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25}, 
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, 
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, 
> prefetchOnOpen=false
> 2016-11-08 04:23:26,715 INFO  
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created 
> cacheConfig for piwbr: blockCache=LruBlockCache{blockCount=96, 
> currentSize=19178440, freeSize=12819533880, maxSize=12838712320, 
> heapSize=19178440, minSize=12196776960, minFactor=0.95, multiSize=6098388480, 
> multiFactor=0.5, singleSize=3049194240, singleFactor=0.25}, 
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, 
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, 
> prefetchOnOpen=false
> 2016-11-08 04:23:26,717 INFO  
> [StoreOpener-d55ef81c2f8299abbddfce0445067830-1] hfile.CacheConfig: Created 
> cacheConfig for tiny: blockCache=LruBlockCache{blockCount=96, 
> currentSize=19178440, freeSize=12819533880, maxSize=12838712320, 
> heapSize=19178440, minSize=12196776960, 

[jira] [Commented] (HBASE-17246) TestSerialReplication#testRegionMerge fails in master branch

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15717471#comment-15717471
 ] 

Hadoop QA commented on HBASE-17246:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
36s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 19s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
33m 54s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 12s 
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
9s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 10s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12841613/HBASE-17246.01.patch |
| JIRA Issue | HBASE-17246 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 8d923bff1312 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 7775fed |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4771/testReport/ |
| modules | C: hbase-client U: hbase-client |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4771/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> TestSerialReplication#testRegionMerge fails in master branch
> 
>
> Key: HBASE-17246
> URL: https://issues.apache.org/jira/browse/HBASE-17246
> Project: HBase
>  Issue Type: Test
>Reporter: Ted Yu
>Assignee: Phil Yang
> 

[jira] [Commented] (HBASE-17234) Allow alternate Readers/Writers; currently hardcoded

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15717410#comment-15717410
 ] 

stack commented on HBASE-17234:
---

Chatting w/ [~mbertozzi] today, he makes a good point. Factory for Writers make 
sense. At read time, no. At read time, factory makes no sense. At read time you 
need to peek at the file to see what Reader implementation to use. This means 
that whatever the Writer, they always write a trailer aways of the same format. 
A parse of the format figures which Reader impl to use.

> Allow alternate Readers/Writers; currently hardcoded
> 
>
> Key: HBASE-17234
> URL: https://issues.apache.org/jira/browse/HBASE-17234
> Project: HBase
>  Issue Type: Task
>  Components: io
>Reporter: stack
> Attachments: HBASE-17234.master.001.patch
>
>
> Allow alternate HFile Reader and Writers. For Writers, we have WriterFactory 
> so you'd think it possible to supply a different Writer but in actuality, 
> WriterFactory is hardcoded.
> Read side does something else altogether complicated by fact that Reader 
> presumes trailer and that it has to take a Stream.
> Yeah, expecting someone would provide their own Reader/Writer is a little 
> unexpected but



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17246) TestSerialReplication#testRegionMerge fails in master branch

[Phil Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phil Yang updated HBASE-17246:
--
Status: Patch Available  (was: Open)

> TestSerialReplication#testRegionMerge fails in master branch
> 
>
> Key: HBASE-17246
> URL: https://issues.apache.org/jira/browse/HBASE-17246
> Project: HBase
>  Issue Type: Test
>Reporter: Ted Yu
>Assignee: Phil Yang
> Attachments: HBASE-17246.01.patch, 
> org.apache.hadoop.hbase.replication.TestSerialReplication-output.txt
>
>
> TestSerialReplication#testRegionMerge fails intermittently in master branch.
> {code}
> testRegionMerge(org.apache.hadoop.hbase.replication.TestSerialReplication)  
> Time elapsed: 15.193 sec  <<< FAILURE!
> java.lang.AssertionError: expected:<9> but was:<4>
>   at 
> org.apache.hadoop.hbase.replication.TestSerialReplication.testRegionMerge(TestSerialReplication.java:305)
> {code}
> I can reproduce the test failure locally.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HBASE-17246) TestSerialReplication#testRegionMerge fails in master branch

[Phil Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Phil Yang reassigned HBASE-17246:
-

Assignee: Phil Yang

> TestSerialReplication#testRegionMerge fails in master branch
> 
>
> Key: HBASE-17246
> URL: https://issues.apache.org/jira/browse/HBASE-17246
> Project: HBase
>  Issue Type: Test
>Reporter: Ted Yu
>Assignee: Phil Yang
> Attachments: 
> org.apache.hadoop.hbase.replication.TestSerialReplication-output.txt
>
>
> TestSerialReplication#testRegionMerge fails intermittently in master branch.
> {code}
> testRegionMerge(org.apache.hadoop.hbase.replication.TestSerialReplication)  
> Time elapsed: 15.193 sec  <<< FAILURE!
> java.lang.AssertionError: expected:<9> but was:<4>
>   at 
> org.apache.hadoop.hbase.replication.TestSerialReplication.testRegionMerge(TestSerialReplication.java:305)
> {code}
> I can reproduce the test failure locally.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16941) FavoredNodes - Split/Merge code paths

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15717323#comment-15717323
 ] 

Hadoop QA commented on HBASE-16941:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 31s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
30s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
29s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
28s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
4s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 55s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
28m 20s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 43s 
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 103m 31s 
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
29s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 151m 4s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12841604/HBASE-16941.master.011.patch
 |
| JIRA Issue | HBASE-16941 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 9c06e285b2dc 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
 |
| git revision | master / 7775fed |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4770/testReport/ |
| modules | C: hbase-common hbase-server U: . |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4770/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> FavoredNodes - Split/Merge code paths
> 

[jira] [Commented] (HBASE-16941) FavoredNodes - Split/Merge code paths

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15717271#comment-15717271
 ] 

Hadoop QA commented on HBASE-16941:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 20s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
29m 1s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 49s 
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 84m 12s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
33s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 130m 43s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestHRegion |
| Timed out junit tests | org.apache.hadoop.hbase.client.TestMetaWithReplicas |
|   | org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor |
|   | org.apache.hadoop.hbase.client.TestLeaseRenewal |
|   | org.apache.hadoop.hbase.client.TestTableSnapshotScanner |
|   | org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12841603/HBASE-16941.master.010.patch
 |
| JIRA Issue | HBASE-16941 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 7940e7ef47f5 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 7775fed |
| 

[jira] [Updated] (HBASE-17077) Don't copy the replication queue belonging to the peer which has been deleted

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-17077:
---
Fix Version/s: 0.98.24

> Don't copy the replication queue belonging to the peer which has been deleted
> -
>
> Key: HBASE-17077
> URL: https://issues.apache.org/jira/browse/HBASE-17077
> Project: HBase
>  Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Fix For: 2.0.0, 1.4.0, 0.98.24
>
> Attachments: HBASE-17077-v1.patch, HBASE-17077.patch
>
>
> When a region server is dead, then other live region servers will transfer 
> the dead rs's replication queue to their own queue. Now the live rs first 
> copy the wals queue to its own znode, then create a new replication source to 
> replicate the wals. But if the queue belong to a peer has been deleted, it 
> copy the queue, too. The current steps is:
> 1. copy the queue to its own znode
> 2. found the queue belong to a peer has been deleted
> 3. remove the queue and don't create a new replication source for it
> There is a small improvement. The live region server doesn't need to copy the 
> queue to its own znode. The new steps is:
> 1. found the queue belong to a peer has been deleted
> 2. remove the queue directly instead of copy it



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16941) FavoredNodes - Split/Merge code paths

[Thiruvel Thirumoolan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thiruvel Thirumoolan updated HBASE-16941:
-
Attachment: HBASE-16941.master.011.patch

> FavoredNodes - Split/Merge code paths
> -
>
> Key: HBASE-16941
> URL: https://issues.apache.org/jira/browse/HBASE-16941
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Thiruvel Thirumoolan
>Assignee: Thiruvel Thirumoolan
> Fix For: 2.0.0
>
> Attachments: HBASE-16941.master.001.patch, 
> HBASE-16941.master.002.patch, HBASE-16941.master.003.patch, 
> HBASE-16941.master.004.patch, HBASE-16941.master.005.patch, 
> HBASE-16941.master.006.patch, HBASE-16941.master.007.patch, 
> HBASE-16941.master.008.patch, HBASE-16941.master.009.patch, 
> HBASE-16941.master.010.patch, HBASE-16941.master.011.patch
>
>
> This jira is to deal with the split/merge logic discussed as part of 
> HBASE-15532. The design document can be seen at HBASE-15531. The specific 
> changes are:
> Split and merged regions should inherit favored node information from parent 
> regions. For splits also include some randomness so even if there are 
> subsequent splits, the regions will be more or less distributed. For split, 
> we include 2 FN from the parent and generate one random node.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16941) FavoredNodes - Split/Merge code paths

[Thiruvel Thirumoolan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thiruvel Thirumoolan updated HBASE-16941:
-
Attachment: HBASE-16941.master.010.patch

> FavoredNodes - Split/Merge code paths
> -
>
> Key: HBASE-16941
> URL: https://issues.apache.org/jira/browse/HBASE-16941
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Thiruvel Thirumoolan
>Assignee: Thiruvel Thirumoolan
> Fix For: 2.0.0
>
> Attachments: HBASE-16941.master.001.patch, 
> HBASE-16941.master.002.patch, HBASE-16941.master.003.patch, 
> HBASE-16941.master.004.patch, HBASE-16941.master.005.patch, 
> HBASE-16941.master.006.patch, HBASE-16941.master.007.patch, 
> HBASE-16941.master.008.patch, HBASE-16941.master.009.patch, 
> HBASE-16941.master.010.patch
>
>
> This jira is to deal with the split/merge logic discussed as part of 
> HBASE-15532. The design document can be seen at HBASE-15531. The specific 
> changes are:
> Split and merged regions should inherit favored node information from parent 
> regions. For splits also include some randomness so even if there are 
> subsequent splits, the regions will be more or less distributed. For split, 
> we include 2 FN from the parent and generate one random node.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15902) Scan Object

[Sudeep Sunthankar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudeep Sunthankar updated HBASE-15902:
--
Attachment: HBASE-15902.HBASE-14850.v4.patch

This patch consits of the below fix based on the last feedback.

# Added Family() and HasFamilies() methods in Scan
# Fixed bug of FamilyMap not being copied in Get copy constructor and 
assignment operator.
# Added tests to verify copying (deep copy) of family map is proper using the 
below code
{code}
 family_map_.insert(scan.family_map_.begin(), scan.family_map_.end());
{code}

Thanks

> Scan Object
> ---
>
> Key: HBASE-15902
> URL: https://issues.apache.org/jira/browse/HBASE-15902
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Sudeep Sunthankar
>Assignee: Sudeep Sunthankar
> Attachments: HBASE-15902.HBASE-14850.patch, 
> HBASE-15902.HBASE-14850.v2.patch, HBASE-15902.HBASE-14850.v3.patch, 
> HBASE-15902.HBASE-14850.v4.patch
>
>
> Patch for creating Scan objects. Scan objects thus created can be used by 
> Table implementation to fetch results for a given row.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (HBASE-15902) Scan Object

[Sudeep Sunthankar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudeep Sunthankar updated HBASE-15902:
--
Comment: was deleted

(was: This patch consits of the below fix based on the last feedback.

# Added Family() and HasFamilies() methods in Scan
# Fixed bug of FamilyMap not being copied in Get copy constructor and 
assignment operator.
# Added tests to verify copying (deep copy) of family map is proper using the 
below code
{code}
 family_map_.insert(scan.family_map_.begin(), scan.family_map_.end());
{code}

Thanks)

> Scan Object
> ---
>
> Key: HBASE-15902
> URL: https://issues.apache.org/jira/browse/HBASE-15902
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Sudeep Sunthankar
>Assignee: Sudeep Sunthankar
> Attachments: HBASE-15902.HBASE-14850.patch, 
> HBASE-15902.HBASE-14850.v2.patch, HBASE-15902.HBASE-14850.v3.patch
>
>
> Patch for creating Scan objects. Scan objects thus created can be used by 
> Table implementation to fetch results for a given row.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17194) Assign the new region to the idle server after splitting

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15717016#comment-15717016
 ] 

Hudson commented on HBASE-17194:


SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2061 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/2061/])
HBASE-17194 Assign the new region to the idle server after splitting (stack: 
rev 7775feda05b0db63178c81910946adfec4c4c41f)
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestBaseLoadBalancer.java


> Assign the new region to the idle server after splitting
> 
>
> Key: HBASE-17194
> URL: https://issues.apache.org/jira/browse/HBASE-17194
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17194.v0.patch, HBASE-17194.v1.patch, 
> HBASE-17194.v2.patch, HBASE-17194.v3.patch, HBASE-17194.v4.patch, 
> HBASE-17194.v4.patch, evaluation-v0.png, tests.xlsx
>
>
> The new regions are assigned to the random servers after splitting, but there 
> always are some idle servers which don’t be assigned any regions on the new 
> cluster. It is a bad start of load balance, hence we should give priority to 
> the idle servers for assignment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17184) Code cleanup of LruBlockCache

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15717014#comment-15717014
 ] 

Hudson commented on HBASE-17184:


SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2061 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/2061/])
HBASE-17184 Revert "Code cleanup of LruBlockCache" Revert "Revert "Code (stack: 
rev 4a20361f1a235e0365c7923e64d20b28166cfd4b)
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java


> Code cleanup of LruBlockCache
> -
>
> Key: HBASE-17184
> URL: https://issues.apache.org/jira/browse/HBASE-17184
> Project: HBase
>  Issue Type: Improvement
>Reporter: Lars Francke
>Assignee: Lars Francke
>Priority: Trivial
> Fix For: 2.0.0
>
> Attachments: HBASE-17184.master.001.patch
>
>
> Initially I only wanted to fix a badly worded Log message but found a bunch 
> of code convention violations etc. so I decided to clean up the class.
> It is only whitespace changes with the exception of the one log message ("too 
> many").



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16409) Row key for bad row should be properly delimited in VerifyReplication

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-16409:
---
Fix Version/s: 0.98.24

> Row key for bad row should be properly delimited in VerifyReplication
> -
>
> Key: HBASE-16409
> URL: https://issues.apache.org/jira/browse/HBASE-16409
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
>  Labels: replication
> Fix For: 2.0.0, 1.4.0, 0.98.24
>
> Attachments: 16409.addendum, 16409.addendum2, 16409.v1.txt, 
> 16409.v2.txt
>
>
> In logFailRowAndIncreaseCounter():
> {code}
>   LOG.error(counter.toString() + ", rowkey=" + 
> Bytes.toString(row.getRow()));
> {code}
> In some cases, the row key has trailing whitespace(s).
> The whitespace wouldn't be easily observable in the above log.
> A delimiter should be put toward the end of the log line.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16772) Add verbose option to VerifyReplication for logging good rows

[Andrew Purtell (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-16772:
---
Fix Version/s: 0.98.24

> Add verbose option to VerifyReplication for logging good rows
> -
>
> Key: HBASE-16772
> URL: https://issues.apache.org/jira/browse/HBASE-16772
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Fix For: 2.0.0, 1.4.0, 0.98.24
>
> Attachments: 16772.v1.txt, 16772.v2.txt, 16772.v3.txt
>
>
> Sometimes user wants to perform further verification when VerifyReplication 
> reports all good rows.
> This issue is to add a configuration ('--verbose=true' default 'false') to 
> facilitate more thorough verification.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-17247) Questions Operators Ask

[stack (JIRA)]

stack created HBASE-17247:
-

 Summary: Questions Operators Ask
 Key: HBASE-17247
 URL: https://issues.apache.org/jira/browse/HBASE-17247
 Project: HBase
  Issue Type: Umbrella
Reporter: stack


Keep in here list of questions operators have difficulty answering. In 
subtasks, provide metrics/answers to answer them.

Here is the kickoff:

bq. What are the requests in the 99.99th percentile doing? Why are they so slow?

Lets fill in others in here.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17246) TestSerialReplication#testRegionMerge fails in master branch

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716868#comment-15716868
 ] 

Ted Yu commented on HBASE-17246:


As of trunk build #2060, this test has been blacklisted:
{code}
'EXCLUDES=**/TestSnapshotCloneIndependence.java,**/TestMobExportSnapshot.java,**/TestHRegion.java,**/TestStochasticLoadBalancer2.java,**/TestQuotaThrottle.java,**/TestMobSecureExportSnapshot.java,**/TestSerialReplication.java,**/TestVisibilityLabelsReplication.java,**/TestReplicasClient.java,**/TestBlockEvictionFromClient.java,**/TestHRegionWithInMemoryFlush.java,**/TestSecureExportSnapshot.java,**/TestAsyncLogRolling.java,**/TestExportSnapshot.java,**/TestReplicationWithTags.java,**/TestVisibilityLabelReplicationWithExpAsString.java'
{code}

> TestSerialReplication#testRegionMerge fails in master branch
> 
>
> Key: HBASE-17246
> URL: https://issues.apache.org/jira/browse/HBASE-17246
> Project: HBase
>  Issue Type: Test
>Reporter: Ted Yu
> Attachments: 
> org.apache.hadoop.hbase.replication.TestSerialReplication-output.txt
>
>
> TestSerialReplication#testRegionMerge fails intermittently in master branch.
> {code}
> testRegionMerge(org.apache.hadoop.hbase.replication.TestSerialReplication)  
> Time elapsed: 15.193 sec  <<< FAILURE!
> java.lang.AssertionError: expected:<9> but was:<4>
>   at 
> org.apache.hadoop.hbase.replication.TestSerialReplication.testRegionMerge(TestSerialReplication.java:305)
> {code}
> I can reproduce the test failure locally.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17246) TestSerialReplication#testRegionMerge fails in master branch

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-17246:
---
Attachment: 
org.apache.hadoop.hbase.replication.TestSerialReplication-output.txt

> TestSerialReplication#testRegionMerge fails in master branch
> 
>
> Key: HBASE-17246
> URL: https://issues.apache.org/jira/browse/HBASE-17246
> Project: HBase
>  Issue Type: Test
>Reporter: Ted Yu
> Attachments: 
> org.apache.hadoop.hbase.replication.TestSerialReplication-output.txt
>
>
> TestSerialReplication#testRegionMerge fails intermittently in master branch.
> {code}
> testRegionMerge(org.apache.hadoop.hbase.replication.TestSerialReplication)  
> Time elapsed: 15.193 sec  <<< FAILURE!
> java.lang.AssertionError: expected:<9> but was:<4>
>   at 
> org.apache.hadoop.hbase.replication.TestSerialReplication.testRegionMerge(TestSerialReplication.java:305)
> {code}
> I can reproduce the test failure locally.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17241) Avoid compacting already compacted mob files with _del files

[huaxiang sun (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716856#comment-15716856
 ] 

huaxiang sun commented on HBASE-17241:
--

Thanks Ted. The failure is caused by the patch, need to change the test case as 
well. Will upload a new patch.

> Avoid compacting already compacted  mob files with _del files
> -
>
> Key: HBASE-17241
> URL: https://issues.apache.org/jira/browse/HBASE-17241
> Project: HBase
>  Issue Type: Improvement
>  Components: mob
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
> Attachments: HBASE-17241.master.001.patch
>
>
> Today if there is only one file in the partition, and there is no _del files, 
> the file is skipped. With del file, the current logic is to compact the 
> already-compacted file with _del file. Let's say there is one mob file 
> regionA20161101***, which was compacted. On 12/1/2016, there is _del file 
> regionB20161201**_del, mob compaction kicks in, regionA20161101*** is less 
> than the threshold, and it is picked for compaction. Since there is a _del 
> file, regionA20161101 and regionB20161201***_del are compacted into 
> regionA20161101**_1 . After that, regionB20161201**_del cannot be deleted 
> since it is not a allFile compaction. The next mob compaction, 
> regionA20161101**_1 and regionB20161201**_del will be picked up again and be 
> compacted into regionA20161101***_2. So in this case, it will cause more 
> unnecessary IOs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-17246) TestSerialReplication#testRegionMerge fails in master branch

[Ted Yu (JIRA)]

Ted Yu created HBASE-17246:
--

 Summary: TestSerialReplication#testRegionMerge fails in master 
branch
 Key: HBASE-17246
 URL: https://issues.apache.org/jira/browse/HBASE-17246
 Project: HBase
  Issue Type: Test
Reporter: Ted Yu


TestSerialReplication#testRegionMerge fails intermittently in master branch.
{code}
testRegionMerge(org.apache.hadoop.hbase.replication.TestSerialReplication)  
Time elapsed: 15.193 sec  <<< FAILURE!
java.lang.AssertionError: expected:<9> but was:<4>
at 
org.apache.hadoop.hbase.replication.TestSerialReplication.testRegionMerge(TestSerialReplication.java:305)
{code}
I can reproduce the test failure locally.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17241) Avoid compacting already compacted mob files with _del files

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716847#comment-15716847
 ] 

Ted Yu commented on HBASE-17241:


{code}
90  public int getFileNumbers () {
{code}
Rename the method getNumberOfFiles or getFileCount

Please check the test failure of TestPartitionedMobCompactor

> Avoid compacting already compacted  mob files with _del files
> -
>
> Key: HBASE-17241
> URL: https://issues.apache.org/jira/browse/HBASE-17241
> Project: HBase
>  Issue Type: Improvement
>  Components: mob
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
> Attachments: HBASE-17241.master.001.patch
>
>
> Today if there is only one file in the partition, and there is no _del files, 
> the file is skipped. With del file, the current logic is to compact the 
> already-compacted file with _del file. Let's say there is one mob file 
> regionA20161101***, which was compacted. On 12/1/2016, there is _del file 
> regionB20161201**_del, mob compaction kicks in, regionA20161101*** is less 
> than the threshold, and it is picked for compaction. Since there is a _del 
> file, regionA20161101 and regionB20161201***_del are compacted into 
> regionA20161101**_1 . After that, regionB20161201**_del cannot be deleted 
> since it is not a allFile compaction. The next mob compaction, 
> regionA20161101**_1 and regionB20161201**_del will be picked up again and be 
> compacted into regionA20161101***_2. So in this case, it will cause more 
> unnecessary IOs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15902) Scan Object

[Sudeep Sunthankar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudeep Sunthankar updated HBASE-15902:
--
Attachment: (was: HBASE-15902.HBASE-14850.v4.patch)

> Scan Object
> ---
>
> Key: HBASE-15902
> URL: https://issues.apache.org/jira/browse/HBASE-15902
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Sudeep Sunthankar
>Assignee: Sudeep Sunthankar
> Attachments: HBASE-15902.HBASE-14850.patch, 
> HBASE-15902.HBASE-14850.v2.patch, HBASE-15902.HBASE-14850.v3.patch
>
>
> Patch for creating Scan objects. Scan objects thus created can be used by 
> Table implementation to fetch results for a given row.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17240) ImportTsv encounters ClassNotFoundException for MasterProtos$MasterService$BlockingInterface

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716834#comment-15716834
 ] 

Hadoop QA commented on HBASE-17240:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 3s 
{color} | {color:blue} The patch file was not named according to hbase's naming 
conventions. Please see 
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for 
instructions. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
8s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
25m 20s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 101m 30s 
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
15s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 138m 47s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12841557/17240.v2.txt |
| JIRA Issue | HBASE-17240 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux b350aa907724 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 7775fed |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4768/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4768/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> ImportTsv encounters ClassNotFoundException for 
> MasterProtos$MasterService$BlockingInterface

[jira] [Commented] (HBASE-16337) Removing peers seem to be leaving spare queues

[C. Albert (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716830#comment-15716830
 ] 

C. Albert commented on HBASE-16337:
---

This issue looks like a duplicate of HBASE-16336.

> Removing peers seem to be leaving spare queues
> --
>
> Key: HBASE-16337
> URL: https://issues.apache.org/jira/browse/HBASE-16337
> Project: HBase
>  Issue Type: Sub-task
>  Components: Replication
>Reporter: Joseph
>
> I have been running IntegrationTestReplication repeatedly with the backported 
> Replication Table changes. Every other iteration of the test fails with, but 
> these queues should have been deleted when we removed the peers. I believe 
> this may be related to HBASE-16096, HBASE-16208, or HBASE-16081.
> 16/08/02 08:36:07 ERROR util.AbstractHBaseTool: Error running command-line 
> tool
> org.apache.hadoop.hbase.replication.ReplicationException: undeleted queue for 
> peerId: TestPeer, replicator: 
> hbase4124.ash2.facebook.com,16020,1470150251042, queueId: TestPeer
>   at 
> org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.checkQueuesDeleted(ReplicationPeersZKImpl.java:544)
>   at 
> org.apache.hadoop.hbase.replication.ReplicationPeersZKImpl.addPeer(ReplicationPeersZKImpl.java:127)
>   at 
> org.apache.hadoop.hbase.client.replication.ReplicationAdmin.addPeer(ReplicationAdmin.java:200)
>   at 
> org.apache.hadoop.hbase.test.IntegrationTestReplication$VerifyReplicationLoop.setupTablesAndReplication(IntegrationTestReplication.java:239)
>   at 
> org.apache.hadoop.hbase.test.IntegrationTestReplication$VerifyReplicationLoop.run(IntegrationTestReplication.java:325)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>   at 
> org.apache.hadoop.hbase.test.IntegrationTestReplication.runTestFromCommandLine(IntegrationTestReplication.java:418)
>   at 
> org.apache.hadoop.hbase.IntegrationTestBase.doWork(IntegrationTestBase.java:134)
>   at 
> org.apache.hadoop.hbase.util.AbstractHBaseTool.run(AbstractHBaseTool.java:112)
>   at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>   at 
> org.apache.hadoop.hbase.test.IntegrationTestReplication.main(IntegrationTestReplication.java:424)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15902) Scan Object

[Sudeep Sunthankar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716828#comment-15716828
 ] 

Sudeep Sunthankar commented on HBASE-15902:
---

Thanks Enis,

The below statement does a deep copy of the family map. I have added test cases 
to verify the same in the latest patch.
{code}
 family_map_.insert(scan.family_map_.begin(), scan.family_map_.end());
{code}

> Scan Object
> ---
>
> Key: HBASE-15902
> URL: https://issues.apache.org/jira/browse/HBASE-15902
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Sudeep Sunthankar
>Assignee: Sudeep Sunthankar
> Attachments: HBASE-15902.HBASE-14850.patch, 
> HBASE-15902.HBASE-14850.v2.patch, HBASE-15902.HBASE-14850.v3.patch, 
> HBASE-15902.HBASE-14850.v4.patch
>
>
> Patch for creating Scan objects. Scan objects thus created can be used by 
> Table implementation to fetch results for a given row.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-15902) Scan Object

[Sudeep Sunthankar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-15902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sudeep Sunthankar updated HBASE-15902:
--
Attachment: HBASE-15902.HBASE-14850.v4.patch

This patch consits of the below fix based on the last feedback.

# Added Family() and HasFamilies() methods in Scan
# Fixed bug of FamilyMap not being copied in Get copy constructor and 
assignment operator.
# Added tests to verify copying (deep copy) of family map is proper using the 
below code
{code}
 family_map_.insert(scan.family_map_.begin(), scan.family_map_.end());
{code}

Thanks

> Scan Object
> ---
>
> Key: HBASE-15902
> URL: https://issues.apache.org/jira/browse/HBASE-15902
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Sudeep Sunthankar
>Assignee: Sudeep Sunthankar
> Attachments: HBASE-15902.HBASE-14850.patch, 
> HBASE-15902.HBASE-14850.v2.patch, HBASE-15902.HBASE-14850.v3.patch, 
> HBASE-15902.HBASE-14850.v4.patch
>
>
> Patch for creating Scan objects. Scan objects thus created can be used by 
> Table implementation to fetch results for a given row.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17245) replace HTableDescriptor#htd.getColumnFamilies().length with a low-cost implementation

[huaxiang sun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

huaxiang sun updated HBASE-17245:
-
Summary: replace HTableDescriptor#htd.getColumnFamilies().length with a 
low-cost implementation  (was: Optimize 
HTableDescriptor#htd.getColumnFamilies().length )

> replace HTableDescriptor#htd.getColumnFamilies().length with a low-cost 
> implementation
> --
>
> Key: HBASE-17245
> URL: https://issues.apache.org/jira/browse/HBASE-17245
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
>
> Found quite a few places doing something like
> {code}
> if (htd.getColumnFamilies().length == 0) {
> {code}
> HTableDescriptor#htd.getColumnFamilies() does a few memory allocations. The 
> purpose is to get the family number, it can be replaced with 
> {code}
> +  public int getColumnFamilyCount() {
>  +return families.size();
>  +  }
> {code}
> in HTableDescriptor which does not need these memory allocations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-17245) Optimize HTableDescriptor#htd.getColumnFamilies().length

[huaxiang sun (JIRA)]

huaxiang sun created HBASE-17245:


 Summary: Optimize HTableDescriptor#htd.getColumnFamilies().length 
 Key: HBASE-17245
 URL: https://issues.apache.org/jira/browse/HBASE-17245
 Project: HBase
  Issue Type: Improvement
Affects Versions: 2.0.0
Reporter: huaxiang sun
Assignee: huaxiang sun
Priority: Minor


Found quite a few places doing something like

{code}
if (htd.getColumnFamilies().length == 0) {
{code}

HTableDescriptor#htd.getColumnFamilies() does a few memory allocations. The 
purpose is to get the family number, it can be replaced with 
{code}
+  public int getColumnFamilyCount() {
 +return families.size();
 +  }
{code}
in HTableDescriptor which does not need these memory allocations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17240) ImportTsv encounters ClassNotFoundException for MasterProtos$MasterService$BlockingInterface

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716786#comment-15716786
 ] 

Jerry He commented on HBASE-17240:
--

Oh, the shaded proto class in already in there.
{noformat}
   org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos.class, // 
hbase-protocol-shaded
+  org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos.class, // 
hbase-protocol-shaded
{noformat}

HBASE-17166 just added it. No need to add it again.
ImportTsv.createSubmittableJob() already calls 
TableMapReduceUtil.addDependencyJars()

Maybe an old build? 

> ImportTsv encounters ClassNotFoundException for 
> MasterProtos$MasterService$BlockingInterface
> 
>
> Key: HBASE-17240
> URL: https://issues.apache.org/jira/browse/HBASE-17240
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
> Fix For: 2.0.0
>
> Attachments: 17240.v1.txt, 17240.v2.txt
>
>
> [~romil.choksi] reported the following problem.
> With command:
> {code}
> hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.separator=, 
> -Dimporttsv.bulk.output=output -Dimporttsv.columns=HBASE_ROW_KEY,f:count 
> wordcount word_count.csv
> {code}
> The following error showed up:
> {code}
> 2016-11-29 06:39:48,861 INFO  [main] mapreduce.Job: Task Id : 
> attempt_1479850535804_0004_m_00_2, Status : FAILED
> Error: java.lang.ClassNotFoundException: 
> org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$BlockingInterface
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:264)
> at 
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:225)
> at 
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:122)
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver.init(DefaultVisibilityExpressionResolver.java:75)
> at org.apache.hadoop.hbase.mapreduce.CellCreator.(CellCreator.java:48)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.setup(TsvImporterMapper.java:115)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17243) Reuse CompactionPartitionId and avoid creating MobFileName in PartitionedMobCompactor to avoid unnecessary new objects

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716764#comment-15716764
 ] 

Hadoop QA commented on HBASE-17243:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 4s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
29m 24s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 100m 34s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
15s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 143m 33s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12841551/HBASE-17243-master-002.patch
 |
| JIRA Issue | HBASE-17243 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux b2f219c713c8 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 7775fed |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4767/artifact/patchprocess/patch-unit-hbase-server.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4767/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4767/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> Reuse CompactionPartitionId and avoid creating MobFileName in 
> PartitionedMobCompactor to avoid unnecessary new objects
> 

[jira] [Created] (HBASE-17244) Refactor StartcodeAgnosticServerName so it doesn't extend ServerName

[Thiruvel Thirumoolan (JIRA)]

Thiruvel Thirumoolan created HBASE-17244:


 Summary: Refactor StartcodeAgnosticServerName so it doesn't extend 
ServerName
 Key: HBASE-17244
 URL: https://issues.apache.org/jira/browse/HBASE-17244
 Project: HBase
  Issue Type: Sub-task
Reporter: Thiruvel Thirumoolan
Assignee: Thiruvel Thirumoolan
Priority: Minor


Follow up jira to address [~toffer]'s review comments @ 
https://reviews.apache.org/r/53242/. /cc [~devaraj].

{quote}
Apart from sharing a lot of functionality as ServerName it seems there is no 
need to be a subclass. In fact it is prolly not good design to do so as it may 
cause unwanted mixups for the user. (ie both instances in the same collection). 
Since this is functionally correct. We can address this in a separate jira. 
Prolly something we should do before we pull in the next jira.
{quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17243) Reuse CompactionPartitionId and avoid creating MobFileName in PartitionedMobCompactor to avoid unnecessary new objects

[Matteo Bertozzi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716653#comment-15716653
 ] 

Matteo Bertozzi commented on HBASE-17243:
-

+1
test failures seems unrelated

> Reuse CompactionPartitionId and avoid creating MobFileName in 
> PartitionedMobCompactor to avoid unnecessary new objects
> --
>
> Key: HBASE-17243
> URL: https://issues.apache.org/jira/browse/HBASE-17243
> Project: HBase
>  Issue Type: Improvement
>  Components: mob
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Attachments: HBASE-17243-master-001.patch, 
> HBASE-17243-master-002.patch
>
>
> In today's select() implementation, when it is an existing id, the new 
> allocated object is discarded. It should be reused. fileName is created to 
> getStartKey and getDate(), utility APIs can be created to directly get these 
> fields from the string.
> {code}
>   } else if (allFiles || linkedFile.getLen() < mergeableSize) {
> // add all files if allFiles is true,
> // otherwise add the small files to the merge pool
> MobFileName fileName = 
> MobFileName.create(linkedFile.getPath().getName());
> CompactionPartitionId id = new 
> CompactionPartitionId(fileName.getStartKey(),
>   fileName.getDate());
> CompactionPartition compactionPartition = filesToCompact.get(id);
> if (compactionPartition == null) {
>   compactionPartition = new CompactionPartition(id);
>   compactionPartition.addFile(file);
>   filesToCompact.put(id, compactionPartition);
> } else {
>   compactionPartition.addFile(file);
> }
> selectedFileCount++;
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17242) Isolate Legacy Scanner Logic

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716610#comment-15716610
 ] 

stack commented on HBASE-17242:
---

We want to minimize adding public methods on Scan. Its a user-facing Class. It 
is already carrying loads of surface. The added method also seems like facility 
that will not last. One day we'll purge the legacy matcher.

Suggest that for above reasons we add the method instead as private utility on 
StoreScanner (though it won't be as pretty).

> Isolate Legacy Scanner Logic
> 
>
> Key: HBASE-17242
> URL: https://issues.apache.org/jira/browse/HBASE-17242
> Project: HBase
>  Issue Type: Improvement
>Reporter: John Leach
>Assignee: John Leach
>Priority: Trivial
> Attachments: HBASE-17242.patch
>
>
> Duplicative logic in StoreScanner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17243) Reuse CompactionPartitionId and avoid creating MobFileName in PartitionedMobCompactor to avoid unnecessary new objects

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716591#comment-15716591
 ] 

Hadoop QA commented on HBASE-17243:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
28m 37s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 9s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 84m 45s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
19s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 125m 36s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.replication.TestSerialReplication |
| Timed out junit tests | 
org.apache.hadoop.hbase.namespace.TestNamespaceAuditor |
|   | org.apache.hadoop.hbase.TestRegionRebalancing |
|   | 
org.apache.hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes |
|   | org.apache.hadoop.hbase.quotas.TestQuotaThrottle |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12841542/HBASE-17243-master-001.patch
 |
| JIRA Issue | HBASE-17243 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux df0069310367 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 4a20361 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4766/artifact/patchprocess/patch-unit-hbase-server.txt
 |
| unit test logs |  
https://builds.apache.org/job/PreCommit-HBASE-Build/4766/artifact/patchprocess/patch-unit-hbase-server.txt
 |
|  Test Results | 

[jira] [Commented] (HBASE-16119) Procedure v2 - Reimplement merge

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716581#comment-15716581
 ] 

stack commented on HBASE-16119:
---

Needs a release note.

Adds a method to Admin. Is this an incompat change? Allowed but should we tick 
the incompat box?  Thanks.

> Procedure v2 - Reimplement merge
> 
>
> Key: HBASE-16119
> URL: https://issues.apache.org/jira/browse/HBASE-16119
> Project: HBase
>  Issue Type: Sub-task
>  Components: proc-v2, Region Assignment
>Affects Versions: 2.0.0
>Reporter: Matteo Bertozzi
>Assignee: Stephen Yuan Jiang
> Fix For: 2.0.0
>
> Attachments: HBASE-16119.v1-master.patch, HBASE-16119.v2-master.patch
>
>
> use the proc-v2 state machine for merge. also update the logic to have a 
> single meta-writer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17237) Override the correct compact method in HMobStore

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716248#comment-15716248
 ] 

Ted Yu commented on HBASE-17237:


lgtm

Does TestHRegion pass locally with patch ?

> Override the correct compact method in HMobStore
> 
>
> Key: HBASE-17237
> URL: https://issues.apache.org/jira/browse/HBASE-17237
> Project: HBase
>  Issue Type: Bug
>  Components: mob
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Attachments: HBASE-17237.patch
>
>
> The method Store#compact(CompactionContext compaction, ThroughputController 
> throughputController) is deprecated now, and HMobStore overrides it. 
> HMobStore needs to override compact(CompactionContext compaction, 
> ThroughputController throughputController, User user) instead.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17240) ImportTsv encounters ClassNotFoundException for MasterProtos$MasterService$BlockingInterface

[Ted Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-17240:
---
Attachment: 17240.v2.txt

> ImportTsv encounters ClassNotFoundException for 
> MasterProtos$MasterService$BlockingInterface
> 
>
> Key: HBASE-17240
> URL: https://issues.apache.org/jira/browse/HBASE-17240
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
> Fix For: 2.0.0
>
> Attachments: 17240.v1.txt, 17240.v2.txt
>
>
> [~romil.choksi] reported the following problem.
> With command:
> {code}
> hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.separator=, 
> -Dimporttsv.bulk.output=output -Dimporttsv.columns=HBASE_ROW_KEY,f:count 
> wordcount word_count.csv
> {code}
> The following error showed up:
> {code}
> 2016-11-29 06:39:48,861 INFO  [main] mapreduce.Job: Task Id : 
> attempt_1479850535804_0004_m_00_2, Status : FAILED
> Error: java.lang.ClassNotFoundException: 
> org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$BlockingInterface
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:264)
> at 
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:225)
> at 
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:122)
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver.init(DefaultVisibilityExpressionResolver.java:75)
> at org.apache.hadoop.hbase.mapreduce.CellCreator.(CellCreator.java:48)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.setup(TsvImporterMapper.java:115)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17240) ImportTsv encounters ClassNotFoundException for MasterProtos$MasterService$BlockingInterface

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716240#comment-15716240
 ] 

Ted Yu commented on HBASE-17240:


Addressed Jerry's comment with patch v2.

> ImportTsv encounters ClassNotFoundException for 
> MasterProtos$MasterService$BlockingInterface
> 
>
> Key: HBASE-17240
> URL: https://issues.apache.org/jira/browse/HBASE-17240
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
> Fix For: 2.0.0
>
> Attachments: 17240.v1.txt, 17240.v2.txt
>
>
> [~romil.choksi] reported the following problem.
> With command:
> {code}
> hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.separator=, 
> -Dimporttsv.bulk.output=output -Dimporttsv.columns=HBASE_ROW_KEY,f:count 
> wordcount word_count.csv
> {code}
> The following error showed up:
> {code}
> 2016-11-29 06:39:48,861 INFO  [main] mapreduce.Job: Task Id : 
> attempt_1479850535804_0004_m_00_2, Status : FAILED
> Error: java.lang.ClassNotFoundException: 
> org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$BlockingInterface
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:264)
> at 
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:225)
> at 
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:122)
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver.init(DefaultVisibilityExpressionResolver.java:75)
> at org.apache.hadoop.hbase.mapreduce.CellCreator.(CellCreator.java:48)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.setup(TsvImporterMapper.java:115)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17240) ImportTsv encounters ClassNotFoundException for MasterProtos$MasterService$BlockingInterface

[Ted Yu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716238#comment-15716238
 ] 

Ted Yu commented on HBASE-17240:


Haven't had a chance to pinpoint the JIRA.

> ImportTsv encounters ClassNotFoundException for 
> MasterProtos$MasterService$BlockingInterface
> 
>
> Key: HBASE-17240
> URL: https://issues.apache.org/jira/browse/HBASE-17240
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
> Fix For: 2.0.0
>
> Attachments: 17240.v1.txt, 17240.v2.txt
>
>
> [~romil.choksi] reported the following problem.
> With command:
> {code}
> hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.separator=, 
> -Dimporttsv.bulk.output=output -Dimporttsv.columns=HBASE_ROW_KEY,f:count 
> wordcount word_count.csv
> {code}
> The following error showed up:
> {code}
> 2016-11-29 06:39:48,861 INFO  [main] mapreduce.Job: Task Id : 
> attempt_1479850535804_0004_m_00_2, Status : FAILED
> Error: java.lang.ClassNotFoundException: 
> org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$BlockingInterface
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:264)
> at 
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:225)
> at 
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:122)
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver.init(DefaultVisibilityExpressionResolver.java:75)
> at org.apache.hadoop.hbase.mapreduce.CellCreator.(CellCreator.java:48)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.setup(TsvImporterMapper.java:115)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17243) Reuse CompactionPartitionId and avoid creating MobFileName in PartitionedMobCompactor to avoid unnecessary new objects

[huaxiang sun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

huaxiang sun updated HBASE-17243:
-
Attachment: HBASE-17243-master-002.patch

Thanks Matteo, upload a new patch addressing your comments.

> Reuse CompactionPartitionId and avoid creating MobFileName in 
> PartitionedMobCompactor to avoid unnecessary new objects
> --
>
> Key: HBASE-17243
> URL: https://issues.apache.org/jira/browse/HBASE-17243
> Project: HBase
>  Issue Type: Improvement
>  Components: mob
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Attachments: HBASE-17243-master-001.patch, 
> HBASE-17243-master-002.patch
>
>
> In today's select() implementation, when it is an existing id, the new 
> allocated object is discarded. It should be reused. fileName is created to 
> getStartKey and getDate(), utility APIs can be created to directly get these 
> fields from the string.
> {code}
>   } else if (allFiles || linkedFile.getLen() < mergeableSize) {
> // add all files if allFiles is true,
> // otherwise add the small files to the merge pool
> MobFileName fileName = 
> MobFileName.create(linkedFile.getPath().getName());
> CompactionPartitionId id = new 
> CompactionPartitionId(fileName.getStartKey(),
>   fileName.getDate());
> CompactionPartition compactionPartition = filesToCompact.get(id);
> if (compactionPartition == null) {
>   compactionPartition = new CompactionPartition(id);
>   compactionPartition.addFile(file);
>   filesToCompact.put(id, compactionPartition);
> } else {
>   compactionPartition.addFile(file);
> }
> selectedFileCount++;
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17228) precommit grep -c ERROR may grab non errors

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716117#comment-15716117
 ] 

Hudson commented on HBASE-17228:


FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #2060 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/2060/])
HBASE-17228 precommit grep -c ERROR may grab non errors I tested this by 
(stack: rev 8f8daafee0d3d6b80fd01d9e04b7417fc31b0ff7)
* (edit) dev-support/hbase-personality.sh


> precommit grep -c ERROR may grab non errors
> ---
>
> Key: HBASE-17228
> URL: https://issues.apache.org/jira/browse/HBASE-17228
> Project: HBase
>  Issue Type: Bug
>  Components: scripts
>Reporter: Matteo Bertozzi
>Priority: Minor
> Attachments: HBASE-17228.master.001.patch
>
>
> it looks like that we do a simple "grep -c ERROR" to count the errors that we 
> have from the build.
> https://github.com/apache/hbase/blob/master/dev-support/hbase-personality.sh#L305
> but in this way we ended up with a count=1 just because we have one enum 
> called ERROR_CODE in hbase. and the enum shows up as debug message
> {noformat}
> $ grep ERROR patch-hbaseprotoc-hbase-server.txt 
> [DEBUG] adding entry 
> org/apache/hadoop/hbase/util/HBaseFsck$ErrorReporter$ERROR_CODE.class
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17081) Flush the entire CompactingMemStore content to disk

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716097#comment-15716097
 ] 

stack commented on HBASE-17081:
---

[~anoop.hbase] and [~ram_krish] You ok w/ commit? I am but after doing my 
review saw your comments up on rb which seem more substantial than mine. Thanks.

> Flush the entire CompactingMemStore content to disk
> ---
>
> Key: HBASE-17081
> URL: https://issues.apache.org/jira/browse/HBASE-17081
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Anastasia Braginsky
>Assignee: Anastasia Braginsky
> Attachments: HBASE-17081-V01.patch, HBASE-17081-V02.patch, 
> HBASE-17081-V03.patch, HBASE-17081-V04.patch, HBASE-17081-V05.patch, 
> Pipelinememstore_fortrunk_3.patch
>
>
> Part of CompactingMemStore's memory is held by an active segment, and another 
> part is divided between immutable segments in the compacting pipeline. Upon 
> flush-to-disk request we want to flush all of it to disk, in contrast to 
> flushing only tail of the compacting pipeline.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17235) Improvement in creation of CIS for onheap buffer cases

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716090#comment-15716090
 ] 

stack commented on HBASE-17235:
---

+1

> Improvement in creation of CIS for onheap buffer cases
> --
>
> Key: HBASE-17235
> URL: https://issues.apache.org/jira/browse/HBASE-17235
> Project: HBase
>  Issue Type: Improvement
>  Components: rpc
>Affects Versions: 2.0.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: HBASE-17235.patch, HBASE-17235_1.patch
>
>
> {code}
>   if (buf.hasArray()) {
> cis = CodedInputStream.newInstance(buf.array(), offset, buf.limit());
>   } else {
> {code}
> Currently we do this for onheap buffers incase there is no reservoir or the 
> size is less than the minSizeforReservoir. I could see that even if reservoir 
> is there there are requests which goes with the above way of creating CIS. 
> This could be made efficient to avoid underlying copies by just doing this
> {code}
> cis = UnsafeByteOperations.unsafeWrap(buf.array(), offset, 
> buf.limit()).newCodedInput();
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716076#comment-15716076
 ] 

stack commented on HBASE-15314:
---

Oh, are you running w/ your patch?

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Amal Joshy
> Attachments: HBASE-15314-v2.patch, HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716075#comment-15716075
 ] 

stack commented on HBASE-15314:
---

I added you [~aartokhy] I started to review up on gh but probably best to do it 
here.  Use the little tool ./dev-support/submit-patch.py  Thanks for working on 
this.

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Amal Joshy
> Attachments: HBASE-15314-v2.patch, HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17194) Assign the new region to the idle server after splitting

[stack (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-17194:
--
  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Pushed to master. Thanks [~chia7712]

> Assign the new region to the idle server after splitting
> 
>
> Key: HBASE-17194
> URL: https://issues.apache.org/jira/browse/HBASE-17194
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17194.v0.patch, HBASE-17194.v1.patch, 
> HBASE-17194.v2.patch, HBASE-17194.v3.patch, HBASE-17194.v4.patch, 
> HBASE-17194.v4.patch, evaluation-v0.png, tests.xlsx
>
>
> The new regions are assigned to the random servers after splitting, but there 
> always are some idle servers which don’t be assigned any regions on the new 
> cluster. It is a bad start of load balance, hence we should give priority to 
> the idle servers for assignment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17243) Reuse CompactionPartitionId and avoid creating MobFileName in PartitionedMobCompactor to avoid unnecessary new objects

[Matteo Bertozzi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716028#comment-15716028
 ] 

Matteo Bertozzi commented on HBASE-17243:
-

do the _END_INDEX constants needs to be public? it seems something that only 
the MobFileName should know about.

the constructor new CompactionPartitionId("", "") with the two empty string 
looks a bit weird, maybe add an empty constructor since now the object is 
mutable.

> Reuse CompactionPartitionId and avoid creating MobFileName in 
> PartitionedMobCompactor to avoid unnecessary new objects
> --
>
> Key: HBASE-17243
> URL: https://issues.apache.org/jira/browse/HBASE-17243
> Project: HBase
>  Issue Type: Improvement
>  Components: mob
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Attachments: HBASE-17243-master-001.patch
>
>
> In today's select() implementation, when it is an existing id, the new 
> allocated object is discarded. It should be reused. fileName is created to 
> getStartKey and getDate(), utility APIs can be created to directly get these 
> fields from the string.
> {code}
>   } else if (allFiles || linkedFile.getLen() < mergeableSize) {
> // add all files if allFiles is true,
> // otherwise add the small files to the merge pool
> MobFileName fileName = 
> MobFileName.create(linkedFile.getPath().getName());
> CompactionPartitionId id = new 
> CompactionPartitionId(fileName.getStartKey(),
>   fileName.getDate());
> CompactionPartition compactionPartition = filesToCompact.get(id);
> if (compactionPartition == null) {
>   compactionPartition = new CompactionPartition(id);
>   compactionPartition.addFile(file);
>   filesToCompact.put(id, compactionPartition);
> } else {
>   compactionPartition.addFile(file);
> }
> selectedFileCount++;
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17231) Region#getCellCompartor sp?

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15716020#comment-15716020
 ] 

stack commented on HBASE-17231:
---

Issue has been there a long time:

commit 977f867439e960c668ee6311e47c904efc40f219
Author: ramkrishna 
Date:   Tue May 5 11:38:10 2015 +0530

HBASE-10800 - Use CellComparator instead of KVComparator (Ram)

Yeah, probably needs deprecation cycle at this stage.

Its CPs so safe to remove in 2.0.

A deprecation would be nice to have.

I think this issue too trivial to for a deprecation. Will take a while to do 
deprecation.

> Region#getCellCompartor sp?
> ---
>
> Key: HBASE-17231
> URL: https://issues.apache.org/jira/browse/HBASE-17231
> Project: HBase
>  Issue Type: Bug
>Reporter: John Leach
>Assignee: John Leach
>Priority: Trivial
> Attachments: HBASE-17231.patch
>
>
> Region#getCellCompartor -> Region#getCellComparator



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17243) Reuse CompactionPartitionId and avoid creating MobFileName in PartitionedMobCompactor to avoid unnecessary new objects

[huaxiang sun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

huaxiang sun updated HBASE-17243:
-
Status: Patch Available  (was: Open)

> Reuse CompactionPartitionId and avoid creating MobFileName in 
> PartitionedMobCompactor to avoid unnecessary new objects
> --
>
> Key: HBASE-17243
> URL: https://issues.apache.org/jira/browse/HBASE-17243
> Project: HBase
>  Issue Type: Improvement
>  Components: mob
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Attachments: HBASE-17243-master-001.patch
>
>
> In today's select() implementation, when it is an existing id, the new 
> allocated object is discarded. It should be reused. fileName is created to 
> getStartKey and getDate(), utility APIs can be created to directly get these 
> fields from the string.
> {code}
>   } else if (allFiles || linkedFile.getLen() < mergeableSize) {
> // add all files if allFiles is true,
> // otherwise add the small files to the merge pool
> MobFileName fileName = 
> MobFileName.create(linkedFile.getPath().getName());
> CompactionPartitionId id = new 
> CompactionPartitionId(fileName.getStartKey(),
>   fileName.getDate());
> CompactionPartition compactionPartition = filesToCompact.get(id);
> if (compactionPartition == null) {
>   compactionPartition = new CompactionPartition(id);
>   compactionPartition.addFile(file);
>   filesToCompact.put(id, compactionPartition);
> } else {
>   compactionPartition.addFile(file);
> }
> selectedFileCount++;
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17243) Reuse CompactionPartitionId and avoid creating MobFileName in PartitionedMobCompactor to avoid unnecessary new objects

[huaxiang sun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

huaxiang sun updated HBASE-17243:
-
Attachment: HBASE-17243-master-001.patch

> Reuse CompactionPartitionId and avoid creating MobFileName in 
> PartitionedMobCompactor to avoid unnecessary new objects
> --
>
> Key: HBASE-17243
> URL: https://issues.apache.org/jira/browse/HBASE-17243
> Project: HBase
>  Issue Type: Improvement
>  Components: mob
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
>Priority: Minor
> Attachments: HBASE-17243-master-001.patch
>
>
> In today's select() implementation, when it is an existing id, the new 
> allocated object is discarded. It should be reused. fileName is created to 
> getStartKey and getDate(), utility APIs can be created to directly get these 
> fields from the string.
> {code}
>   } else if (allFiles || linkedFile.getLen() < mergeableSize) {
> // add all files if allFiles is true,
> // otherwise add the small files to the merge pool
> MobFileName fileName = 
> MobFileName.create(linkedFile.getPath().getName());
> CompactionPartitionId id = new 
> CompactionPartitionId(fileName.getStartKey(),
>   fileName.getDate());
> CompactionPartition compactionPartition = filesToCompact.get(id);
> if (compactionPartition == null) {
>   compactionPartition = new CompactionPartition(id);
>   compactionPartition.addFile(file);
>   filesToCompact.put(id, compactionPartition);
> } else {
>   compactionPartition.addFile(file);
> }
> selectedFileCount++;
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17184) Code cleanup of LruBlockCache

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715999#comment-15715999
 ] 

stack commented on HBASE-17184:
---

np [~lars_francke] I fixed it. I did git am because it looked like the patch 
was complete. I should have noticed the missing JIRA number. Fixed now. Thanks 
for contrib boss.

> Code cleanup of LruBlockCache
> -
>
> Key: HBASE-17184
> URL: https://issues.apache.org/jira/browse/HBASE-17184
> Project: HBase
>  Issue Type: Improvement
>Reporter: Lars Francke
>Assignee: Lars Francke
>Priority: Trivial
> Fix For: 2.0.0
>
> Attachments: HBASE-17184.master.001.patch
>
>
> Initially I only wanted to fix a badly worded Log message but found a bunch 
> of code convention violations etc. so I decided to clean up the class.
> It is only whitespace changes with the exception of the one log message ("too 
> many").



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17240) ImportTsv encounters ClassNotFoundException for MasterProtos$MasterService$BlockingInterface

[Jerry He (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715937#comment-15715937
 ] 

Jerry He commented on HBASE-17240:
--

Update the list in TableMapReduceUtil.addHBaseDependencyJars() ?

> ImportTsv encounters ClassNotFoundException for 
> MasterProtos$MasterService$BlockingInterface
> 
>
> Key: HBASE-17240
> URL: https://issues.apache.org/jira/browse/HBASE-17240
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
> Fix For: 2.0.0
>
> Attachments: 17240.v1.txt
>
>
> [~romil.choksi] reported the following problem.
> With command:
> {code}
> hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.separator=, 
> -Dimporttsv.bulk.output=output -Dimporttsv.columns=HBASE_ROW_KEY,f:count 
> wordcount word_count.csv
> {code}
> The following error showed up:
> {code}
> 2016-11-29 06:39:48,861 INFO  [main] mapreduce.Job: Task Id : 
> attempt_1479850535804_0004_m_00_2, Status : FAILED
> Error: java.lang.ClassNotFoundException: 
> org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$BlockingInterface
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:264)
> at 
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:225)
> at 
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:122)
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver.init(DefaultVisibilityExpressionResolver.java:75)
> at org.apache.hadoop.hbase.mapreduce.CellCreator.(CellCreator.java:48)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.setup(TsvImporterMapper.java:115)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-17243) Reuse CompactionPartitionId and avoid creating MobFileName in PartitionedMobCompactor to avoid unnecessary new objects

[huaxiang sun (JIRA)]

huaxiang sun created HBASE-17243:


 Summary: Reuse CompactionPartitionId and avoid creating 
MobFileName in PartitionedMobCompactor to avoid unnecessary new objects
 Key: HBASE-17243
 URL: https://issues.apache.org/jira/browse/HBASE-17243
 Project: HBase
  Issue Type: Improvement
  Components: mob
Affects Versions: 2.0.0
Reporter: huaxiang sun
Assignee: huaxiang sun
Priority: Minor


In today's select() implementation, when it is an existing id, the new 
allocated object is discarded. It should be reused. fileName is created to 
getStartKey and getDate(), utility APIs can be created to directly get these 
fields from the string.

{code}
  } else if (allFiles || linkedFile.getLen() < mergeableSize) {
// add all files if allFiles is true,
// otherwise add the small files to the merge pool
MobFileName fileName = 
MobFileName.create(linkedFile.getPath().getName());
CompactionPartitionId id = new 
CompactionPartitionId(fileName.getStartKey(),
  fileName.getDate());
CompactionPartition compactionPartition = filesToCompact.get(id);
if (compactionPartition == null) {
  compactionPartition = new CompactionPartition(id);
  compactionPartition.addFile(file);
  filesToCompact.put(id, compactionPartition);
} else {
  compactionPartition.addFile(file);
}
selectedFileCount++;
  }
}
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16894) Create more than 1 split per region, generalize HBASE-12590

[Jerry He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jerry He updated HBASE-16894:
-
Labels:   (was: beginner beginners)

> Create more than 1 split per region, generalize HBASE-12590
> ---
>
> Key: HBASE-16894
> URL: https://issues.apache.org/jira/browse/HBASE-16894
> Project: HBase
>  Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Yi Liang
>
> A common request from users is to be able to better control how many map 
> tasks are created per region. Right now, it is always 1 region = 1 input 
> split = 1 map task. Same goes for Spark since it uses the TIF. With region 
> sizes as large as 50 GBs, it is desirable to be able to create more than 1 
> split per region.
> HBASE-12590 adds a config property for MR jobs to be able to handle skew in 
> region sizes. The algorithm is roughly: 
> {code}
> If (region size >= average size*ratio) : cut the region into two MR input 
> splits
> If (average size <= region size < average size*ratio) : one region as one MR 
> input split
> If (sum of several continuous regions size < average size * ratio): combine 
> these regions into one MR input split.
> {code}
> Although we can set data skew ratio to be 0.5 or something to abuse 
> HBASE-12590 into creating more than 1 split task per region, it is not ideal. 
> But there is no way to create more with the patch as it is. For example we 
> cannot create more than 2 tasks per region. 
> If we want to fix this properly, we should extend the approach in 
> HBASE-12590, and make it so that the client can specify the desired num of 
> mappers, or desired split size, and the TIF generates the splits based on the 
> current region sizes very similar to the algorithm in HBASE-12590, but a more 
> generic way. This also would eliminate the hand tuning of data skew ratio.
> We also can think about the guidepost approach that Phoenix has in the stats 
> table which is used for exactly this purpose. Right now, the region can be 
> split into powers of two assuming uniform distribution within the region. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17241) Avoid compacting already compacted mob files with _del files

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715810#comment-15715810
 ] 

Hadoop QA commented on HBASE-17241:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
25m 34s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 110m 16s 
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
15s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 147m 49s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.mob.compactions.TestPartitionedMobCompactor 
|
|   | hadoop.hbase.regionserver.TestHRegionWithInMemoryFlush |
|   | hadoop.hbase.replication.TestSerialReplication |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.2 Server=1.12.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12841513/HBASE-17241.master.001.patch
 |
| JIRA Issue | HBASE-17241 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux eafc2e1a9d04 3.13.0-100-generic #147-Ubuntu SMP Tue Oct 18 
16:48:51 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 9c13219 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4765/artifact/patchprocess/patch-unit-hbase-server.txt
 |
| unit test logs |  
https://builds.apache.org/job/PreCommit-HBASE-Build/4765/artifact/patchprocess/patch-unit-hbase-server.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4765/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output | 

[jira] [Commented] (HBASE-16894) Create more than 1 split per region, generalize HBASE-12590

[Yi Liang (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715737#comment-15715737
 ] 

Yi Liang commented on HBASE-16894:
--

After discuss with Enis, we have some ideas for this jira,
 (1) Given the requirements in HBASE-16894, generalize the logic of doing more 
than 1 mapper / reducer per region. This will use the RegionSplitter to 
calculate the split points within the regions if needed. This will assume 
uniform distribution, and will calculate split points from just the start / end 
keys, not the actual data. 
 (2) Find a way to calculate split points from existing data. 

Notice that doing (1) does not require (2) but (1) will give us a lot of 
benefits even without (2). So, I would suggest we should concentrate on doing 
the patch for (1) in HBASE-16894. Again, we won't use any statistics or ask the 
regionservers, etc for this. We will use just the region start / end key 
boundaries and uniform distribution. 

After doing (1), we can create a follow up jira to do (2). There is already 
some work that is recent in HBASE-16169. This makes it so that the client asks 
every regionserver for the region size calculation. I think we can piggy-back 
on this and have the region server return some split points based on the actual 
indexes as well as a part of the RegionLoad object. The Region itself can 
maintain a small list of guideposts and make that available in the RegionLoad 
object when asked. This way we won't have to maintain this info externally in a 
table or something, but it will still be available on demand.  

Again, I think we should do the patch for (1) first, then only after that focus 
on solving (2). 

> Create more than 1 split per region, generalize HBASE-12590
> ---
>
> Key: HBASE-16894
> URL: https://issues.apache.org/jira/browse/HBASE-16894
> Project: HBase
>  Issue Type: Improvement
>Reporter: Enis Soztutar
>Assignee: Yi Liang
>  Labels: beginner, beginners
>
> A common request from users is to be able to better control how many map 
> tasks are created per region. Right now, it is always 1 region = 1 input 
> split = 1 map task. Same goes for Spark since it uses the TIF. With region 
> sizes as large as 50 GBs, it is desirable to be able to create more than 1 
> split per region.
> HBASE-12590 adds a config property for MR jobs to be able to handle skew in 
> region sizes. The algorithm is roughly: 
> {code}
> If (region size >= average size*ratio) : cut the region into two MR input 
> splits
> If (average size <= region size < average size*ratio) : one region as one MR 
> input split
> If (sum of several continuous regions size < average size * ratio): combine 
> these regions into one MR input split.
> {code}
> Although we can set data skew ratio to be 0.5 or something to abuse 
> HBASE-12590 into creating more than 1 split task per region, it is not ideal. 
> But there is no way to create more with the patch as it is. For example we 
> cannot create more than 2 tasks per region. 
> If we want to fix this properly, we should extend the approach in 
> HBASE-12590, and make it so that the client can specify the desired num of 
> mappers, or desired split size, and the TIF generates the splits based on the 
> current region sizes very similar to the algorithm in HBASE-12590, but a more 
> generic way. This also would eliminate the hand tuning of data skew ratio.
> We also can think about the guidepost approach that Phoenix has in the stats 
> table which is used for exactly this purpose. Right now, the region can be 
> split into powers of two assuming uniform distribution within the region. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17242) Isolate Legacy Scanner Logic

[John Leach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Leach updated HBASE-17242:
---
Priority: Trivial  (was: Major)

> Isolate Legacy Scanner Logic
> 
>
> Key: HBASE-17242
> URL: https://issues.apache.org/jira/browse/HBASE-17242
> Project: HBase
>  Issue Type: Improvement
>Reporter: John Leach
>Assignee: John Leach
>Priority: Trivial
> Attachments: HBASE-17242.patch
>
>
> Duplicative logic in StoreScanner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17242) Isolate Legacy Scanner Logic

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715691#comment-15715691
 ] 

Hadoop QA commented on HBASE-17242:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
32s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 47s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
23m 9s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 58s 
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 85m 45s 
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
28s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 124m 24s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12841494/HBASE-17242.patch |
| JIRA Issue | HBASE-17242 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 8f97f1af80a6 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / 9c13219 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4764/testReport/ |
| modules | C: hbase-client hbase-server U: . |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4764/console |
| Powered by | Apache Yetus 0.3.0   

[jira] [Commented] (HBASE-15314) Allow more than one backing file in bucketcache

[Zach York (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715676#comment-15715676
 ] 

Zach York commented on HBASE-15314:
---

+1 I have reviewed the patch and think it is much better than the original v2.

[~stack] can you review this? If it needs to be a patch (instead of a PR), can 
you change the assignee so that Aaron can attach the patch?

> Allow more than one backing file in bucketcache
> ---
>
> Key: HBASE-15314
> URL: https://issues.apache.org/jira/browse/HBASE-15314
> Project: HBase
>  Issue Type: Sub-task
>  Components: BucketCache
>Reporter: stack
>Assignee: Amal Joshy
> Attachments: HBASE-15314-v2.patch, HBASE-15314.patch
>
>
> Allow bucketcache use more than just one backing file: e.g. chassis has more 
> than one SSD in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16981) Expand Mob Compaction Partition policy from daily to weekly, monthly and beyond

[huaxiang sun (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715626#comment-15715626
 ] 

huaxiang sun commented on HBASE-16981:
--

Link to the doc

https://docs.google.com/document/d/1y-jCl1TgMecK7j62MHRtholzyoQaXB3bq0jIKQszE64/edit

> Expand Mob Compaction Partition policy from daily to weekly, monthly and 
> beyond
> ---
>
> Key: HBASE-16981
> URL: https://issues.apache.org/jira/browse/HBASE-16981
> Project: HBase
>  Issue Type: New Feature
>  Components: mob
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
> Attachments: HBASE-16981.master.001.patch, 
> HBASE-16981.master.002.patch, 
> Supportingweeklyandmonthlymobcompactionpartitionpolicyinhbase.pdf
>
>
> Today the mob region holds all mob files for all regions. With daily 
> partition mob compaction policy, after major mob compaction, there is still 
> one file per region daily. Given there is 365 days in one year, at least 365 
> files per region. Since HDFS has limitation for number of files under one 
> folder, this is not going to scale if there are lots of regions. To reduce 
> mob file number,  we want to introduce other partition policies such as 
> weekly, monthly to compact mob files within one week or month into one file. 
> This jira is create to track this effort.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16981) Expand Mob Compaction Partition policy from daily to weekly, monthly and beyond

[huaxiang sun (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715621#comment-15715621
 ] 

huaxiang sun commented on HBASE-16981:
--

Hi [~jingcheng.du] and [~anoop.hbase], I upload the initial design doc based on 
your feedback, please comment, thanks.

> Expand Mob Compaction Partition policy from daily to weekly, monthly and 
> beyond
> ---
>
> Key: HBASE-16981
> URL: https://issues.apache.org/jira/browse/HBASE-16981
> Project: HBase
>  Issue Type: New Feature
>  Components: mob
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
> Attachments: HBASE-16981.master.001.patch, 
> HBASE-16981.master.002.patch, 
> Supportingweeklyandmonthlymobcompactionpartitionpolicyinhbase.pdf
>
>
> Today the mob region holds all mob files for all regions. With daily 
> partition mob compaction policy, after major mob compaction, there is still 
> one file per region daily. Given there is 365 days in one year, at least 365 
> files per region. Since HDFS has limitation for number of files under one 
> folder, this is not going to scale if there are lots of regions. To reduce 
> mob file number,  we want to introduce other partition policies such as 
> weekly, monthly to compact mob files within one week or month into one file. 
> This jira is create to track this effort.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17228) precommit grep -c ERROR may grab non errors

[stack (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715618#comment-15715618
 ] 

stack commented on HBASE-17228:
---

I pushed it. Lets try it. Leaving open in case probs. Will resolve if all good.

> precommit grep -c ERROR may grab non errors
> ---
>
> Key: HBASE-17228
> URL: https://issues.apache.org/jira/browse/HBASE-17228
> Project: HBase
>  Issue Type: Bug
>  Components: scripts
>Reporter: Matteo Bertozzi
>Priority: Minor
> Attachments: HBASE-17228.master.001.patch
>
>
> it looks like that we do a simple "grep -c ERROR" to count the errors that we 
> have from the build.
> https://github.com/apache/hbase/blob/master/dev-support/hbase-personality.sh#L305
> but in this way we ended up with a count=1 just because we have one enum 
> called ERROR_CODE in hbase. and the enum shows up as debug message
> {noformat}
> $ grep ERROR patch-hbaseprotoc-hbase-server.txt 
> [DEBUG] adding entry 
> org/apache/hadoop/hbase/util/HBaseFsck$ErrorReporter$ERROR_CODE.class
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-16981) Expand Mob Compaction Partition policy from daily to weekly, monthly and beyond

[huaxiang sun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

huaxiang sun updated HBASE-16981:
-
Status: Open  (was: Patch Available)

Will submit a new patch.

> Expand Mob Compaction Partition policy from daily to weekly, monthly and 
> beyond
> ---
>
> Key: HBASE-16981
> URL: https://issues.apache.org/jira/browse/HBASE-16981
> Project: HBase
>  Issue Type: New Feature
>  Components: mob
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
> Attachments: HBASE-16981.master.001.patch, 
> HBASE-16981.master.002.patch, 
> Supportingweeklyandmonthlymobcompactionpartitionpolicyinhbase.pdf
>
>
> Today the mob region holds all mob files for all regions. With daily 
> partition mob compaction policy, after major mob compaction, there is still 
> one file per region daily. Given there is 365 days in one year, at least 365 
> files per region. Since HDFS has limitation for number of files under one 
> folder, this is not going to scale if there are lots of regions. To reduce 
> mob file number,  we want to introduce other partition policies such as 
> weekly, monthly to compact mob files within one week or month into one file. 
> This jira is create to track this effort.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17112) Prevent setting timestamp of delta operations the same as previous value's

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715545#comment-15715545
 ] 

Hudson commented on HBASE-17112:


SUCCESS: Integrated in Jenkins build HBase-1.1-JDK8 #1908 (See 
[https://builds.apache.org/job/HBase-1.1-JDK8/1908/])
HBASE-17112 Prevent setting timestamp of delta operations being same as 
(yangzhe1991: rev add0259f0f216aa5f46df1041e8c22f9b2127e62)
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java


> Prevent setting timestamp of delta operations the same as previous value's
> --
>
> Key: HBASE-17112
> URL: https://issues.apache.org/jira/browse/HBASE-17112
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.1.7, 0.98.23, 1.2.4
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 2.0.0, 1.4.0, 1.2.5, 1.1.8
>
> Attachments: HBASE-17112-branch-1-v1.patch, 
> HBASE-17112-branch-1-v1.patch, HBASE-17112-branch-1.1-v1.patch, 
> HBASE-17112-branch-1.1-v1.patch, HBASE-17112-v1.patch, HBASE-17112-v2.patch, 
> HBASE-17112-v2.patch
>
>
> In delta operations, Increment and Append. We will read current value first 
> and then write the new whole result into WAL as the type of Put with current 
> timestamp. If the previous ts is larger than current ts, we will use the 
> previous ts.
> If we have two Puts with same TS, we will ignore the Put with lower sequence 
> id. It is not friendly with versioning. And for replication we will drop 
> sequence id  while writing to peer cluster so in the slave we don't know what 
> the order they are being written. If the pushing is disordered, the result 
> will be wrong.
> We can set the new ts to previous+1 if the previous is not less than now.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16886) hbase-client: scanner with reversed=true and small=true gets no result

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715544#comment-15715544
 ] 

Hudson commented on HBASE-16886:


SUCCESS: Integrated in Jenkins build HBase-1.1-JDK8 #1908 (See 
[https://builds.apache.org/job/HBase-1.1-JDK8/1908/])
HBASE-16886: hbase-client: scanner with reversed=true and small=true 
(yangzhe1991: rev baf940969a8a608c3015e64b32e3a217b826ff48)
* (edit) 
hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestClientSmallReversedScanner.java
* (add) 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSmallReversedScanner.java
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientSmallReversedScanner.java


> hbase-client: scanner with reversed=true and small=true gets no result
> --
>
> Key: HBASE-16886
> URL: https://issues.apache.org/jira/browse/HBASE-16886
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.2.3, 1.1.7, 0.98.23
>Reporter: huzheng
>Assignee: huzheng
>  Labels: patch
> Fix For: 2.0.0, 1.4.0, 1.2.5, 1.1.8
>
> Attachments: 16886.addendum, 16886.v4.branch-1.patch, 
> HBASE-16886.v0.patch, HBASE-16886.v1.patch, HBASE-16886.v2.patch, 
> HBASE-16886.v3.patch, HBASE-16886.v4.0.98.patch, 
> HBASE-16886.v4.branch-1.patch, HBASE-16886.v4.branch-1.patch, 
> HBASE-16886.v4.branch-1.patch, HBASE-16886.v4.master.patch, 
> TestReversedSmallScan.java
>
>
> Assume HBase have four regions (-oo, b), [b, c), [c, d), [d,+oo) , and all 
> rowKeys are located in region [d, +oo).  using a Reversed Small Scanner will 
> get no result.
> Attached file show this failed test case.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17161) MOB : Make ref cell creation more efficient

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715536#comment-15715536
 ] 

Hudson commented on HBASE-17161:


SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2059 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/2059/])
HBASE-17161 MOB : Make ref cell creation more efficient. (anoopsamjohn: rev 
4b3a097e5a62063e6bc92bec63bcd91bbef6)
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/util/HFileTestUtil.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobUtils.java
* (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/CellUtil.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreFlusher.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HMobStore.java
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHMobStore.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/mob/compactions/PartitionedMobCompactor.java
* (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/TagUtil.java
* (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValueUtil.java
* (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/ExtendedCell.java


> MOB : Make ref cell creation more efficient
> ---
>
> Key: HBASE-17161
> URL: https://issues.apache.org/jira/browse/HBASE-17161
> Project: HBase
>  Issue Type: Improvement
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-17161.patch, HBASE-17161.patch, 
> HBASE-17161_V2.patch, HBASE-17161_V2.patch
>
>
> When we flush MOB data, ref cells are created per actual MOB cell. This 
> creates lots of garbage
> Refer MobUtils#createMobRefCell
> We need to add 2 tags into the ref cell. An ArrayList is created with default 
> size creating ref array. Call to CellUtil.getTags will create a new ArrayList 
> even if the original cell is having no tags.
> A new KV is created which will create a new backing byte[] and do copy.  Also 
> along with each of the flush/compaction op, a fresh Tag object is created for 
> TableName tag.
> Fixes include
> 1. A table name tag is not going to change per HStore.  Even both the new 
> tags to be added.  Will keep a byte[] of these 2 tags at MobHStore level so 
> that all flush and compactions in this store can use it.
> 2. Create a new MobRefCell just like TagRewriteCell where only value and tags 
> part will be diff from the original cell.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17232) Replace HashSet with ArrayList to accumulate delayed scanners in KVHeap and StoreScanner.

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715535#comment-15715535
 ] 

Hudson commented on HBASE-17232:


SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #2059 (See 
[https://builds.apache.org/job/HBase-Trunk_matrix/2059/])
HBASE-17232 Replace HashSet with ArrayList to accumulate delayed (binlijin: rev 
9c13219f0d142051bb31f3523f2f5b8e7f231d90)
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java


> Replace HashSet with ArrayList to accumulate delayed scanners in KVHeap and 
> StoreScanner.
> -
>
> Key: HBASE-17232
> URL: https://issues.apache.org/jira/browse/HBASE-17232
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: binlijin
>Assignee: binlijin
> Fix For: 2.0.0
>
> Attachments: HBASE-17232.patch
>
>
> HashSet is slow than ArrayList, also generate more garbage.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17112) Prevent setting timestamp of delta operations the same as previous value's

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715515#comment-15715515
 ] 

Hudson commented on HBASE-17112:


FAILURE: Integrated in Jenkins build HBase-1.1-JDK7 #1824 (See 
[https://builds.apache.org/job/HBase-1.1-JDK7/1824/])
HBASE-17112 Prevent setting timestamp of delta operations being same as 
(yangzhe1991: rev add0259f0f216aa5f46df1041e8c22f9b2127e62)
* (edit) 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
* (edit) 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java


> Prevent setting timestamp of delta operations the same as previous value's
> --
>
> Key: HBASE-17112
> URL: https://issues.apache.org/jira/browse/HBASE-17112
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 1.1.7, 0.98.23, 1.2.4
>Reporter: Phil Yang
>Assignee: Phil Yang
> Fix For: 2.0.0, 1.4.0, 1.2.5, 1.1.8
>
> Attachments: HBASE-17112-branch-1-v1.patch, 
> HBASE-17112-branch-1-v1.patch, HBASE-17112-branch-1.1-v1.patch, 
> HBASE-17112-branch-1.1-v1.patch, HBASE-17112-v1.patch, HBASE-17112-v2.patch, 
> HBASE-17112-v2.patch
>
>
> In delta operations, Increment and Append. We will read current value first 
> and then write the new whole result into WAL as the type of Put with current 
> timestamp. If the previous ts is larger than current ts, we will use the 
> previous ts.
> If we have two Puts with same TS, we will ignore the Put with lower sequence 
> id. It is not friendly with versioning. And for replication we will drop 
> sequence id  while writing to peer cluster so in the slave we don't know what 
> the order they are being written. If the pushing is disordered, the result 
> will be wrong.
> We can set the new ts to previous+1 if the previous is not less than now.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16886) hbase-client: scanner with reversed=true and small=true gets no result

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715514#comment-15715514
 ] 

Hudson commented on HBASE-16886:


FAILURE: Integrated in Jenkins build HBase-1.1-JDK7 #1824 (See 
[https://builds.apache.org/job/HBase-1.1-JDK7/1824/])
HBASE-16886: hbase-client: scanner with reversed=true and small=true 
(yangzhe1991: rev baf940969a8a608c3015e64b32e3a217b826ff48)
* (edit) 
hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestClientSmallReversedScanner.java
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientSmallReversedScanner.java
* (add) 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSmallReversedScanner.java


> hbase-client: scanner with reversed=true and small=true gets no result
> --
>
> Key: HBASE-16886
> URL: https://issues.apache.org/jira/browse/HBASE-16886
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.2.3, 1.1.7, 0.98.23
>Reporter: huzheng
>Assignee: huzheng
>  Labels: patch
> Fix For: 2.0.0, 1.4.0, 1.2.5, 1.1.8
>
> Attachments: 16886.addendum, 16886.v4.branch-1.patch, 
> HBASE-16886.v0.patch, HBASE-16886.v1.patch, HBASE-16886.v2.patch, 
> HBASE-16886.v3.patch, HBASE-16886.v4.0.98.patch, 
> HBASE-16886.v4.branch-1.patch, HBASE-16886.v4.branch-1.patch, 
> HBASE-16886.v4.branch-1.patch, HBASE-16886.v4.master.patch, 
> TestReversedSmallScan.java
>
>
> Assume HBase have four regions (-oo, b), [b, c), [c, d), [d,+oo) , and all 
> rowKeys are located in region [d, +oo).  using a Reversed Small Scanner will 
> get no result.
> Attached file show this failed test case.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying

[Anoop Sam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715402#comment-15715402
 ] 

Anoop Sam John commented on HBASE-14882:


1.  Pls see ValueAndTagRewriteCell and how this write is handled
2. All these individual component byte[]s are independent byte[]s.  And we 
consider  java array object overhead in heapSize(). This is actually overhead 
only..So we better consider it in overhead. In case of KV#heapOverhead() we 
don't consider Array size because mostly many KVs share same byte[].  Atleast 
in server side when we have it.  So this was removed to avoid we overestimate 
heap size and overhead.  This was fixed recently.  But for this new Cell impl, 
we must consider the array overheads as part of heapOverhead()
3. Hmm..  align() is ideally required..   Our bad.. missed some places// May be 
another issue u can raise to fix all possible places?  Am sure many a places it 
might be missing!


> Provide a Put API that adds the provided family, qualifier, value without 
> copying
> -
>
> Key: HBASE-14882
> URL: https://issues.apache.org/jira/browse/HBASE-14882
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 1.2.0
>Reporter: Jerry He
>Assignee: Xiang Li
> Fix For: 2.0.0
>
> Attachments: HBASE-14882.master.000.patch, 
> HBASE-14882.master.001.patch, HBASE-14882.master.002.patch, 
> HBASE-14882.master.003.patch, HBASE-14882.master.004.patch
>
>
> In the Put API, we have addImmutable()
> {code}
>  /**
>* See {@link #addColumn(byte[], byte[], byte[])}. This version expects
>* that the underlying arrays won't change. It's intended
>* for usage internal HBase to and for advanced client applications.
>*/
>   public Put addImmutable(byte [] family, byte [] qualifier, byte [] value)
> {code}
> But in the implementation, the family, qualifier and value are still being 
> copied locally to create kv.
> Hopefully we should provide an API that truly uses immutable family, 
> qualifier and value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17241) Avoid compacting already compacted mob files with _del files

[huaxiang sun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

huaxiang sun updated HBASE-17241:
-
Attachment: HBASE-17241.master.001.patch

> Avoid compacting already compacted  mob files with _del files
> -
>
> Key: HBASE-17241
> URL: https://issues.apache.org/jira/browse/HBASE-17241
> Project: HBase
>  Issue Type: Improvement
>  Components: mob
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
> Attachments: HBASE-17241.master.001.patch
>
>
> Today if there is only one file in the partition, and there is no _del files, 
> the file is skipped. With del file, the current logic is to compact the 
> already-compacted file with _del file. Let's say there is one mob file 
> regionA20161101***, which was compacted. On 12/1/2016, there is _del file 
> regionB20161201**_del, mob compaction kicks in, regionA20161101*** is less 
> than the threshold, and it is picked for compaction. Since there is a _del 
> file, regionA20161101 and regionB20161201***_del are compacted into 
> regionA20161101**_1 . After that, regionB20161201**_del cannot be deleted 
> since it is not a allFile compaction. The next mob compaction, 
> regionA20161101**_1 and regionB20161201**_del will be picked up again and be 
> compacted into regionA20161101***_2. So in this case, it will cause more 
> unnecessary IOs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HBASE-17228) precommit grep -c ERROR may grab non errors

[Matteo Bertozzi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715392#comment-15715392
 ] 

Matteo Bertozzi edited comment on HBASE-17228 at 12/2/16 3:21 PM:
--

yeah, I think the patch it's ok. +1 
I did a quick check and it looks like the errors are around \[ERROR\]. 
the only thing it seems that we end up with too many errors, but that it was 
also true before. for example a simple typo in one protobuf line gives me 36 
error, all marked as \[ERROR\].
{noformat}
[ERROR] PROTOC FAILED: [libprotobuf WARNING 
google/protobuf/compiler/parser.cc:546] No syntax specified for the proto file: 
Admin.proto. Please use 'syntax = "proto2";' or 'syntax = "proto3";' to specify 
a syntax version. (Defaulted to proto2 syntax.)
...
[ERROR] Failed to execute goal 
org.xolstice.maven.plugins:protobuf-maven-plugin:0.5.0:compile (compile-protoc) 
on project hbase-protocol-shaded: protoc did not exit cleanly. Review output 
for more information. -> [Help 1]
{noformat}


was (Author: mbertozzi):
yeah, I think the patch it's ok. I did a quick check and it looks like the 
errors are around \[ERROR\]. 
the only thing it seems that we end up with too many errors, but that it was 
also true before. for example a simple typo in one protobuf line gives me 36 
error, all marked as \[ERROR\].
{noformat}
[ERROR] PROTOC FAILED: [libprotobuf WARNING 
google/protobuf/compiler/parser.cc:546] No syntax specified for the proto file: 
Admin.proto. Please use 'syntax = "proto2";' or 'syntax = "proto3";' to specify 
a syntax version. (Defaulted to proto2 syntax.)
...
[ERROR] Failed to execute goal 
org.xolstice.maven.plugins:protobuf-maven-plugin:0.5.0:compile (compile-protoc) 
on project hbase-protocol-shaded: protoc did not exit cleanly. Review output 
for more information. -> [Help 1]
{noformat}

> precommit grep -c ERROR may grab non errors
> ---
>
> Key: HBASE-17228
> URL: https://issues.apache.org/jira/browse/HBASE-17228
> Project: HBase
>  Issue Type: Bug
>  Components: scripts
>Reporter: Matteo Bertozzi
>Priority: Minor
> Attachments: HBASE-17228.master.001.patch
>
>
> it looks like that we do a simple "grep -c ERROR" to count the errors that we 
> have from the build.
> https://github.com/apache/hbase/blob/master/dev-support/hbase-personality.sh#L305
> but in this way we ended up with a count=1 just because we have one enum 
> called ERROR_CODE in hbase. and the enum shows up as debug message
> {noformat}
> $ grep ERROR patch-hbaseprotoc-hbase-server.txt 
> [DEBUG] adding entry 
> org/apache/hadoop/hbase/util/HBaseFsck$ErrorReporter$ERROR_CODE.class
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17228) precommit grep -c ERROR may grab non errors

[Matteo Bertozzi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715392#comment-15715392
 ] 

Matteo Bertozzi commented on HBASE-17228:
-

yeah, I think the patch it's ok. I did a quick check and it looks like the 
errors are around \[ERROR\]. 
the only thing it seems that we end up with too many errors, but that it was 
also true before. for example a simple typo in one protobuf line gives me 36 
error, all marked as \[ERROR\].
{noformat}
[ERROR] PROTOC FAILED: [libprotobuf WARNING 
google/protobuf/compiler/parser.cc:546] No syntax specified for the proto file: 
Admin.proto. Please use 'syntax = "proto2";' or 'syntax = "proto3";' to specify 
a syntax version. (Defaulted to proto2 syntax.)
...
[ERROR] Failed to execute goal 
org.xolstice.maven.plugins:protobuf-maven-plugin:0.5.0:compile (compile-protoc) 
on project hbase-protocol-shaded: protoc did not exit cleanly. Review output 
for more information. -> [Help 1]
{noformat}

> precommit grep -c ERROR may grab non errors
> ---
>
> Key: HBASE-17228
> URL: https://issues.apache.org/jira/browse/HBASE-17228
> Project: HBase
>  Issue Type: Bug
>  Components: scripts
>Reporter: Matteo Bertozzi
>Priority: Minor
> Attachments: HBASE-17228.master.001.patch
>
>
> it looks like that we do a simple "grep -c ERROR" to count the errors that we 
> have from the build.
> https://github.com/apache/hbase/blob/master/dev-support/hbase-personality.sh#L305
> but in this way we ended up with a count=1 just because we have one enum 
> called ERROR_CODE in hbase. and the enum shows up as debug message
> {noformat}
> $ grep ERROR patch-hbaseprotoc-hbase-server.txt 
> [DEBUG] adding entry 
> org/apache/hadoop/hbase/util/HBaseFsck$ErrorReporter$ERROR_CODE.class
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17241) Avoid compacting already compacted mob files with _del files

[huaxiang sun (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

huaxiang sun updated HBASE-17241:
-
Status: Patch Available  (was: Open)

> Avoid compacting already compacted  mob files with _del files
> -
>
> Key: HBASE-17241
> URL: https://issues.apache.org/jira/browse/HBASE-17241
> Project: HBase
>  Issue Type: Improvement
>  Components: mob
>Affects Versions: 2.0.0
>Reporter: huaxiang sun
>Assignee: huaxiang sun
> Attachments: HBASE-17241.master.001.patch
>
>
> Today if there is only one file in the partition, and there is no _del files, 
> the file is skipped. With del file, the current logic is to compact the 
> already-compacted file with _del file. Let's say there is one mob file 
> regionA20161101***, which was compacted. On 12/1/2016, there is _del file 
> regionB20161201**_del, mob compaction kicks in, regionA20161101*** is less 
> than the threshold, and it is picked for compaction. Since there is a _del 
> file, regionA20161101 and regionB20161201***_del are compacted into 
> regionA20161101**_1 . After that, regionB20161201**_del cannot be deleted 
> since it is not a allFile compaction. The next mob compaction, 
> regionA20161101**_1 and regionB20161201**_del will be picked up again and be 
> compacted into regionA20161101***_2. So in this case, it will cause more 
> unnecessary IOs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16841) Data loss in MOB files after cloning a snapshot and deleting that snapshot

[Matteo Bertozzi (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715370#comment-15715370
 ] 

Matteo Bertozzi commented on HBASE-16841:
-

+1

> Data loss in MOB files after cloning a snapshot and deleting that snapshot
> --
>
> Key: HBASE-16841
> URL: https://issues.apache.org/jira/browse/HBASE-16841
> Project: HBase
>  Issue Type: Bug
>  Components: mob, snapshots
>Reporter: Jingcheng Du
>Assignee: Jingcheng Du
> Attachments: HBASE-16841-V2.patch, HBASE-16841-V3.patch, 
> HBASE-16841-V4.patch, HBASE-16841-V5.patch, HBASE-16841-V6.patch, 
> HBASE-16841.patch
>
>
> Running the following steps will probably lose MOB data when working with 
> snapshots.
> 1. Create a mob-enabled table by running create 't1', {NAME => 'f1', IS_MOB 
> => true, MOB_THRESHOLD => 0}.
> 2. Put millions of data.
> 3. Run {{snapshot 't1','t1_snapshot'}} to take a snapshot for this table t1.
> 4. Run {{clone_snapshot 't1_snapshot','t1_cloned'}} to clone this snapshot.
> 5. Run {{delete_snapshot 't1_snapshot'}} to delete this snapshot.
> 6. Run {{disable 't1'}} and {{delete 't1'}} to delete the table.
> 7. Now go to the archive directory of t1, the number of .link directories is 
> different from the number of hfiles which means some data will be lost after 
> the hfile cleaner runs.
> This is because, when taking a snapshot on a enabled mob table, each region 
> flushes itself and takes a snapshot, and the mob snapshot is taken only if 
> the current region is first region of the table. At that time, the flushing 
> of some regions might not be finished, and some mob files are not flushed to 
> disk yet. Eventually some mob files are not recorded in the snapshot manifest.
> To solve this, we need to take the mob snapshot at last after the snapshots 
> on all the online and offline regions are finished in 
> {{EnabledTableSnapshotHandler}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17242) Isolate Legacy Scanner Logic

[John Leach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Leach updated HBASE-17242:
---
Status: Patch Available  (was: Open)

> Isolate Legacy Scanner Logic
> 
>
> Key: HBASE-17242
> URL: https://issues.apache.org/jira/browse/HBASE-17242
> Project: HBase
>  Issue Type: Improvement
>Reporter: John Leach
>Assignee: John Leach
> Attachments: HBASE-17242.patch
>
>
> Duplicative logic in StoreScanner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17242) Isolate Legacy Scanner Logic

[John Leach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Leach updated HBASE-17242:
---
Attachment: HBASE-17242.patch

> Isolate Legacy Scanner Logic
> 
>
> Key: HBASE-17242
> URL: https://issues.apache.org/jira/browse/HBASE-17242
> Project: HBase
>  Issue Type: Improvement
>Reporter: John Leach
>Assignee: John Leach
> Attachments: HBASE-17242.patch
>
>
> Duplicative logic in StoreScanner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-17242) Isolate Legacy Scanner Logic

[John Leach (JIRA)]

John Leach created HBASE-17242:
--

 Summary: Isolate Legacy Scanner Logic
 Key: HBASE-17242
 URL: https://issues.apache.org/jira/browse/HBASE-17242
 Project: HBase
  Issue Type: Improvement
Reporter: John Leach


Duplicative logic in StoreScanner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HBASE-17242) Isolate Legacy Scanner Logic

[John Leach (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Leach reassigned HBASE-17242:
--

Assignee: John Leach

> Isolate Legacy Scanner Logic
> 
>
> Key: HBASE-17242
> URL: https://issues.apache.org/jira/browse/HBASE-17242
> Project: HBase
>  Issue Type: Improvement
>Reporter: John Leach
>Assignee: John Leach
>
> Duplicative logic in StoreScanner.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-17241) Avoid compacting already compacted mob files with _del files

[huaxiang sun (JIRA)]

huaxiang sun created HBASE-17241:


 Summary: Avoid compacting already compacted  mob files with _del 
files
 Key: HBASE-17241
 URL: https://issues.apache.org/jira/browse/HBASE-17241
 Project: HBase
  Issue Type: Improvement
  Components: mob
Affects Versions: 2.0.0
Reporter: huaxiang sun
Assignee: huaxiang sun


Today if there is only one file in the partition, and there is no _del files, 
the file is skipped. With del file, the current logic is to compact the 
already-compacted file with _del file. Let's say there is one mob file 
regionA20161101***, which was compacted. On 12/1/2016, there is _del file 
regionB20161201**_del, mob compaction kicks in, regionA20161101*** is less than 
the threshold, and it is picked for compaction. Since there is a _del file, 
regionA20161101 and regionB20161201***_del are compacted into 
regionA20161101**_1 . After that, regionB20161201**_del cannot be deleted since 
it is not a allFile compaction. The next mob compaction, regionA20161101**_1 
and regionB20161201**_del will be picked up again and be compacted into 
regionA20161101***_2. So in this case, it will cause more unnecessary IOs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (HBASE-17231) Region#getCellCompartor sp?

[John Leach (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715305#comment-15715305
 ] 

John Leach edited comment on HBASE-17231 at 12/2/16 2:46 PM:
-

Makes sense.  Let me know, I can slap patches up.  Is there a labeling approach 
for the different branches or do I have to create different jiras?


was (Author: jleach):
Makes sense.  Let me know, I can slap patches up.  Is there a labeling approach 
for the different branches or do I have to create different jiras?

Thanks,
John

> Region#getCellCompartor sp?
> ---
>
> Key: HBASE-17231
> URL: https://issues.apache.org/jira/browse/HBASE-17231
> Project: HBase
>  Issue Type: Bug
>Reporter: John Leach
>Assignee: John Leach
>Priority: Trivial
> Attachments: HBASE-17231.patch
>
>
> Region#getCellCompartor -> Region#getCellComparator



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17231) Region#getCellCompartor sp?

[John Leach (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715305#comment-15715305
 ] 

John Leach commented on HBASE-17231:


Makes sense.  Let me know, I can slap patches up.  Is there a labeling approach 
for the different branches or do I have to create different jiras?

Thanks,
John

> Region#getCellCompartor sp?
> ---
>
> Key: HBASE-17231
> URL: https://issues.apache.org/jira/browse/HBASE-17231
> Project: HBase
>  Issue Type: Bug
>Reporter: John Leach
>Assignee: John Leach
>Priority: Trivial
> Attachments: HBASE-17231.patch
>
>
> Region#getCellCompartor -> Region#getCellComparator



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16886) hbase-client: scanner with reversed=true and small=true gets no result

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715288#comment-15715288
 ] 

Hudson commented on HBASE-16886:


SUCCESS: Integrated in Jenkins build HBase-1.2-JDK7 #77 (See 
[https://builds.apache.org/job/HBase-1.2-JDK7/77/])
HBASE-16886: hbase-client: scanner with reversed=true and small=true 
(yangzhe1991: rev c9f388bece8a89ff52baf358df63c8e5fbd7dc36)
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientSmallReversedScanner.java
* (edit) 
hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestClientSmallReversedScanner.java
* (add) 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSmallReversedScanner.java


> hbase-client: scanner with reversed=true and small=true gets no result
> --
>
> Key: HBASE-16886
> URL: https://issues.apache.org/jira/browse/HBASE-16886
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.2.3, 1.1.7, 0.98.23
>Reporter: huzheng
>Assignee: huzheng
>  Labels: patch
> Fix For: 2.0.0, 1.4.0, 1.2.5, 1.1.8
>
> Attachments: 16886.addendum, 16886.v4.branch-1.patch, 
> HBASE-16886.v0.patch, HBASE-16886.v1.patch, HBASE-16886.v2.patch, 
> HBASE-16886.v3.patch, HBASE-16886.v4.0.98.patch, 
> HBASE-16886.v4.branch-1.patch, HBASE-16886.v4.branch-1.patch, 
> HBASE-16886.v4.branch-1.patch, HBASE-16886.v4.master.patch, 
> TestReversedSmallScan.java
>
>
> Assume HBase have four regions (-oo, b), [b, c), [c, d), [d,+oo) , and all 
> rowKeys are located in region [d, +oo).  using a Reversed Small Scanner will 
> get no result.
> Attached file show this failed test case.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-16886) hbase-client: scanner with reversed=true and small=true gets no result

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-16886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715268#comment-15715268
 ] 

Hudson commented on HBASE-16886:


SUCCESS: Integrated in Jenkins build HBase-1.2-JDK8 #71 (See 
[https://builds.apache.org/job/HBase-1.2-JDK8/71/])
HBASE-16886: hbase-client: scanner with reversed=true and small=true 
(yangzhe1991: rev c9f388bece8a89ff52baf358df63c8e5fbd7dc36)
* (add) 
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSmallReversedScanner.java
* (edit) 
hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestClientSmallReversedScanner.java
* (edit) 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientSmallReversedScanner.java


> hbase-client: scanner with reversed=true and small=true gets no result
> --
>
> Key: HBASE-16886
> URL: https://issues.apache.org/jira/browse/HBASE-16886
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 1.3.0, 1.4.0, 1.2.3, 1.1.7, 0.98.23
>Reporter: huzheng
>Assignee: huzheng
>  Labels: patch
> Fix For: 2.0.0, 1.4.0, 1.2.5, 1.1.8
>
> Attachments: 16886.addendum, 16886.v4.branch-1.patch, 
> HBASE-16886.v0.patch, HBASE-16886.v1.patch, HBASE-16886.v2.patch, 
> HBASE-16886.v3.patch, HBASE-16886.v4.0.98.patch, 
> HBASE-16886.v4.branch-1.patch, HBASE-16886.v4.branch-1.patch, 
> HBASE-16886.v4.branch-1.patch, HBASE-16886.v4.master.patch, 
> TestReversedSmallScan.java
>
>
> Assume HBase have four regions (-oo, b), [b, c), [c, d), [d,+oo) , and all 
> rowKeys are located in region [d, +oo).  using a Reversed Small Scanner will 
> get no result.
> Attached file show this failed test case.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17161) MOB : Make ref cell creation more efficient

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715254#comment-15715254
 ] 

Hadoop QA commented on HBASE-17161:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
30s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 45s 
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
7s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 4s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 4s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
28m 52s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 52s 
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 96m 7s {color} 
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
35s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 144m 17s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.replication.TestSerialReplication |
|   | hadoop.hbase.regionserver.TestHRegionWithInMemoryFlush |
| Timed out junit tests | 
org.apache.hadoop.hbase.TestHColumnDescriptorDefaultVersions |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12841465/HBASE-17161_V2.patch |
| JIRA Issue | HBASE-17161 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux f7c98557446d 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / cb5c4c1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| unit | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4762/artifact/patchprocess/patch-unit-hbase-server.txt
 |
| unit test logs |  

[jira] [Commented] (HBASE-17240) ImportTsv encounters ClassNotFoundException for MasterProtos$MasterService$BlockingInterface

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715148#comment-15715148
 ] 

Hadoop QA commented on HBASE-17240:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 1s 
{color} | {color:blue} The patch file was not named according to hbase's naming 
conventions. Please see 
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for 
instructions. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m 
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s 
{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
35s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
38s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
32s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s 
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 
23m 24s {color} | {color:green} Patch does not cause any errors with Hadoop 
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 85m 36s 
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
16s {color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 119m 50s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:8d52d23 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12841461/17240.v1.txt |
| JIRA Issue | HBASE-17240 |
| Optional Tests |  asflicense  javac  javadoc  unit  findbugs  hadoopcheck  
hbaseanti  checkstyle  compile  |
| uname | Linux 190e6367888c 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
 |
| git revision | master / cb5c4c1 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4761/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output | 
https://builds.apache.org/job/PreCommit-HBASE-Build/4761/console |
| Powered by | Apache Yetus 0.3.0   http://yetus.apache.org |


This message was automatically generated.



> ImportTsv encounters ClassNotFoundException for 
> MasterProtos$MasterService$BlockingInterface
> 

[jira] [Updated] (HBASE-17232) Replace HashSet with ArrayList to accumulate delayed scanners in KVHeap and StoreScanner.

[binlijin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

binlijin updated HBASE-17232:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Replace HashSet with ArrayList to accumulate delayed scanners in KVHeap and 
> StoreScanner.
> -
>
> Key: HBASE-17232
> URL: https://issues.apache.org/jira/browse/HBASE-17232
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: binlijin
>Assignee: binlijin
> Fix For: 2.0.0
>
> Attachments: HBASE-17232.patch
>
>
> HashSet is slow than ArrayList, also generate more garbage.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17232) Replace HashSet with ArrayList to accumulate delayed scanners in KVHeap and StoreScanner.

[binlijin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715099#comment-15715099
 ] 

binlijin commented on HBASE-17232:
--

Push to master.
[~anoop.hbase] thanks very much for the review.

> Replace HashSet with ArrayList to accumulate delayed scanners in KVHeap and 
> StoreScanner.
> -
>
> Key: HBASE-17232
> URL: https://issues.apache.org/jira/browse/HBASE-17232
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: binlijin
>Assignee: binlijin
> Fix For: 2.0.0
>
> Attachments: HBASE-17232.patch
>
>
> HashSet is slow than ArrayList, also generate more garbage.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17240) ImportTsv encounters ClassNotFoundException for MasterProtos$MasterService$BlockingInterface

[Pankaj Kumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715087#comment-15715087
 ] 

Pankaj Kumar commented on HBASE-17240:
--

Which JIRA has introduced this problem?

> ImportTsv encounters ClassNotFoundException for 
> MasterProtos$MasterService$BlockingInterface
> 
>
> Key: HBASE-17240
> URL: https://issues.apache.org/jira/browse/HBASE-17240
> Project: HBase
>  Issue Type: Bug
>Reporter: Ted Yu
> Fix For: 2.0.0
>
> Attachments: 17240.v1.txt
>
>
> [~romil.choksi] reported the following problem.
> With command:
> {code}
> hbase org.apache.hadoop.hbase.mapreduce.ImportTsv -Dimporttsv.separator=, 
> -Dimporttsv.bulk.output=output -Dimporttsv.columns=HBASE_ROW_KEY,f:count 
> wordcount word_count.csv
> {code}
> The following error showed up:
> {code}
> 2016-11-29 06:39:48,861 INFO  [main] mapreduce.Job: Task Id : 
> attempt_1479850535804_0004_m_00_2, Status : FAILED
> Error: java.lang.ClassNotFoundException: 
> org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$BlockingInterface
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:264)
> at 
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:225)
> at 
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:122)
> at 
> org.apache.hadoop.hbase.mapreduce.DefaultVisibilityExpressionResolver.init(DefaultVisibilityExpressionResolver.java:75)
> at org.apache.hadoop.hbase.mapreduce.CellCreator.(CellCreator.java:48)
> at 
> org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.setup(TsvImporterMapper.java:115)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17194) Assign the new region to the idle server after splitting

[ChiaPing Tsai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ChiaPing Tsai updated HBASE-17194:
--
Status: Open  (was: Patch Available)

> Assign the new region to the idle server after splitting
> 
>
> Key: HBASE-17194
> URL: https://issues.apache.org/jira/browse/HBASE-17194
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17194.v0.patch, HBASE-17194.v1.patch, 
> HBASE-17194.v2.patch, HBASE-17194.v3.patch, HBASE-17194.v4.patch, 
> HBASE-17194.v4.patch, evaluation-v0.png, tests.xlsx
>
>
> The new regions are assigned to the random servers after splitting, but there 
> always are some idle servers which don’t be assigned any regions on the new 
> cluster. It is a bad start of load balance, hence we should give priority to 
> the idle servers for assignment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-17194) Assign the new region to the idle server after splitting

[ChiaPing Tsai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HBASE-17194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ChiaPing Tsai updated HBASE-17194:
--
Status: Patch Available  (was: Open)

> Assign the new region to the idle server after splitting
> 
>
> Key: HBASE-17194
> URL: https://issues.apache.org/jira/browse/HBASE-17194
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17194.v0.patch, HBASE-17194.v1.patch, 
> HBASE-17194.v2.patch, HBASE-17194.v3.patch, HBASE-17194.v4.patch, 
> HBASE-17194.v4.patch, evaluation-v0.png, tests.xlsx
>
>
> The new regions are assigned to the random servers after splitting, but there 
> always are some idle servers which don’t be assigned any regions on the new 
> cluster. It is a bad start of load balance, hence we should give priority to 
> the idle servers for assignment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-17194) Assign the new region to the idle server after splitting

[ChiaPing Tsai (JIRA)]

[ 
https://issues.apache.org/jira/browse/HBASE-17194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715066#comment-15715066
 ] 

ChiaPing Tsai commented on HBASE-17194:
---

TestSerialReplication pass locally.

> Assign the new region to the idle server after splitting
> 
>
> Key: HBASE-17194
> URL: https://issues.apache.org/jira/browse/HBASE-17194
> Project: HBase
>  Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-17194.v0.patch, HBASE-17194.v1.patch, 
> HBASE-17194.v2.patch, HBASE-17194.v3.patch, HBASE-17194.v4.patch, 
> HBASE-17194.v4.patch, evaluation-v0.png, tests.xlsx
>
>
> The new regions are assigned to the random servers after splitting, but there 
> always are some idle servers which don’t be assigned any regions on the new 
> cluster. It is a bad start of load balance, hence we should give priority to 
> the idle servers for assignment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)