[jira] [Commented] (HBASE-14825) HBase Ref Guide corrections of typos/misspellings
[
https://issues.apache.org/jira/browse/HBASE-14825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025799#comment-15025799
]
Hudson commented on HBASE-14825:
FAILURE: Integrated in HBase-Trunk_matrix #497 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/497/])
Commit for HBASE-14825 -- corrections of typos, misspellings, and
(mstanleyjones: rev 6a493ddff70d5247ef6a254115c94032cff584f9)
* src/main/asciidoc/_chapters/architecture.adoc
* hbase-common/src/main/resources/hbase-default.xml
* src/main/asciidoc/_chapters/ops_mgt.adoc
* hbase-rest/src/test/resources/hbase-site.xml
* src/main/asciidoc/_chapters/case_studies.adoc
* hbase-server/src/test/resources/hbase-site.xml
* src/main/asciidoc/_chapters/datamodel.adoc
* src/main/asciidoc/_chapters/external_apis.adoc
* src/main/asciidoc/_chapters/upgrading.adoc
* src/main/asciidoc/_chapters/faq.adoc
* src/main/asciidoc/_chapters/schema_design.adoc
* src/main/asciidoc/_chapters/appendix_contributing_to_documentation.adoc
* src/main/asciidoc/_chapters/hbase_mob.adoc
* src/main/asciidoc/_chapters/cp.adoc
* src/main/asciidoc/_chapters/shell.adoc
* src/main/asciidoc/_chapters/community.adoc
* src/main/asciidoc/_chapters/zookeeper.adoc
* src/main/asciidoc/_chapters/compression.adoc
* src/main/asciidoc/_chapters/mapreduce.adoc
* hbase-server/src/test/resources/hbase-site2.xml
* src/main/asciidoc/_chapters/security.adoc
* hbase-thrift/src/test/resources/hbase-site.xml
* src/main/asciidoc/_chapters/spark.adoc
* src/main/asciidoc/_chapters/developer.adoc
* src/main/asciidoc/_chapters/unit_testing.adoc
* src/main/asciidoc/_chapters/appendix_hfile_format.adoc
* src/main/asciidoc/_chapters/hbck_in_depth.adoc
* src/main/asciidoc/_chapters/performance.adoc
* hbase-spark/src/test/resources/hbase-site.xml
* src/main/asciidoc/_chapters/rpc.adoc
* src/main/asciidoc/_chapters/configuration.adoc
* src/main/asciidoc/_chapters/hbase-default.adoc
* src/main/asciidoc/_chapters/troubleshooting.adoc
> HBase Ref Guide corrections of typos/misspellings
> -
>
> Key: HBASE-14825
> URL: https://issues.apache.org/jira/browse/HBASE-14825
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.0.0
>Reporter: Daniel Vimont
>Assignee: Daniel Vimont
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-14825-v2.patch, HBASE-14825-v3.patch,
> HBASE-14825-v4.patch, HBASE-14825-v5-test.patch, HBASE-14825-v6.patch,
> HBASE-14825.patch, HBASE-14825_misty_example.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Found the following list of typos/misspellings on the book.html page, and
> thought I would make corrections to the appropriate src/main/asciidoc files
> in which they are located. (This is just a good opportunity for me to become
> familiar with submission of fixes/patches as a prelude to beginning to make
> some coding contributions. This is also my first submission to the JIRA
> system, so corrections to content/conventions are welcome!)
> [Note: I see that [~misty] may be in the midst of a reformatting task --
> HBASE-14823 -- that might involve these same asciidoc files. Please advise
> if I should wait on this task to avoid a possibly cumbersome Git
> reconciliation mess. (?)]
> Here is the list of typos/misspellings. The format of each item is (a) the
> problem is presented in brackets on the first line, and (b) the phrase (as it
> currently appears in the text) is on the second line.
> ===
> ["you" should be "your", and "Kimballs'" should be "Kimball's" (move the
> apostrophe) in the following:]
> A useful read setting config on you hadoop cluster is Aaron Kimballs'
> Configuration Parameters: What can you just ignore?
> [Period needed after "a"]
> a.k.a pseudo-distributed
> ["empty" is misspelled]
> The default value in this configuration has been intentionally left emtpy in
> order to honor the old hbase.regionserver.global.memstore.upperLimit property
> if present.
> [All occurrences of "a HBase" should be changed to "an HBase" -- 15
> occurrences found]
> ["file path are" should be "file paths are"]
> By default, all of HBase's ZooKeeper file path are configured with a relative
> path, so they will all go under this directory unless changed.
> ["times" -- plural required]
> How many time to retry attempting to write a version file before just
> aborting.
> ["separated" is misspelled]
> Each attempt is seperated by the hbase.server.thread.wakefrequency
> milliseconds.
> [space needed after quotation mark (include"limit)]
> Because this limit represents the "automatic include"limit...
> [space needed ("ashbase:metadata" should be "as hbase:metadata")]
> This helps to keep compaction of lean tables (such ashbase:meta) fast.
> [Acronym "ide" should be capitalized
[jira] [Commented] (HBASE-14871) Allow specifying the base branch for make_patch
[ https://issues.apache.org/jira/browse/HBASE-14871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025797#comment-15025797 ] Hudson commented on HBASE-14871: FAILURE: Integrated in HBase-Trunk_matrix #497 (See [https://builds.apache.org/job/HBase-Trunk_matrix/497/]) HBASE-14871 Allow specifying the base branch for make_patch (eclark: rev 4cc341b9c23183fe12225fb03d30ac975a87d07c) * dev-support/make_patch.sh > Allow specifying the base branch for make_patch > --- > > Key: HBASE-14871 > URL: https://issues.apache.org/jira/browse/HBASE-14871 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0 > > Attachments: HBASE-14871-v2.patch, HBASE-14871-v3.patch, > HBASE-14871.patch > > > Not all branches will be based off of origin/*. Lets allow the user to > specify which branch to base the patch off of. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14821) CopyTable should allow overriding more config properties for peer cluster
[
https://issues.apache.org/jira/browse/HBASE-14821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025798#comment-15025798
]
Hudson commented on HBASE-14821:
FAILURE: Integrated in HBase-Trunk_matrix #497 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/497/])
HBASE-14821 Allow configuration overrides in TableOutputFormat (garyh: rev
8b67df69486222641071ebe1cd08e220d840b17f)
* hbase-common/src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableOutputFormat.java
* hbase-common/src/test/java/org/apache/hadoop/hbase/TestHBaseConfiguration.java
> CopyTable should allow overriding more config properties for peer cluster
> -
>
> Key: HBASE-14821
> URL: https://issues.apache.org/jira/browse/HBASE-14821
> Project: HBase
> Issue Type: Improvement
> Components: mapreduce
>Reporter: Gary Helmling
>Assignee: Gary Helmling
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14821.patch
>
>
> When using CopyTable across two separate clusters, you can specify the ZK
> quorum for the destination cluster, but not much else in configuration
> overrides. This can be a problem when the cluster configurations differ,
> such as when using security with different configurations for server
> principals.
> We should provide a general way to override configuration properties for the
> peer / destination cluster. One option would be to allow use of a prefix for
> command line properties ("peer.property."). Properties matching this prefix
> will be stripped and merged to the peer configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14875) Forward port HBASE-14207 'Region was hijacked and remained in transition when RS failed to open a region and later regionplan changed to new RS on retry'
[ https://issues.apache.org/jira/browse/HBASE-14875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14875: --- Assignee: (was: Ted Yu) Fix Version/s: 1.1.3 > Forward port HBASE-14207 'Region was hijacked and remained in transition when > RS failed to open a region and later regionplan changed to new RS on retry' > - > > Key: HBASE-14875 > URL: https://issues.apache.org/jira/browse/HBASE-14875 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu > Fix For: 1.2.0, 1.3.0, 1.1.3 > > Attachments: 14875-branch-1.txt > > > HBASE-14207 was integrated to 0.98 > However, for hbase 1.x where zookeeper is involved in region assignment, the > fix from HBASE-14207 applies. > HBASE-14207 has been closed. So opening a new JIRA for forward porting. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14875) Forward port HBASE-14207 'Region was hijacked and remained in transition when RS failed to open a region and later regionplan changed to new RS on retry'
[ https://issues.apache.org/jira/browse/HBASE-14875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-14875: --- Assignee: Ted Yu > Forward port HBASE-14207 'Region was hijacked and remained in transition when > RS failed to open a region and later regionplan changed to new RS on retry' > - > > Key: HBASE-14875 > URL: https://issues.apache.org/jira/browse/HBASE-14875 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 1.2.0, 1.3.0, 1.1.3 > > Attachments: 14875-branch-1.txt > > > HBASE-14207 was integrated to 0.98 > However, for hbase 1.x where zookeeper is involved in region assignment, the > fix from HBASE-14207 applies. > HBASE-14207 has been closed. So opening a new JIRA for forward porting. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14875) Forward port HBASE-14207 'Region was hijacked and remained in transition when RS failed to open a region and later regionplan changed to new RS on retry'
[ https://issues.apache.org/jira/browse/HBASE-14875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14875: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for the review, Stephen. > Forward port HBASE-14207 'Region was hijacked and remained in transition when > RS failed to open a region and later regionplan changed to new RS on retry' > - > > Key: HBASE-14875 > URL: https://issues.apache.org/jira/browse/HBASE-14875 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu > Fix For: 1.2.0, 1.3.0, 1.1.3 > > Attachments: 14875-branch-1.txt > > > HBASE-14207 was integrated to 0.98 > However, for hbase 1.x where zookeeper is involved in region assignment, the > fix from HBASE-14207 applies. > HBASE-14207 has been closed. So opening a new JIRA for forward porting. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14769) Remove unused functions and duplicate javadocs from HBaseAdmin
[
https://issues.apache.org/jira/browse/HBASE-14769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025618#comment-15025618
]
Appy commented on HBASE-14769:
--
Confirmed that they are being inherited.
{quote}
/**
* {@inheritDoc}
* This receives puts *and* deletes.
*/
@Override
public MatchCode checkColumn(Cell cell, byte type) throws IOException {
return MatchCode.INCLUDE;
}
{quote}
Generates this
http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/regionserver/ScanWildcardColumnTracker.html#checkColumn(org.apache.hadoop.hbase.Cell,%20byte)
> Remove unused functions and duplicate javadocs from HBaseAdmin
> ---
>
> Key: HBASE-14769
> URL: https://issues.apache.org/jira/browse/HBASE-14769
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
>Assignee: Appy
> Fix For: 2.0.0
>
> Attachments: HBASE-14769-master-v2.patch,
> HBASE-14769-master-v3.patch, HBASE-14769-master-v4.patch,
> HBASE-14769-master.patch
>
>
> HBaseAdmin is marked private, so removing the functions not being used
> anywhere.
> Also, the javadocs of overridden functions are same as corresponding ones in
> Admin.java. Since javadocs are automatically inherited from the interface
> class, we can remove these redundant 100s of lines.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14861) HBASE_ZNODE_FILE on master server is overwritten by regionserver process in case of master-rs collocation
[
https://issues.apache.org/jira/browse/HBASE-14861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-14861:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
> HBASE_ZNODE_FILE on master server is overwritten by regionserver process in
> case of master-rs collocation
> --
>
> Key: HBASE-14861
> URL: https://issues.apache.org/jira/browse/HBASE-14861
> Project: HBase
> Issue Type: Bug
> Components: Operability
>Affects Versions: 2.0.0
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
> Fix For: 2.0.0
>
> Attachments: HBASE-14861.patch, HBASE-14861v2.patch,
> HBASE-14861v3.patch
>
>
> In case of master-rs collocation HBASE_ZNODE_FILE is overwritten by
> regionserver process in HRegionServer#handleReportForDutyResponse() here is
> how it looks on master server:
> {code}
> [hbase@hnode2 hbase]$ cat hbase-hbase-master.znode
> /hbase/rs/hnode2,16000,1448022074888
> {code}
> it contains regionserver znode path instead of String value of master's
> ServerName. This affects ZNodeClearer#clear() in way that will not clear
> master znode in case we detect master crash. At end this will extend
> failover time until master znode expires configured in zookeeper by
> maxSessionTimeout parameter (40s in my case).
> I have notice this on mater branch but it can be case in other branches where
> we are collocating master and rs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14825) HBase Ref Guide corrections of typos/misspellings
[
https://issues.apache.org/jira/browse/HBASE-14825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025662#comment-15025662
]
Daniel Vimont commented on HBASE-14825:
---
Excellent! I was hoping there was some kind of admin override to the
line-length limitation, when longer lines are required/reasonable.
Thanks!!
> HBase Ref Guide corrections of typos/misspellings
> -
>
> Key: HBASE-14825
> URL: https://issues.apache.org/jira/browse/HBASE-14825
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.0.0
>Reporter: Daniel Vimont
>Assignee: Daniel Vimont
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-14825-v2.patch, HBASE-14825-v3.patch,
> HBASE-14825-v4.patch, HBASE-14825-v5-test.patch, HBASE-14825-v6.patch,
> HBASE-14825.patch, HBASE-14825_misty_example.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Found the following list of typos/misspellings on the book.html page, and
> thought I would make corrections to the appropriate src/main/asciidoc files
> in which they are located. (This is just a good opportunity for me to become
> familiar with submission of fixes/patches as a prelude to beginning to make
> some coding contributions. This is also my first submission to the JIRA
> system, so corrections to content/conventions are welcome!)
> [Note: I see that [~misty] may be in the midst of a reformatting task --
> HBASE-14823 -- that might involve these same asciidoc files. Please advise
> if I should wait on this task to avoid a possibly cumbersome Git
> reconciliation mess. (?)]
> Here is the list of typos/misspellings. The format of each item is (a) the
> problem is presented in brackets on the first line, and (b) the phrase (as it
> currently appears in the text) is on the second line.
> ===
> ["you" should be "your", and "Kimballs'" should be "Kimball's" (move the
> apostrophe) in the following:]
> A useful read setting config on you hadoop cluster is Aaron Kimballs'
> Configuration Parameters: What can you just ignore?
> [Period needed after "a"]
> a.k.a pseudo-distributed
> ["empty" is misspelled]
> The default value in this configuration has been intentionally left emtpy in
> order to honor the old hbase.regionserver.global.memstore.upperLimit property
> if present.
> [All occurrences of "a HBase" should be changed to "an HBase" -- 15
> occurrences found]
> ["file path are" should be "file paths are"]
> By default, all of HBase's ZooKeeper file path are configured with a relative
> path, so they will all go under this directory unless changed.
> ["times" -- plural required]
> How many time to retry attempting to write a version file before just
> aborting.
> ["separated" is misspelled]
> Each attempt is seperated by the hbase.server.thread.wakefrequency
> milliseconds.
> [space needed after quotation mark (include"limit)]
> Because this limit represents the "automatic include"limit...
> [space needed ("ashbase:metadata" should be "as hbase:metadata")]
> This helps to keep compaction of lean tables (such ashbase:meta) fast.
> [Acronym "ide" should be capitalized for clarity: IDE]
> Setting this to true can be useful in contexts other than the other side of a
> maven generation; i.e. running in an ide.
> [RuntimeException missing an "e"]
> You'll want to set this boolean to true to avoid seeing the RuntimException
> complaint:
> [Space missing after "secure"]
> FS Permissions for the root directory in a secure(kerberos) setup.
> ["mutations" misspelled]
> ...will be created which will tail the logs and replicate the mutatations to
> region replicas for tables that have region replication > 1.
> ["it such that" should be "is such that"]
> If your working set it such that block cache does you no good...
> ["an" should be "and"]
> See the Deveraj Das an Nicolas Liochon blog post...
> [Tag "" should be ""]
> hbase.coprocessor.master.classes
> [Misspelling of "implementations"]
> Those consumers are coprocessors, phoenix, replication endpoint
> implemnetations or similar.
> [Misspelling of "cluster"]
> On upgrade, before running a rolling restart over the cluser...
> ["effect" should be "affect"]
> If NOT using BucketCache, this change does not effect you.
> [Need space after "throw"]
> This will throw`java.lang.NoSuchMethodError...
> ["habasee" should be "hbase"]
> You can pass commands to the HBase Shell in non-interactive mode (see
> hbasee.shell.noninteractive)...
> ["ie" should be "i.e."]
> Restrict the amount of resources (ie regions, tables) a namespace can consume.
> ["an" should be "and"]
> ...but can be conjured on the fly while the table is up an running.
> [Malformed link (text appears as follows when rendered in a browser):]
> Puts are executed via Table.put
[jira] [Created] (HBASE-14881) Provide a Put API that uses the provided row without coping
Jerry He created HBASE-14881: Summary: Provide a Put API that uses the provided row without coping Key: HBASE-14881 URL: https://issues.apache.org/jira/browse/HBASE-14881 Project: HBase Issue Type: Improvement Reporter: Jerry He The current available Put API always makes a copy of the rowkey. Let's provide an API that accepts an immutable byte array as rowkey without making a copy. There are cases where the caller of Put has created the immutable byte array (e.g from a serializer) and will not change it for the Put duration. We can avoid making a copy again. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14821) CopyTable should allow overriding more config properties for peer cluster
[
https://issues.apache.org/jira/browse/HBASE-14821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025857#comment-15025857
]
Hudson commented on HBASE-14821:
SUCCESS: Integrated in HBase-1.2-IT #306 (See
[https://builds.apache.org/job/HBase-1.2-IT/306/])
HBASE-14821 Allow configuration overrides in TableOutputFormat (garyh: rev
3ce461f87cbbb10901208704e54536daf7f5c32e)
* hbase-common/src/test/java/org/apache/hadoop/hbase/TestHBaseConfiguration.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableOutputFormat.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java
> CopyTable should allow overriding more config properties for peer cluster
> -
>
> Key: HBASE-14821
> URL: https://issues.apache.org/jira/browse/HBASE-14821
> Project: HBase
> Issue Type: Improvement
> Components: mapreduce
>Reporter: Gary Helmling
>Assignee: Gary Helmling
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14821.patch
>
>
> When using CopyTable across two separate clusters, you can specify the ZK
> quorum for the destination cluster, but not much else in configuration
> overrides. This can be a problem when the cluster configurations differ,
> such as when using security with different configurations for server
> principals.
> We should provide a general way to override configuration properties for the
> peer / destination cluster. One option would be to allow use of a prefix for
> command line properties ("peer.property."). Properties matching this prefix
> will be stripped and merged to the peer configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14875) Forward port HBASE-14207 'Region was hijacked and remained in transition when RS failed to open a region and later regionplan changed to new RS on retry'
[ https://issues.apache.org/jira/browse/HBASE-14875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025856#comment-15025856 ] Hudson commented on HBASE-14875: SUCCESS: Integrated in HBase-1.2-IT #306 (See [https://builds.apache.org/job/HBase-1.2-IT/306/]) HBASE-14875 Forward port HBASE-14207 'Region was hijacked and remained (tedyu: rev a4613f03cfb8a006263cb10b2891fe60a21cf6c8) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java > Forward port HBASE-14207 'Region was hijacked and remained in transition when > RS failed to open a region and later regionplan changed to new RS on retry' > - > > Key: HBASE-14875 > URL: https://issues.apache.org/jira/browse/HBASE-14875 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu > Fix For: 1.2.0, 1.3.0, 1.1.3 > > Attachments: 14875-branch-1.txt > > > HBASE-14207 was integrated to 0.98 > However, for hbase 1.x where zookeeper is involved in region assignment, the > fix from HBASE-14207 applies. > HBASE-14207 has been closed. So opening a new JIRA for forward porting. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14207) Region was hijacked and remained in transition when RS failed to open a region and later regionplan changed to new RS on retry
[
https://issues.apache.org/jira/browse/HBASE-14207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025858#comment-15025858
]
Hudson commented on HBASE-14207:
SUCCESS: Integrated in HBase-1.2-IT #306 (See
[https://builds.apache.org/job/HBase-1.2-IT/306/])
HBASE-14875 Forward port HBASE-14207 'Region was hijacked and remained (tedyu:
rev a4613f03cfb8a006263cb10b2891fe60a21cf6c8)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
> Region was hijacked and remained in transition when RS failed to open a
> region and later regionplan changed to new RS on retry
> --
>
> Key: HBASE-14207
> URL: https://issues.apache.org/jira/browse/HBASE-14207
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 0.98.6
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 0.98.15
>
> Attachments: HBASE-14207-0.98-V2.patch, HBASE-14207-0.98-V2.patch,
> HBASE-14207-0.98.patch
>
>
> On production environment, following events happened
> 1. Master is trying to assign a region to RS, but due to
> KeeperException$SessionExpiredException RS failed to open the region.
> In RS log, saw multiple WARN log related to
> KeeperException$SessionExpiredException
> > KeeperErrorCode = Session expired for
> /hbase/region-in-transition/08f1935d652e5dbdac09b423b8f9401b
> > Unable to get data of znode
> /hbase/region-in-transition/08f1935d652e5dbdac09b423b8f9401b
> 2. Master retried to assign the region to same RS, but RS again failed.
> 3. On second retry new plan formed and this time plan destination (RS) is
> different, so master send the request to new RS to open the region. But new
> RS failed to open the region as there was server mismatch in ZNODE than the
> expected current server name.
> Logs Snippet:
> {noformat}
> HM
> 2015-07-14 03:50:29,759 | INFO | master:T101PC03VM13:21300 | Processing
> 08f1935d652e5dbdac09b423b8f9401b in state: M_ZK_REGION_OFFLINE |
> org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:644)
> 2015-07-14 03:50:29,759 | INFO | master:T101PC03VM13:21300 | Transitioned
> {08f1935d652e5dbdac09b423b8f9401b state=OFFLINE, ts=1436817029679,
> server=null} to {08f1935d652e5dbdac09b423b8f9401b state=PENDING_OPEN,
> ts=1436817029759, server=T101PC03VM13,21302,1436816690692} |
> org.apache.hadoop.hbase.master.RegionStates.updateRegionState(RegionStates.java:327)
> 2015-07-14 03:50:29,760 | INFO | master:T101PC03VM13:21300 | Processed
> region 08f1935d652e5dbdac09b423b8f9401b in state M_ZK_REGION_OFFLINE, on
> server: T101PC03VM13,21302,1436816690692 |
> org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:768)
> 2015-07-14 03:50:29,800 | INFO |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Assigning
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> T101PC03VM13,21302,1436816690692 |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1983)
> 2015-07-14 03:50:29,801 | WARN |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Failed assignment of
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> T101PC03VM13,21302,1436816690692, trying to assign elsewhere instead; try=1
> of 10 |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2077)
> 2015-07-14 03:50:29,802 | INFO |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Trying to re-assign
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> the same failed server. |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2123)
> 2015-07-14 03:50:31,804 | INFO |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Assigning
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> T101PC03VM13,21302,1436816690692 |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1983)
> 2015-07-14 03:50:31,806 | WARN |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Failed assignment of
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> T101PC03VM13,21302,1436816690692, trying to assign elsewhere instead; try=2
> of 10 |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2077)
> 2015-07-14 03:50:31,807 | INFO |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Transitioned
> {08f1935d652e5dbdac09b423b8f9401b state=PENDING_OPEN, ts=1436817031804,
> server=T101PC03VM13,21302,1436816690692} to {08f1935d652e5dbdac09b423b8f9401b
>
[jira] [Updated] (HBASE-8978) Restore TestLogRollAbort upon review
[ https://issues.apache.org/jira/browse/HBASE-8978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nate Edel updated HBASE-8978: - Description: Gary, would you mind taking a look? No rush. The test was added by HBASE-4282. It is a good test against data loss but it is around deferred flush which has changed a bunch since test was originally written. I removed it over in hbase-8977 because I caught it hanging and breaking our build (hbase-8939). If you run it, you'll see it can't shutdown. We are bound up waiting on a regionserver to exit. was: Gary, would you mind taking a look? No rush. The test was added by HBASE-4282. It is a good test against data loss but it is around deferred flush which has changed a bunch since test was originally written. I removed it over in hbase-8977 because I caught it hanging and breaking our build (hbase-8939). If you run it, you'll see it can't shutdown. We are bound up waiting on a regionserver to exit. > Restore TestLogRollAbort upon review > > > Key: HBASE-8978 > URL: https://issues.apache.org/jira/browse/HBASE-8978 > Project: HBase > Issue Type: Task > Components: test >Reporter: stack >Assignee: Gary Helmling > Fix For: 0.98.0, 0.95.2 > > Attachments: 8978.txt, 8978v2.txt > > > Gary, would you mind taking a look? No rush. > The test was added by HBASE-4282. It is a good test against data loss but it > is around deferred flush which has changed a bunch since test was originally > written. > I removed it over in hbase-8977 because I caught it hanging and breaking our > build (hbase-8939). If you run it, you'll see it can't shutdown. We are > bound up waiting on a regionserver to exit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14860) Improve BoundedByteBufferPool; make lockless
[ https://issues.apache.org/jira/browse/HBASE-14860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hiroshi Ikeda updated HBASE-14860: -- Attachment: HBASE-14860-addendum.patch Added an addendum patch; Fixed the bug and added a test for conversion methods, which contained the bug. Sorry for troubling and thanks all. > Improve BoundedByteBufferPool; make lockless > > > Key: HBASE-14860 > URL: https://issues.apache.org/jira/browse/HBASE-14860 > Project: HBase > Issue Type: Improvement >Reporter: Hiroshi Ikeda >Assignee: Hiroshi Ikeda >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14860-addendum.patch, HBASE-14860.patch > > > Make it unblocking. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14769) Remove unused functions and duplicate javadocs from HBaseAdmin
[
https://issues.apache.org/jira/browse/HBASE-14769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025928#comment-15025928
]
Hadoop QA commented on HBASE-14769:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12774193/HBASE-14769-master-v5.patch
against master branch at commit 1178c4b89bad28210fe8d2c207cce814d6fd9a21.
ATTACHMENT ID: 12774193
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 29 new
or modified tests.
{color:red}-1 javac{color}. The patch appears to cause mvn compile goal to
fail with Hadoop version 2.4.0.
Compilation errors resume:
[ERROR] COMPILATION ERROR :
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java:[47,35]
package com.google.gson.annotations does not exist
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestFIFOCompactionPolicy.java:[155,14]
method tableExists in class org.apache.hadoop.hbase.client.HBaseAdmin cannot
be applied to given types;
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestFIFOCompactionPolicy.java:[156,12]
method disableTable in class org.apache.hadoop.hbase.client.HBaseAdmin cannot
be applied to given types;
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestFIFOCompactionPolicy.java:[157,12]
method deleteTable in class org.apache.hadoop.hbase.client.HBaseAdmin cannot
be applied to given types;
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestFIFOCompactionPolicy.java:[184,14]
method tableExists in class org.apache.hadoop.hbase.client.HBaseAdmin cannot
be applied to given types;
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestFIFOCompactionPolicy.java:[185,12]
method disableTable in class org.apache.hadoop.hbase.client.HBaseAdmin cannot
be applied to given types;
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestFIFOCompactionPolicy.java:[186,12]
method deleteTable in class org.apache.hadoop.hbase.client.HBaseAdmin cannot
be applied to given types;
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestFIFOCompactionPolicy.java:[215,14]
method tableExists in class org.apache.hadoop.hbase.client.HBaseAdmin cannot
be applied to given types;
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestFIFOCompactionPolicy.java:[216,12]
method disableTable in class org.apache.hadoop.hbase.client.HBaseAdmin cannot
be applied to given types;
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestFIFOCompactionPolicy.java:[217,12]
method deleteTable in class org.apache.hadoop.hbase.client.HBaseAdmin cannot
be applied to given types;
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.2:testCompile
(default-testCompile) on project hbase-server: Compilation failure: Compilation
failure:
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java:[47,35]
package com.google.gson.annotations does not exist
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestFIFOCompactionPolicy.java:[155,14]
method tableExists in class org.apache.hadoop.hbase.client.HBaseAdmin cannot
be applied to given types;
[ERROR] required: org.apache.hadoop.hbase.TableName
[ERROR] found: java.lang.String
[ERROR] reason: actual argument java.lang.String cannot be converted to
org.apache.hadoop.hbase.TableName by method invocation conversion
[ERROR]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestFIFOCompactionPolicy.java:[156,12]
method disableTable in class org.apache.hadoop.hbase.client.HBaseAdmin cannot
be applied to given types;
[ERROR] required:
[jira] [Created] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without coping
Jerry He created HBASE-14882:
Summary: Provide a Put API that adds the provided family,
qualifier, value without coping
Key: HBASE-14882
URL: https://issues.apache.org/jira/browse/HBASE-14882
Project: HBase
Issue Type: Improvement
Reporter: Jerry He
In the Put API, we have addImmutable()
{code}
/**
* See {@link #addColumn(byte[], byte[], byte[])}. This version expects
* that the underlying arrays won't change. It's intended
* for usage internal HBase to and for advanced client applications.
*/
public Put addImmutable(byte [] family, byte [] qualifier, byte [] value)
{code}
But in the implementation the row, family. qualifier and value are still being
copied locally to create kv.
Hopefully we should provide an API that truely uses immutable family, qualifier
and value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-14860) Improve BoundedByteBufferPool; make lockless
[ https://issues.apache.org/jira/browse/HBASE-14860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-14860. --- Resolution: Fixed I pushed the addendum. Thanks [~ikeda] Nice test. > Improve BoundedByteBufferPool; make lockless > > > Key: HBASE-14860 > URL: https://issues.apache.org/jira/browse/HBASE-14860 > Project: HBase > Issue Type: Improvement >Reporter: Hiroshi Ikeda >Assignee: Hiroshi Ikeda >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14860-addendum.patch, HBASE-14860.patch > > > Make it unblocking. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14821) CopyTable should allow overriding more config properties for peer cluster
[
https://issues.apache.org/jira/browse/HBASE-14821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026028#comment-15026028
]
Hudson commented on HBASE-14821:
FAILURE: Integrated in HBase-1.2 #399 (See
[https://builds.apache.org/job/HBase-1.2/399/])
HBASE-14821 Allow configuration overrides in TableOutputFormat (garyh: rev
3ce461f87cbbb10901208704e54536daf7f5c32e)
* hbase-common/src/test/java/org/apache/hadoop/hbase/TestHBaseConfiguration.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableOutputFormat.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java
> CopyTable should allow overriding more config properties for peer cluster
> -
>
> Key: HBASE-14821
> URL: https://issues.apache.org/jira/browse/HBASE-14821
> Project: HBase
> Issue Type: Improvement
> Components: mapreduce
>Reporter: Gary Helmling
>Assignee: Gary Helmling
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14821.patch
>
>
> When using CopyTable across two separate clusters, you can specify the ZK
> quorum for the destination cluster, but not much else in configuration
> overrides. This can be a problem when the cluster configurations differ,
> such as when using security with different configurations for server
> principals.
> We should provide a general way to override configuration properties for the
> peer / destination cluster. One option would be to allow use of a prefix for
> command line properties ("peer.property."). Properties matching this prefix
> will be stripped and merged to the peer configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14875) Forward port HBASE-14207 'Region was hijacked and remained in transition when RS failed to open a region and later regionplan changed to new RS on retry'
[ https://issues.apache.org/jira/browse/HBASE-14875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026027#comment-15026027 ] Hudson commented on HBASE-14875: FAILURE: Integrated in HBase-1.2 #399 (See [https://builds.apache.org/job/HBase-1.2/399/]) HBASE-14875 Forward port HBASE-14207 'Region was hijacked and remained (tedyu: rev a4613f03cfb8a006263cb10b2891fe60a21cf6c8) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java > Forward port HBASE-14207 'Region was hijacked and remained in transition when > RS failed to open a region and later regionplan changed to new RS on retry' > - > > Key: HBASE-14875 > URL: https://issues.apache.org/jira/browse/HBASE-14875 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu > Fix For: 1.2.0, 1.3.0, 1.1.3 > > Attachments: 14875-branch-1.txt > > > HBASE-14207 was integrated to 0.98 > However, for hbase 1.x where zookeeper is involved in region assignment, the > fix from HBASE-14207 applies. > HBASE-14207 has been closed. So opening a new JIRA for forward porting. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14207) Region was hijacked and remained in transition when RS failed to open a region and later regionplan changed to new RS on retry
[
https://issues.apache.org/jira/browse/HBASE-14207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026029#comment-15026029
]
Hudson commented on HBASE-14207:
FAILURE: Integrated in HBase-1.2 #399 (See
[https://builds.apache.org/job/HBase-1.2/399/])
HBASE-14875 Forward port HBASE-14207 'Region was hijacked and remained (tedyu:
rev a4613f03cfb8a006263cb10b2891fe60a21cf6c8)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
> Region was hijacked and remained in transition when RS failed to open a
> region and later regionplan changed to new RS on retry
> --
>
> Key: HBASE-14207
> URL: https://issues.apache.org/jira/browse/HBASE-14207
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 0.98.6
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 0.98.15
>
> Attachments: HBASE-14207-0.98-V2.patch, HBASE-14207-0.98-V2.patch,
> HBASE-14207-0.98.patch
>
>
> On production environment, following events happened
> 1. Master is trying to assign a region to RS, but due to
> KeeperException$SessionExpiredException RS failed to open the region.
> In RS log, saw multiple WARN log related to
> KeeperException$SessionExpiredException
> > KeeperErrorCode = Session expired for
> /hbase/region-in-transition/08f1935d652e5dbdac09b423b8f9401b
> > Unable to get data of znode
> /hbase/region-in-transition/08f1935d652e5dbdac09b423b8f9401b
> 2. Master retried to assign the region to same RS, but RS again failed.
> 3. On second retry new plan formed and this time plan destination (RS) is
> different, so master send the request to new RS to open the region. But new
> RS failed to open the region as there was server mismatch in ZNODE than the
> expected current server name.
> Logs Snippet:
> {noformat}
> HM
> 2015-07-14 03:50:29,759 | INFO | master:T101PC03VM13:21300 | Processing
> 08f1935d652e5dbdac09b423b8f9401b in state: M_ZK_REGION_OFFLINE |
> org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:644)
> 2015-07-14 03:50:29,759 | INFO | master:T101PC03VM13:21300 | Transitioned
> {08f1935d652e5dbdac09b423b8f9401b state=OFFLINE, ts=1436817029679,
> server=null} to {08f1935d652e5dbdac09b423b8f9401b state=PENDING_OPEN,
> ts=1436817029759, server=T101PC03VM13,21302,1436816690692} |
> org.apache.hadoop.hbase.master.RegionStates.updateRegionState(RegionStates.java:327)
> 2015-07-14 03:50:29,760 | INFO | master:T101PC03VM13:21300 | Processed
> region 08f1935d652e5dbdac09b423b8f9401b in state M_ZK_REGION_OFFLINE, on
> server: T101PC03VM13,21302,1436816690692 |
> org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:768)
> 2015-07-14 03:50:29,800 | INFO |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Assigning
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> T101PC03VM13,21302,1436816690692 |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1983)
> 2015-07-14 03:50:29,801 | WARN |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Failed assignment of
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> T101PC03VM13,21302,1436816690692, trying to assign elsewhere instead; try=1
> of 10 |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2077)
> 2015-07-14 03:50:29,802 | INFO |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Trying to re-assign
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> the same failed server. |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2123)
> 2015-07-14 03:50:31,804 | INFO |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Assigning
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> T101PC03VM13,21302,1436816690692 |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1983)
> 2015-07-14 03:50:31,806 | WARN |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Failed assignment of
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> T101PC03VM13,21302,1436816690692, trying to assign elsewhere instead; try=2
> of 10 |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2077)
> 2015-07-14 03:50:31,807 | INFO |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Transitioned
> {08f1935d652e5dbdac09b423b8f9401b state=PENDING_OPEN, ts=1436817031804,
> server=T101PC03VM13,21302,1436816690692} to {08f1935d652e5dbdac09b423b8f9401b
> state=OFFLINE,
[jira] [Commented] (HBASE-14875) Forward port HBASE-14207 'Region was hijacked and remained in transition when RS failed to open a region and later regionplan changed to new RS on retry'
[ https://issues.apache.org/jira/browse/HBASE-14875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026083#comment-15026083 ] Hudson commented on HBASE-14875: FAILURE: Integrated in HBase-1.3 #396 (See [https://builds.apache.org/job/HBase-1.3/396/]) HBASE-14875 Forward port HBASE-14207 'Region was hijacked and remained (tedyu: rev 1917516ffd7a5d5c9f4402c820e7b1ce8811e716) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java > Forward port HBASE-14207 'Region was hijacked and remained in transition when > RS failed to open a region and later regionplan changed to new RS on retry' > - > > Key: HBASE-14875 > URL: https://issues.apache.org/jira/browse/HBASE-14875 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu > Fix For: 1.2.0, 1.3.0, 1.1.3 > > Attachments: 14875-branch-1.txt > > > HBASE-14207 was integrated to 0.98 > However, for hbase 1.x where zookeeper is involved in region assignment, the > fix from HBASE-14207 applies. > HBASE-14207 has been closed. So opening a new JIRA for forward porting. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14207) Region was hijacked and remained in transition when RS failed to open a region and later regionplan changed to new RS on retry
[
https://issues.apache.org/jira/browse/HBASE-14207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026084#comment-15026084
]
Hudson commented on HBASE-14207:
FAILURE: Integrated in HBase-1.3 #396 (See
[https://builds.apache.org/job/HBase-1.3/396/])
HBASE-14875 Forward port HBASE-14207 'Region was hijacked and remained (tedyu:
rev 1917516ffd7a5d5c9f4402c820e7b1ce8811e716)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
> Region was hijacked and remained in transition when RS failed to open a
> region and later regionplan changed to new RS on retry
> --
>
> Key: HBASE-14207
> URL: https://issues.apache.org/jira/browse/HBASE-14207
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 0.98.6
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 0.98.15
>
> Attachments: HBASE-14207-0.98-V2.patch, HBASE-14207-0.98-V2.patch,
> HBASE-14207-0.98.patch
>
>
> On production environment, following events happened
> 1. Master is trying to assign a region to RS, but due to
> KeeperException$SessionExpiredException RS failed to open the region.
> In RS log, saw multiple WARN log related to
> KeeperException$SessionExpiredException
> > KeeperErrorCode = Session expired for
> /hbase/region-in-transition/08f1935d652e5dbdac09b423b8f9401b
> > Unable to get data of znode
> /hbase/region-in-transition/08f1935d652e5dbdac09b423b8f9401b
> 2. Master retried to assign the region to same RS, but RS again failed.
> 3. On second retry new plan formed and this time plan destination (RS) is
> different, so master send the request to new RS to open the region. But new
> RS failed to open the region as there was server mismatch in ZNODE than the
> expected current server name.
> Logs Snippet:
> {noformat}
> HM
> 2015-07-14 03:50:29,759 | INFO | master:T101PC03VM13:21300 | Processing
> 08f1935d652e5dbdac09b423b8f9401b in state: M_ZK_REGION_OFFLINE |
> org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:644)
> 2015-07-14 03:50:29,759 | INFO | master:T101PC03VM13:21300 | Transitioned
> {08f1935d652e5dbdac09b423b8f9401b state=OFFLINE, ts=1436817029679,
> server=null} to {08f1935d652e5dbdac09b423b8f9401b state=PENDING_OPEN,
> ts=1436817029759, server=T101PC03VM13,21302,1436816690692} |
> org.apache.hadoop.hbase.master.RegionStates.updateRegionState(RegionStates.java:327)
> 2015-07-14 03:50:29,760 | INFO | master:T101PC03VM13:21300 | Processed
> region 08f1935d652e5dbdac09b423b8f9401b in state M_ZK_REGION_OFFLINE, on
> server: T101PC03VM13,21302,1436816690692 |
> org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:768)
> 2015-07-14 03:50:29,800 | INFO |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Assigning
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> T101PC03VM13,21302,1436816690692 |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1983)
> 2015-07-14 03:50:29,801 | WARN |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Failed assignment of
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> T101PC03VM13,21302,1436816690692, trying to assign elsewhere instead; try=1
> of 10 |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2077)
> 2015-07-14 03:50:29,802 | INFO |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Trying to re-assign
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> the same failed server. |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2123)
> 2015-07-14 03:50:31,804 | INFO |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Assigning
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> T101PC03VM13,21302,1436816690692 |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1983)
> 2015-07-14 03:50:31,806 | WARN |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Failed assignment of
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> T101PC03VM13,21302,1436816690692, trying to assign elsewhere instead; try=2
> of 10 |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2077)
> 2015-07-14 03:50:31,807 | INFO |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Transitioned
> {08f1935d652e5dbdac09b423b8f9401b state=PENDING_OPEN, ts=1436817031804,
> server=T101PC03VM13,21302,1436816690692} to {08f1935d652e5dbdac09b423b8f9401b
> state=OFFLINE,
[jira] [Commented] (HBASE-14463) Severe performance downgrade when parallel reading a single key from BucketCache
[
https://issues.apache.org/jira/browse/HBASE-14463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026111#comment-15026111
]
Anoop Sam John commented on HBASE-14463:
0.98 backport is not so straight as it need HBASE-14268. No need to keep the
fixed jira open. For 98 backport pls open a new jira referring to this. I will
close this.
> Severe performance downgrade when parallel reading a single key from
> BucketCache
>
>
> Key: HBASE-14463
> URL: https://issues.apache.org/jira/browse/HBASE-14463
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.14, 1.1.2
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14463-branch-1-v12.txt, GC_with_WeakObjectPool.png,
> HBASE-14463.branch-0.98.patch, HBASE-14463.patch, HBASE-14463_v11.patch,
> HBASE-14463_v12.patch, HBASE-14463_v12.patch, HBASE-14463_v2.patch,
> HBASE-14463_v3.patch, HBASE-14463_v4.patch, HBASE-14463_v5.patch,
> TestBucketCache-new_with_IdLock.png,
> TestBucketCache-new_with_IdReadWriteLock.png,
> TestBucketCache_with_IdLock-latest.png, TestBucketCache_with_IdLock.png,
> TestBucketCache_with_IdReadWriteLock-latest.png,
> TestBucketCache_with_IdReadWriteLock-resolveLockLeak.png,
> TestBucketCache_with_IdReadWriteLock.png, pe_use_same_keys.patch,
> test-results.tar.gz
>
>
> We store feature data of online items in HBase, do machine learning on these
> features, and supply the outputs to our online search engine. In such
> scenario we will launch hundreds of yarn workers and each worker will read
> all features of one item(i.e. single rowkey in HBase), so there'll be heavy
> parallel reading on a single rowkey.
> We were using LruCache but start to try BucketCache recently to resolve gc
> issue, and just as titled we have observed severe performance downgrade.
> After some analytics we found the root cause is the lock in
> BucketCache#getBlock, as shown below
> {code}
> try {
> lockEntry = offsetLock.getLockEntry(bucketEntry.offset());
> // ...
> if (bucketEntry.equals(backingMap.get(key))) {
> // ...
> int len = bucketEntry.getLength();
> Cacheable cachedBlock = ioEngine.read(bucketEntry.offset(), len,
> bucketEntry.deserializerReference(this.deserialiserMap));
> {code}
> Since ioEnging.read involves array copy, it's much more time-costed than the
> operation in LruCache. And since we're using synchronized in
> IdLock#getLockEntry, parallel read dropping on the same bucket would be
> executed in serial, which causes a really bad performance.
> To resolve the problem, we propose to use ReentranceReadWriteLock in
> BucketCache, and introduce a new class called IdReadWriteLock to implement it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14463) Severe performance downgrade when parallel reading a single key from BucketCache
[
https://issues.apache.org/jira/browse/HBASE-14463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-14463:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: (was: 0.98.17)
Status: Resolved (was: Patch Available)
> Severe performance downgrade when parallel reading a single key from
> BucketCache
>
>
> Key: HBASE-14463
> URL: https://issues.apache.org/jira/browse/HBASE-14463
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.14, 1.1.2
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14463-branch-1-v12.txt, GC_with_WeakObjectPool.png,
> HBASE-14463.branch-0.98.patch, HBASE-14463.patch, HBASE-14463_v11.patch,
> HBASE-14463_v12.patch, HBASE-14463_v12.patch, HBASE-14463_v2.patch,
> HBASE-14463_v3.patch, HBASE-14463_v4.patch, HBASE-14463_v5.patch,
> TestBucketCache-new_with_IdLock.png,
> TestBucketCache-new_with_IdReadWriteLock.png,
> TestBucketCache_with_IdLock-latest.png, TestBucketCache_with_IdLock.png,
> TestBucketCache_with_IdReadWriteLock-latest.png,
> TestBucketCache_with_IdReadWriteLock-resolveLockLeak.png,
> TestBucketCache_with_IdReadWriteLock.png, pe_use_same_keys.patch,
> test-results.tar.gz
>
>
> We store feature data of online items in HBase, do machine learning on these
> features, and supply the outputs to our online search engine. In such
> scenario we will launch hundreds of yarn workers and each worker will read
> all features of one item(i.e. single rowkey in HBase), so there'll be heavy
> parallel reading on a single rowkey.
> We were using LruCache but start to try BucketCache recently to resolve gc
> issue, and just as titled we have observed severe performance downgrade.
> After some analytics we found the root cause is the lock in
> BucketCache#getBlock, as shown below
> {code}
> try {
> lockEntry = offsetLock.getLockEntry(bucketEntry.offset());
> // ...
> if (bucketEntry.equals(backingMap.get(key))) {
> // ...
> int len = bucketEntry.getLength();
> Cacheable cachedBlock = ioEngine.read(bucketEntry.offset(), len,
> bucketEntry.deserializerReference(this.deserialiserMap));
> {code}
> Since ioEnging.read involves array copy, it's much more time-costed than the
> operation in LruCache. And since we're using synchronized in
> IdLock#getLockEntry, parallel read dropping on the same bucket would be
> executed in serial, which causes a really bad performance.
> To resolve the problem, we propose to use ReentranceReadWriteLock in
> BucketCache, and introduce a new class called IdReadWriteLock to implement it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14463) Severe performance downgrade when parallel reading a single key from BucketCache
[
https://issues.apache.org/jira/browse/HBASE-14463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026120#comment-15026120
]
Yu Li commented on HBASE-14463:
---
OK, thanks for the follow-up [~anoop.hbase]!
[~apurtell], whenever you want this in 98, just let me know.
> Severe performance downgrade when parallel reading a single key from
> BucketCache
>
>
> Key: HBASE-14463
> URL: https://issues.apache.org/jira/browse/HBASE-14463
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.14, 1.1.2
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14463-branch-1-v12.txt, GC_with_WeakObjectPool.png,
> HBASE-14463.branch-0.98.patch, HBASE-14463.patch, HBASE-14463_v11.patch,
> HBASE-14463_v12.patch, HBASE-14463_v12.patch, HBASE-14463_v2.patch,
> HBASE-14463_v3.patch, HBASE-14463_v4.patch, HBASE-14463_v5.patch,
> TestBucketCache-new_with_IdLock.png,
> TestBucketCache-new_with_IdReadWriteLock.png,
> TestBucketCache_with_IdLock-latest.png, TestBucketCache_with_IdLock.png,
> TestBucketCache_with_IdReadWriteLock-latest.png,
> TestBucketCache_with_IdReadWriteLock-resolveLockLeak.png,
> TestBucketCache_with_IdReadWriteLock.png, pe_use_same_keys.patch,
> test-results.tar.gz
>
>
> We store feature data of online items in HBase, do machine learning on these
> features, and supply the outputs to our online search engine. In such
> scenario we will launch hundreds of yarn workers and each worker will read
> all features of one item(i.e. single rowkey in HBase), so there'll be heavy
> parallel reading on a single rowkey.
> We were using LruCache but start to try BucketCache recently to resolve gc
> issue, and just as titled we have observed severe performance downgrade.
> After some analytics we found the root cause is the lock in
> BucketCache#getBlock, as shown below
> {code}
> try {
> lockEntry = offsetLock.getLockEntry(bucketEntry.offset());
> // ...
> if (bucketEntry.equals(backingMap.get(key))) {
> // ...
> int len = bucketEntry.getLength();
> Cacheable cachedBlock = ioEngine.read(bucketEntry.offset(), len,
> bucketEntry.deserializerReference(this.deserialiserMap));
> {code}
> Since ioEnging.read involves array copy, it's much more time-costed than the
> operation in LruCache. And since we're using synchronized in
> IdLock#getLockEntry, parallel read dropping on the same bucket would be
> executed in serial, which causes a really bad performance.
> To resolve the problem, we propose to use ReentranceReadWriteLock in
> BucketCache, and introduce a new class called IdReadWriteLock to implement it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14703) not collect stats when call HTable.mutateRow
[
https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026145#comment-15026145
]
Heng Chen commented on HBASE-14703:
---
Oh, i figure out why TestCheckAndMutate failed with patch_v6.
It is because of here which you fix in
HBASE-14703-v6_with-check-and-mutate.patch
{code}
- } catch(NoSuchColumnFamilyException e) {
+ } catch (RetriesExhaustedWithDetailsException e) {
+try {
+ throw e.getCause(0);
+} catch (NoSuchColumnFamilyException e1) {
+ // expected
+}
{code}
So as your logic, if processed is false and no exception in
ClientProtos.MultiResponse, there will no exception throw out (you remove it
in ResponseConverter), and there is no retry.
It is different with original logic (original logic will retry).
IMO we should NOT retry in checkAndXXX if processed is false and just return
false to users, but we need to confirm it. wdyt?
I will extract a new callable class to avoid useless repeated code. The
callable class will be useful in future when we unify other calls.
btw.
Takes advantage of the 'exists' flag in Result to track processed state is a
good idea. Thanks [~jesse_yates] for your nice patch.
> not collect stats when call HTable.mutateRow
> -
>
> Key: HBASE-14703
> URL: https://issues.apache.org/jira/browse/HBASE-14703
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
>Assignee: Heng Chen
> Fix For: 2.0.0
>
> Attachments: HBASE-14702_v5.2_addendum-addendum.patch,
> HBASE-14703-5.2-addendum.patch, HBASE-14703-async.patch,
> HBASE-14703-start.patch, HBASE-14703-v4.1.patch, HBASE-14703-v4.patch,
> HBASE-14703-v6_with-check-and-mutate.patch, HBASE-14703.patch,
> HBASE-14703_v1.patch, HBASE-14703_v2.patch, HBASE-14703_v3.patch,
> HBASE-14703_v5.1.patch, HBASE-14703_v5.2.patch, HBASE-14703_v5.patch,
> HBASE-14703_v6.patch
>
>
> In {{AsyncProcess.SingleServerRequestRunnable}}, it seems we update
> serverStatistics twice.
> The first one is that we wrapper {{RetryingCallable}} by
> {{StatsTrackingRpcRetryingCaller}}, and do serverStatistics update when we
> call {{callWithRetries}} and {{callWithoutRetries}}. Relates code like below:
> {code}
> @Override
> public T callWithRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> @Override
> public T callWithoutRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> {code}
> The secondary one is after we get response, in {{receiveMultiAction}}, we do
> update again.
> {code}
> // update the stats about the region, if its a user table. We don't want to
> slow down
> // updates to meta tables, especially from internal updates (master, etc).
> if (AsyncProcess.this.connection.getStatisticsTracker() != null) {
> result = ResultStatsUtil.updateStats(result,
> AsyncProcess.this.connection.getStatisticsTracker(), server, regionName);
> }
> {code}
> It seems that {{StatsTrackingRpcRetryingCaller}} is NOT necessary, remove it?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14861) HBASE_ZNODE_FILE on master server is overwritten by regionserver process in case of master-rs collocation
[
https://issues.apache.org/jira/browse/HBASE-14861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026164#comment-15026164
]
Hudson commented on HBASE-14861:
FAILURE: Integrated in HBase-Trunk_matrix #498 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/498/])
HBASE-14861 HBASE_ZNODE_FILE on master server is overwritten by (tedyu: rev
1178c4b89bad28210fe8d2c207cce814d6fd9a21)
* hbase-server/src/main/java/org/apache/hadoop/hbase/ZNodeClearer.java
> HBASE_ZNODE_FILE on master server is overwritten by regionserver process in
> case of master-rs collocation
> --
>
> Key: HBASE-14861
> URL: https://issues.apache.org/jira/browse/HBASE-14861
> Project: HBase
> Issue Type: Bug
> Components: Operability
>Affects Versions: 2.0.0
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
> Fix For: 2.0.0
>
> Attachments: HBASE-14861.patch, HBASE-14861v2.patch,
> HBASE-14861v3.patch
>
>
> In case of master-rs collocation HBASE_ZNODE_FILE is overwritten by
> regionserver process in HRegionServer#handleReportForDutyResponse() here is
> how it looks on master server:
> {code}
> [hbase@hnode2 hbase]$ cat hbase-hbase-master.znode
> /hbase/rs/hnode2,16000,1448022074888
> {code}
> it contains regionserver znode path instead of String value of master's
> ServerName. This affects ZNodeClearer#clear() in way that will not clear
> master znode in case we detect master crash. At end this will extend
> failover time until master znode expires configured in zookeeper by
> maxSessionTimeout parameter (40s in my case).
> I have notice this on mater branch but it can be case in other branches where
> we are collocating master and rs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14860) Improve BoundedByteBufferPool; make lockless
[ https://issues.apache.org/jira/browse/HBASE-14860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026163#comment-15026163 ] Hudson commented on HBASE-14860: FAILURE: Integrated in HBase-Trunk_matrix #498 (See [https://builds.apache.org/job/HBase-Trunk_matrix/498/]) HBASE-14860 Improve BoundedByteBufferPool; make lockless (stack: rev 5898b95329ecc042b0798123d656013426f48d14) * hbase-common/src/test/java/org/apache/hadoop/hbase/io/TestBoundedByteBufferPool.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/BoundedByteBufferPool.java > Improve BoundedByteBufferPool; make lockless > > > Key: HBASE-14860 > URL: https://issues.apache.org/jira/browse/HBASE-14860 > Project: HBase > Issue Type: Improvement >Reporter: Hiroshi Ikeda >Assignee: Hiroshi Ikeda >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14860-addendum.patch, HBASE-14860.patch > > > Make it unblocking. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14623) Implement dedicated WAL for system tables
[
https://issues.apache.org/jira/browse/HBASE-14623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026169#comment-15026169
]
Hadoop QA commented on HBASE-14623:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12774178/14623-v3.txt
against master branch at commit 1178c4b89bad28210fe8d2c207cce814d6fd9a21.
ATTACHMENT ID: 12774178
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
18693 checkstyle errors (more than the master's current 18685 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16658//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16658//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16658//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16658//console
This message is automatically generated.
> Implement dedicated WAL for system tables
> -
>
> Key: HBASE-14623
> URL: https://issues.apache.org/jira/browse/HBASE-14623
> Project: HBase
> Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0
>
> Attachments: 14623-v1.txt, 14623-v2.txt, 14623-v2.txt, 14623-v2.txt,
> 14623-v2.txt, 14623-v3.txt
>
>
> As Stephen suggested in parent JIRA, dedicating separate WAL for system
> tables (other than hbase:meta) should be done in new JIRA.
> This task is to fulfill the system WAL separation.
> Below is summary of discussion:
> For system table to have its own WAL, we would recover system table faster
> (fast log split, fast log replay). It would probably benefit
> AssignmentManager on system table region assignment. At this time, the new
> AssignmentManager is not planned to change WAL. So the existence of this JIRA
> is good for overall system, not specific to AssignmentManager.
> There are 3 strategies for implementing system table WAL:
> 1. one WAL for all non-meta system tables
> 2. one WAL for each non-meta system table
> 3. one WAL for each region of non-meta system table
> Currently most system tables are one region table (only ACL table may become
> big). Choices 2 and 3 basically are the same.
> From implementation point of view, choices 2 and 3 are cleaner than choice 1
> (as we have already had 1 WAL for META table and we can reuse the logic).
> With choice 2 or 3, assignment manager performance should not be impacted and
> it would be easier for assignment manager to assign system table region (eg.
> without waiting for user table log split to complete for assigning system
> table region).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14689) Addendum and unit test for HBASE-13471
[ https://issues.apache.org/jira/browse/HBASE-14689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-14689: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 0.98.17 Status: Resolved (was: Patch Available) Committed this again (after-revert.patch). Thanks Stephen. > Addendum and unit test for HBASE-13471 > -- > > Key: HBASE-14689 > URL: https://issues.apache.org/jira/browse/HBASE-14689 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.3, 0.98.17, 0.98.16, 1.0.3 > > Attachments: hbase-14689-after-revert.patch, > hbase-14689-after-revert.patch, hbase-14689_v1-branch-1.1.patch, > hbase-14689_v1-branch-1.1.patch, hbase-14689_v1.patch > > > One of our customers ran into HBASE-13471, which resulted in all the handlers > getting blocked and various other issues. While backporting the issue, I > noticed that there is one more case where we might go into infinite loop. In > case a row lock cannot be acquired (due to a previous leak for example which > we have seen in Phoenix before) this will cause similar infinite loop. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14703) not collect stats when call HTable.mutateRow
[
https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14703:
--
Status: Open (was: Patch Available)
> not collect stats when call HTable.mutateRow
> -
>
> Key: HBASE-14703
> URL: https://issues.apache.org/jira/browse/HBASE-14703
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
>Assignee: Heng Chen
> Fix For: 2.0.0
>
> Attachments: HBASE-14702_v5.2_addendum-addendum.patch,
> HBASE-14703-5.2-addendum.patch, HBASE-14703-async.patch,
> HBASE-14703-start.patch, HBASE-14703-v4.1.patch, HBASE-14703-v4.patch,
> HBASE-14703-v6_with-check-and-mutate.patch, HBASE-14703.patch,
> HBASE-14703_v1.patch, HBASE-14703_v2.patch, HBASE-14703_v3.patch,
> HBASE-14703_v5.1.patch, HBASE-14703_v5.2.patch, HBASE-14703_v5.patch,
> HBASE-14703_v6.patch
>
>
> In {{AsyncProcess.SingleServerRequestRunnable}}, it seems we update
> serverStatistics twice.
> The first one is that we wrapper {{RetryingCallable}} by
> {{StatsTrackingRpcRetryingCaller}}, and do serverStatistics update when we
> call {{callWithRetries}} and {{callWithoutRetries}}. Relates code like below:
> {code}
> @Override
> public T callWithRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> @Override
> public T callWithoutRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> {code}
> The secondary one is after we get response, in {{receiveMultiAction}}, we do
> update again.
> {code}
> // update the stats about the region, if its a user table. We don't want to
> slow down
> // updates to meta tables, especially from internal updates (master, etc).
> if (AsyncProcess.this.connection.getStatisticsTracker() != null) {
> result = ResultStatsUtil.updateStats(result,
> AsyncProcess.this.connection.getStatisticsTracker(), server, regionName);
> }
> {code}
> It seems that {{StatsTrackingRpcRetryingCaller}} is NOT necessary, remove it?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14703) not collect stats when call HTable.mutateRow
[
https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14703:
--
Attachment: HBASE-14703_v6-addendum.patch
> not collect stats when call HTable.mutateRow
> -
>
> Key: HBASE-14703
> URL: https://issues.apache.org/jira/browse/HBASE-14703
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
>Assignee: Heng Chen
> Fix For: 2.0.0
>
> Attachments: HBASE-14702_v5.2_addendum-addendum.patch,
> HBASE-14703-5.2-addendum.patch, HBASE-14703-async.patch,
> HBASE-14703-start.patch, HBASE-14703-v4.1.patch, HBASE-14703-v4.patch,
> HBASE-14703-v6_with-check-and-mutate.patch, HBASE-14703.patch,
> HBASE-14703_v1.patch, HBASE-14703_v2.patch, HBASE-14703_v3.patch,
> HBASE-14703_v5.1.patch, HBASE-14703_v5.2.patch, HBASE-14703_v5.patch,
> HBASE-14703_v6-addendum.patch, HBASE-14703_v6.patch
>
>
> In {{AsyncProcess.SingleServerRequestRunnable}}, it seems we update
> serverStatistics twice.
> The first one is that we wrapper {{RetryingCallable}} by
> {{StatsTrackingRpcRetryingCaller}}, and do serverStatistics update when we
> call {{callWithRetries}} and {{callWithoutRetries}}. Relates code like below:
> {code}
> @Override
> public T callWithRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> @Override
> public T callWithoutRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> {code}
> The secondary one is after we get response, in {{receiveMultiAction}}, we do
> update again.
> {code}
> // update the stats about the region, if its a user table. We don't want to
> slow down
> // updates to meta tables, especially from internal updates (master, etc).
> if (AsyncProcess.this.connection.getStatisticsTracker() != null) {
> result = ResultStatsUtil.updateStats(result,
> AsyncProcess.this.connection.getStatisticsTracker(), server, regionName);
> }
> {code}
> It seems that {{StatsTrackingRpcRetryingCaller}} is NOT necessary, remove it?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14623) Implement dedicated WAL for system tables
[ https://issues.apache.org/jira/browse/HBASE-14623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14623: --- Attachment: 14623-v4.txt Patch v4 addresses checkstyle warnings > Implement dedicated WAL for system tables > - > > Key: HBASE-14623 > URL: https://issues.apache.org/jira/browse/HBASE-14623 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 14623-v1.txt, 14623-v2.txt, 14623-v2.txt, 14623-v2.txt, > 14623-v2.txt, 14623-v3.txt, 14623-v4.txt > > > As Stephen suggested in parent JIRA, dedicating separate WAL for system > tables (other than hbase:meta) should be done in new JIRA. > This task is to fulfill the system WAL separation. > Below is summary of discussion: > For system table to have its own WAL, we would recover system table faster > (fast log split, fast log replay). It would probably benefit > AssignmentManager on system table region assignment. At this time, the new > AssignmentManager is not planned to change WAL. So the existence of this JIRA > is good for overall system, not specific to AssignmentManager. > There are 3 strategies for implementing system table WAL: > 1. one WAL for all non-meta system tables > 2. one WAL for each non-meta system table > 3. one WAL for each region of non-meta system table > Currently most system tables are one region table (only ACL table may become > big). Choices 2 and 3 basically are the same. > From implementation point of view, choices 2 and 3 are cleaner than choice 1 > (as we have already had 1 WAL for META table and we can reuse the logic). > With choice 2 or 3, assignment manager performance should not be impacted and > it would be easier for assignment manager to assign system table region (eg. > without waiting for user table log split to complete for assigning system > table region). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14703) not collect stats when call HTable.mutateRow
[
https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026145#comment-15026145
]
Heng Chen edited comment on HBASE-14703 at 11/25/15 4:52 AM:
-
Oh, i figure out why TestCheckAndMutate failed with patch_v6.
It is because of here which you fix in
HBASE-14703-v6_with-check-and-mutate.patch
{code}
- } catch(NoSuchColumnFamilyException e) {
+ } catch (RetriesExhaustedWithDetailsException e) {
+try {
+ throw e.getCause(0);
+} catch (NoSuchColumnFamilyException e1) {
+ // expected
+}
{code}
So as your logic, if processed is false and no exception in
ClientProtos.MultiResponse, there will no exception throw out (you remove it
in ResponseConverter), and there is no retry.
It is different with original logic (original logic will retry).
IMO we should NOT retry in checkAndXXX if processed is false and just return
false to users, but we need to confirm it. wdyt?
I will extract a new callable class to avoid useless repeated code. The
callable class will be useful in future when we unify other calls.
btw.
Takes advantage of the 'exists' flag in Result to track processed state is a
good idea. Thanks [~jesse_yates] for your nice patch.
upload v6_addendum patch based on v6
was (Author: chenheng):
Oh, i figure out why TestCheckAndMutate failed with patch_v6.
It is because of here which you fix in
HBASE-14703-v6_with-check-and-mutate.patch
{code}
- } catch(NoSuchColumnFamilyException e) {
+ } catch (RetriesExhaustedWithDetailsException e) {
+try {
+ throw e.getCause(0);
+} catch (NoSuchColumnFamilyException e1) {
+ // expected
+}
{code}
So as your logic, if processed is false and no exception in
ClientProtos.MultiResponse, there will no exception throw out (you remove it
in ResponseConverter), and there is no retry.
It is different with original logic (original logic will retry).
IMO we should NOT retry in checkAndXXX if processed is false and just return
false to users, but we need to confirm it. wdyt?
I will extract a new callable class to avoid useless repeated code. The
callable class will be useful in future when we unify other calls.
btw.
Takes advantage of the 'exists' flag in Result to track processed state is a
good idea. Thanks [~jesse_yates] for your nice patch.
> not collect stats when call HTable.mutateRow
> -
>
> Key: HBASE-14703
> URL: https://issues.apache.org/jira/browse/HBASE-14703
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
>Assignee: Heng Chen
> Fix For: 2.0.0
>
> Attachments: HBASE-14702_v5.2_addendum-addendum.patch,
> HBASE-14703-5.2-addendum.patch, HBASE-14703-async.patch,
> HBASE-14703-start.patch, HBASE-14703-v4.1.patch, HBASE-14703-v4.patch,
> HBASE-14703-v6_with-check-and-mutate.patch, HBASE-14703.patch,
> HBASE-14703_v1.patch, HBASE-14703_v2.patch, HBASE-14703_v3.patch,
> HBASE-14703_v5.1.patch, HBASE-14703_v5.2.patch, HBASE-14703_v5.patch,
> HBASE-14703_v6-addendum.patch, HBASE-14703_v6.patch
>
>
> In {{AsyncProcess.SingleServerRequestRunnable}}, it seems we update
> serverStatistics twice.
> The first one is that we wrapper {{RetryingCallable}} by
> {{StatsTrackingRpcRetryingCaller}}, and do serverStatistics update when we
> call {{callWithRetries}} and {{callWithoutRetries}}. Relates code like below:
> {code}
> @Override
> public T callWithRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> @Override
> public T callWithoutRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> {code}
> The secondary one is after we get response, in {{receiveMultiAction}}, we do
> update again.
> {code}
> // update the stats about the region, if its a user table. We don't want to
> slow down
> // updates to meta tables, especially from internal updates (master, etc).
> if (AsyncProcess.this.connection.getStatisticsTracker() != null) {
> result = ResultStatsUtil.updateStats(result,
> AsyncProcess.this.connection.getStatisticsTracker(), server, regionName);
> }
> {code}
> It seems that {{StatsTrackingRpcRetryingCaller}} is NOT necessary, remove it?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14703) not collect stats when call HTable.mutateRow
[
https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026202#comment-15026202
]
Jesse Yates commented on HBASE-14703:
-
Hey [~chenheng], can you attach that as a full patch? The way I find people
using addendums is generally for two things: (1) bug fixes immediately after
the fact, (2) helping to show changes. For instance, I used it previously to
show how you go follow up, but its a large change and it would have been hard
to keep track of how it fit onto the base patch, were it a single patch. For
smaller changes, especially that aren't expected to be committed, a new patch
and version is fine, and for really small changes - checkstyles, spelling, etc
- doing point patches is ok (so a 7.1).
The functional problem in particular here is that QA won't by able to handle
addendums, so making it patch available doesn't actually run the test :(
> not collect stats when call HTable.mutateRow
> -
>
> Key: HBASE-14703
> URL: https://issues.apache.org/jira/browse/HBASE-14703
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
>Assignee: Heng Chen
> Fix For: 2.0.0
>
> Attachments: HBASE-14702_v5.2_addendum-addendum.patch,
> HBASE-14703-5.2-addendum.patch, HBASE-14703-async.patch,
> HBASE-14703-start.patch, HBASE-14703-v4.1.patch, HBASE-14703-v4.patch,
> HBASE-14703-v6_with-check-and-mutate.patch, HBASE-14703.patch,
> HBASE-14703_v1.patch, HBASE-14703_v2.patch, HBASE-14703_v3.patch,
> HBASE-14703_v5.1.patch, HBASE-14703_v5.2.patch, HBASE-14703_v5.patch,
> HBASE-14703_v6-addendum.patch, HBASE-14703_v6.patch
>
>
> In {{AsyncProcess.SingleServerRequestRunnable}}, it seems we update
> serverStatistics twice.
> The first one is that we wrapper {{RetryingCallable}} by
> {{StatsTrackingRpcRetryingCaller}}, and do serverStatistics update when we
> call {{callWithRetries}} and {{callWithoutRetries}}. Relates code like below:
> {code}
> @Override
> public T callWithRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> @Override
> public T callWithoutRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> {code}
> The secondary one is after we get response, in {{receiveMultiAction}}, we do
> update again.
> {code}
> // update the stats about the region, if its a user table. We don't want to
> slow down
> // updates to meta tables, especially from internal updates (master, etc).
> if (AsyncProcess.this.connection.getStatisticsTracker() != null) {
> result = ResultStatsUtil.updateStats(result,
> AsyncProcess.this.connection.getStatisticsTracker(), server, regionName);
> }
> {code}
> It seems that {{StatsTrackingRpcRetryingCaller}} is NOT necessary, remove it?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14703) not collect stats when call HTable.mutateRow
[
https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14703:
--
Attachment: (was: HBASE-14703_v6-addendum.patch)
> not collect stats when call HTable.mutateRow
> -
>
> Key: HBASE-14703
> URL: https://issues.apache.org/jira/browse/HBASE-14703
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
>Assignee: Heng Chen
> Fix For: 2.0.0
>
> Attachments: HBASE-14702_v5.2_addendum-addendum.patch,
> HBASE-14703-5.2-addendum.patch, HBASE-14703-async.patch,
> HBASE-14703-start.patch, HBASE-14703-v4.1.patch, HBASE-14703-v4.patch,
> HBASE-14703-v6_with-check-and-mutate.patch, HBASE-14703.patch,
> HBASE-14703_v1.patch, HBASE-14703_v2.patch, HBASE-14703_v3.patch,
> HBASE-14703_v5.1.patch, HBASE-14703_v5.2.patch, HBASE-14703_v5.patch,
> HBASE-14703_v6-addendum.patch, HBASE-14703_v6.patch
>
>
> In {{AsyncProcess.SingleServerRequestRunnable}}, it seems we update
> serverStatistics twice.
> The first one is that we wrapper {{RetryingCallable}} by
> {{StatsTrackingRpcRetryingCaller}}, and do serverStatistics update when we
> call {{callWithRetries}} and {{callWithoutRetries}}. Relates code like below:
> {code}
> @Override
> public T callWithRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> @Override
> public T callWithoutRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> {code}
> The secondary one is after we get response, in {{receiveMultiAction}}, we do
> update again.
> {code}
> // update the stats about the region, if its a user table. We don't want to
> slow down
> // updates to meta tables, especially from internal updates (master, etc).
> if (AsyncProcess.this.connection.getStatisticsTracker() != null) {
> result = ResultStatsUtil.updateStats(result,
> AsyncProcess.this.connection.getStatisticsTracker(), server, regionName);
> }
> {code}
> It seems that {{StatsTrackingRpcRetryingCaller}} is NOT necessary, remove it?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14703) not collect stats when call HTable.mutateRow
[
https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026207#comment-15026207
]
Jesse Yates commented on HBASE-14703:
-
That patch just seems a bit much for right now. It makes the code change we are
putting in bigger, but doesn't really add any more clarity. How about coming
back to that when we do the changes for the rest of the methods? I tend to shy
away from adding heirarchy when its only being slightly used. WDYT?
> not collect stats when call HTable.mutateRow
> -
>
> Key: HBASE-14703
> URL: https://issues.apache.org/jira/browse/HBASE-14703
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
>Assignee: Heng Chen
> Fix For: 2.0.0
>
> Attachments: HBASE-14702_v5.2_addendum-addendum.patch,
> HBASE-14703-5.2-addendum.patch, HBASE-14703-async.patch,
> HBASE-14703-start.patch, HBASE-14703-v4.1.patch, HBASE-14703-v4.patch,
> HBASE-14703-v6_with-check-and-mutate.patch, HBASE-14703.patch,
> HBASE-14703_v1.patch, HBASE-14703_v2.patch, HBASE-14703_v3.patch,
> HBASE-14703_v5.1.patch, HBASE-14703_v5.2.patch, HBASE-14703_v5.patch,
> HBASE-14703_v6-addendum.patch, HBASE-14703_v6.patch
>
>
> In {{AsyncProcess.SingleServerRequestRunnable}}, it seems we update
> serverStatistics twice.
> The first one is that we wrapper {{RetryingCallable}} by
> {{StatsTrackingRpcRetryingCaller}}, and do serverStatistics update when we
> call {{callWithRetries}} and {{callWithoutRetries}}. Relates code like below:
> {code}
> @Override
> public T callWithRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> @Override
> public T callWithoutRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> {code}
> The secondary one is after we get response, in {{receiveMultiAction}}, we do
> update again.
> {code}
> // update the stats about the region, if its a user table. We don't want to
> slow down
> // updates to meta tables, especially from internal updates (master, etc).
> if (AsyncProcess.this.connection.getStatisticsTracker() != null) {
> result = ResultStatsUtil.updateStats(result,
> AsyncProcess.this.connection.getStatisticsTracker(), server, regionName);
> }
> {code}
> It seems that {{StatsTrackingRpcRetryingCaller}} is NOT necessary, remove it?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14703) not collect stats when call HTable.mutateRow
[
https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14703:
--
Attachment: HBASE-14703_v6-addendum.patch
> not collect stats when call HTable.mutateRow
> -
>
> Key: HBASE-14703
> URL: https://issues.apache.org/jira/browse/HBASE-14703
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
>Assignee: Heng Chen
> Fix For: 2.0.0
>
> Attachments: HBASE-14702_v5.2_addendum-addendum.patch,
> HBASE-14703-5.2-addendum.patch, HBASE-14703-async.patch,
> HBASE-14703-start.patch, HBASE-14703-v4.1.patch, HBASE-14703-v4.patch,
> HBASE-14703-v6_with-check-and-mutate.patch, HBASE-14703.patch,
> HBASE-14703_v1.patch, HBASE-14703_v2.patch, HBASE-14703_v3.patch,
> HBASE-14703_v5.1.patch, HBASE-14703_v5.2.patch, HBASE-14703_v5.patch,
> HBASE-14703_v6-addendum.patch, HBASE-14703_v6.patch
>
>
> In {{AsyncProcess.SingleServerRequestRunnable}}, it seems we update
> serverStatistics twice.
> The first one is that we wrapper {{RetryingCallable}} by
> {{StatsTrackingRpcRetryingCaller}}, and do serverStatistics update when we
> call {{callWithRetries}} and {{callWithoutRetries}}. Relates code like below:
> {code}
> @Override
> public T callWithRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> @Override
> public T callWithoutRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> {code}
> The secondary one is after we get response, in {{receiveMultiAction}}, we do
> update again.
> {code}
> // update the stats about the region, if its a user table. We don't want to
> slow down
> // updates to meta tables, especially from internal updates (master, etc).
> if (AsyncProcess.this.connection.getStatisticsTracker() != null) {
> result = ResultStatsUtil.updateStats(result,
> AsyncProcess.this.connection.getStatisticsTracker(), server, regionName);
> }
> {code}
> It seems that {{StatsTrackingRpcRetryingCaller}} is NOT necessary, remove it?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14703) not collect stats when call HTable.mutateRow
[
https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14703:
--
Attachment: HBASE-14703_v7.patch
> not collect stats when call HTable.mutateRow
> -
>
> Key: HBASE-14703
> URL: https://issues.apache.org/jira/browse/HBASE-14703
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
>Assignee: Heng Chen
> Fix For: 2.0.0
>
> Attachments: HBASE-14702_v5.2_addendum-addendum.patch,
> HBASE-14703-5.2-addendum.patch, HBASE-14703-async.patch,
> HBASE-14703-start.patch, HBASE-14703-v4.1.patch, HBASE-14703-v4.patch,
> HBASE-14703-v6_with-check-and-mutate.patch, HBASE-14703.patch,
> HBASE-14703_v1.patch, HBASE-14703_v2.patch, HBASE-14703_v3.patch,
> HBASE-14703_v5.1.patch, HBASE-14703_v5.2.patch, HBASE-14703_v5.patch,
> HBASE-14703_v6-addendum.patch, HBASE-14703_v6.patch, HBASE-14703_v7.patch
>
>
> In {{AsyncProcess.SingleServerRequestRunnable}}, it seems we update
> serverStatistics twice.
> The first one is that we wrapper {{RetryingCallable}} by
> {{StatsTrackingRpcRetryingCaller}}, and do serverStatistics update when we
> call {{callWithRetries}} and {{callWithoutRetries}}. Relates code like below:
> {code}
> @Override
> public T callWithRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> @Override
> public T callWithoutRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> {code}
> The secondary one is after we get response, in {{receiveMultiAction}}, we do
> update again.
> {code}
> // update the stats about the region, if its a user table. We don't want to
> slow down
> // updates to meta tables, especially from internal updates (master, etc).
> if (AsyncProcess.this.connection.getStatisticsTracker() != null) {
> result = ResultStatsUtil.updateStats(result,
> AsyncProcess.this.connection.getStatisticsTracker(), server, regionName);
> }
> {code}
> It seems that {{StatsTrackingRpcRetryingCaller}} is NOT necessary, remove it?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14703) not collect stats when call HTable.mutateRow
[
https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14703:
--
Status: Patch Available (was: Open)
> not collect stats when call HTable.mutateRow
> -
>
> Key: HBASE-14703
> URL: https://issues.apache.org/jira/browse/HBASE-14703
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
>Assignee: Heng Chen
> Fix For: 2.0.0
>
> Attachments: HBASE-14702_v5.2_addendum-addendum.patch,
> HBASE-14703-5.2-addendum.patch, HBASE-14703-async.patch,
> HBASE-14703-start.patch, HBASE-14703-v4.1.patch, HBASE-14703-v4.patch,
> HBASE-14703-v6_with-check-and-mutate.patch, HBASE-14703.patch,
> HBASE-14703_v1.patch, HBASE-14703_v2.patch, HBASE-14703_v3.patch,
> HBASE-14703_v5.1.patch, HBASE-14703_v5.2.patch, HBASE-14703_v5.patch,
> HBASE-14703_v6-addendum.patch, HBASE-14703_v6.patch, HBASE-14703_v7.patch
>
>
> In {{AsyncProcess.SingleServerRequestRunnable}}, it seems we update
> serverStatistics twice.
> The first one is that we wrapper {{RetryingCallable}} by
> {{StatsTrackingRpcRetryingCaller}}, and do serverStatistics update when we
> call {{callWithRetries}} and {{callWithoutRetries}}. Relates code like below:
> {code}
> @Override
> public T callWithRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> @Override
> public T callWithoutRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> {code}
> The secondary one is after we get response, in {{receiveMultiAction}}, we do
> update again.
> {code}
> // update the stats about the region, if its a user table. We don't want to
> slow down
> // updates to meta tables, especially from internal updates (master, etc).
> if (AsyncProcess.this.connection.getStatisticsTracker() != null) {
> result = ResultStatsUtil.updateStats(result,
> AsyncProcess.this.connection.getStatisticsTracker(), server, regionName);
> }
> {code}
> It seems that {{StatsTrackingRpcRetryingCaller}} is NOT necessary, remove it?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14875) Forward port HBASE-14207 'Region was hijacked and remained in transition when RS failed to open a region and later regionplan changed to new RS on retry'
[ https://issues.apache.org/jira/browse/HBASE-14875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026255#comment-15026255 ] Pankaj Kumar commented on HBASE-14875: -- Sure Stephen, will take of this. > Forward port HBASE-14207 'Region was hijacked and remained in transition when > RS failed to open a region and later regionplan changed to new RS on retry' > - > > Key: HBASE-14875 > URL: https://issues.apache.org/jira/browse/HBASE-14875 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu > Fix For: 1.2.0, 1.3.0, 1.1.3 > > Attachments: 14875-branch-1.txt > > > HBASE-14207 was integrated to 0.98 > However, for hbase 1.x where zookeeper is involved in region assignment, the > fix from HBASE-14207 applies. > HBASE-14207 has been closed. So opening a new JIRA for forward porting. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14826) Small improvement in KVHeap seek() API
[ https://issues.apache.org/jira/browse/HBASE-14826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-14826: --- Resolution: Fixed Fix Version/s: 2.0.0 Status: Resolved (was: Patch Available) Resolving this with fixing in trunk only. If needed and others agree we can backport it to other branches also using a backport JIRA. > Small improvement in KVHeap seek() API > -- > > Key: HBASE-14826 > URL: https://issues.apache.org/jira/browse/HBASE-14826 > Project: HBase > Issue Type: Improvement >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14826.patch, HBASE-14826_1.patch > > > Currently in seek/reseek() APIs we tend to do lot of priorityqueue related > operations. We initially add the current scanner to the heap, then poll and > again add the scanner back if the seekKey is greater than the topkey in that > scanner. Since the KVs are always going to be in increasing order and in > ideal scan flow every seek/reseek is followed by a next() call it should be > ok if we start with checking the current scanner and then do a poll to get > the next scanner. Just avoid the initial PQ.add(current) call. This could > save some comparisons. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14873) Problems around BoundedByteBufferPool providing direct buffers
[
https://issues.apache.org/jira/browse/HBASE-14873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026268#comment-15026268
]
ramkrishna.s.vasudevan commented on HBASE-14873:
Ya. I can see this happen. Are you going to work on this [~ikeda]?
> Problems around BoundedByteBufferPool providing direct buffers
> --
>
> Key: HBASE-14873
> URL: https://issues.apache.org/jira/browse/HBASE-14873
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>
> HBASE-13819 made BoundedByteBufferPool provide direct buffers.
> See RpcServer.java:
> {code}
> ...
> class Call implements RpcCallContext {
> protected synchronized void setResponse(...) {
> ...
> this.cellBlock = ipcUtil.buildCellBlock(..., reservoir);
> ...
> bc = new BufferChain(..., this.cellBlock);
> if (connection.useWrap) {
> bc = wrapWithSasl(bc);
> }
> ...
> private BufferChain wrapWithSasl(BufferChain bc) throws IOException {
> ...
> byte[] responseBytes = bc.getBytes();
> ...
> {code}
> {{cellBlock}} is expected to be a direct buffer retrieved from {{reservoir}}
> (but not always), and {{bc}} may be composed of both direct and non-direct
> buffers.
> And then, see BufferChain.java:
> {code}
> byte [] getBytes() {
> ...
> for (ByteBuffer bb: this.buffers) {
> System.arraycopy(bb.array(), ...);
> {code}
> A direct buffer doesn't give its array, and will throw
> UnsupportedOperationException.
> Another problem; {{cellBlock}} is allowed to be a non-direct buffer, and
> after use it will be put to {{reservoir}}, mixing direct and non-direct
> buffers in the pool.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14703) not collect stats when call HTable.mutateRow
[
https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026271#comment-15026271
]
Heng Chen commented on HBASE-14703:
---
{quote}
It makes the code change we are putting in bigger, but doesn't really add any
more clarity
{quote}
I don't think so. Indeed, if we just want to fix this issue as title, it is
easy and no need for this big change. But as you said, we should do it in
right way.
Currently logic is confused, there are two path to communicate with server and
statistics information store in PB result, it is not reasonable. I think we
should make it clear.
After HBASE-14693, there are more statistics we need to collect, i think we
should support an unify way to do it. wdyt?
> not collect stats when call HTable.mutateRow
> -
>
> Key: HBASE-14703
> URL: https://issues.apache.org/jira/browse/HBASE-14703
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
>Assignee: Heng Chen
> Fix For: 2.0.0
>
> Attachments: HBASE-14702_v5.2_addendum-addendum.patch,
> HBASE-14703-5.2-addendum.patch, HBASE-14703-async.patch,
> HBASE-14703-start.patch, HBASE-14703-v4.1.patch, HBASE-14703-v4.patch,
> HBASE-14703-v6_with-check-and-mutate.patch, HBASE-14703.patch,
> HBASE-14703_v1.patch, HBASE-14703_v2.patch, HBASE-14703_v3.patch,
> HBASE-14703_v5.1.patch, HBASE-14703_v5.2.patch, HBASE-14703_v5.patch,
> HBASE-14703_v6-addendum.patch, HBASE-14703_v6.patch, HBASE-14703_v7.patch
>
>
> In {{AsyncProcess.SingleServerRequestRunnable}}, it seems we update
> serverStatistics twice.
> The first one is that we wrapper {{RetryingCallable}} by
> {{StatsTrackingRpcRetryingCaller}}, and do serverStatistics update when we
> call {{callWithRetries}} and {{callWithoutRetries}}. Relates code like below:
> {code}
> @Override
> public T callWithRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> @Override
> public T callWithoutRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> {code}
> The secondary one is after we get response, in {{receiveMultiAction}}, we do
> update again.
> {code}
> // update the stats about the region, if its a user table. We don't want to
> slow down
> // updates to meta tables, especially from internal updates (master, etc).
> if (AsyncProcess.this.connection.getStatisticsTracker() != null) {
> result = ResultStatsUtil.updateStats(result,
> AsyncProcess.this.connection.getStatisticsTracker(), server, regionName);
> }
> {code}
> It seems that {{StatsTrackingRpcRetryingCaller}} is NOT necessary, remove it?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14869) Better request latency histograms
[ https://issues.apache.org/jira/browse/HBASE-14869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026291#comment-15026291 ] Lars Hofhansl commented on HBASE-14869: --- Could also not report ranges without a hit, since we do have the count, we can always deduce the total number of requests we've seen. > Better request latency histograms > - > > Key: HBASE-14869 > URL: https://issues.apache.org/jira/browse/HBASE-14869 > Project: HBase > Issue Type: Brainstorming >Reporter: Lars Hofhansl > Attachments: 14869-test-0.98.txt > > > I just discussed this with a colleague. > The get, put, etc, histograms that each region server keeps are somewhat > useless (depending on what you want to achieve of course), as they are > aggregated and calculated by each region server. > It would be better to record the number of requests in certainly latency > bands in addition to what we do now. > For example the number of gets that took 0-5ms, 6-10ms, 10-20ms, 20-50ms, > 50-100ms, 100-1000ms, > 1000ms, etc. (just as an example, should be > configurable). > That way we can do further calculations after the fact, and answer questions > like: How often did we miss our SLA? Percentage of requests that missed an > SLA, etc. > Comments? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14869) Better request latency histograms
[ https://issues.apache.org/jira/browse/HBASE-14869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-14869: -- Attachment: 14869-test-0.98.txt Had a few minutes to work on this. Simple patch. _Totally untested, might not even compile, just had to park it somewhere_. You get the idea, though, for the value passing through, find the right range, and increment that ranges value. Upon snapshot report all ranges. I went log3 up to 30s, then thoughts it's better to roughly double (60s, 120s, 300s, 600s, > 600s) Note, this is now reported _everywhere_ where use MutableHistogram. Since the ranges cover 1ms-10mins, we can use the same for gets and snapshots, so it should be OK. > Better request latency histograms > - > > Key: HBASE-14869 > URL: https://issues.apache.org/jira/browse/HBASE-14869 > Project: HBase > Issue Type: Brainstorming >Reporter: Lars Hofhansl > Attachments: 14869-test-0.98.txt > > > I just discussed this with a colleague. > The get, put, etc, histograms that each region server keeps are somewhat > useless (depending on what you want to achieve of course), as they are > aggregated and calculated by each region server. > It would be better to record the number of requests in certainly latency > bands in addition to what we do now. > For example the number of gets that took 0-5ms, 6-10ms, 10-20ms, 20-50ms, > 50-100ms, 100-1000ms, > 1000ms, etc. (just as an example, should be > configurable). > That way we can do further calculations after the fact, and answer questions > like: How often did we miss our SLA? Percentage of requests that missed an > SLA, etc. > Comments? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14779) Revamp IntegrationTestMTTR
[ https://issues.apache.org/jira/browse/HBASE-14779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026306#comment-15026306 ] Stephen Yuan Jiang commented on HBASE-14779: [~jmhsieh], did you run the test with your V2 patch? > Revamp IntegrationTestMTTR > -- > > Key: HBASE-14779 > URL: https://issues.apache.org/jira/browse/HBASE-14779 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Attachments: hbase-14779.patch, hbase-14779.v2.patch > > > I've recently been trying to revive IntegrationTestMTTR runs and found that > it tended to not complete in less 6 hours and wasn't written as many of the > other Integration Tests. > I'm going to revamp it a local it run of it can finish in < 30mins and to > make it more configurable for a run against a real cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14703) not collect stats when call HTable.mutateRow
[
https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026271#comment-15026271
]
Heng Chen edited comment on HBASE-14703 at 11/25/15 6:40 AM:
-
{quote}
It makes the code change we are putting in bigger, but doesn't really add any
more clarity
{quote}
I don't think so. Indeed, if we just want to fix this issue as title, it is
easy and no need for this big change. But as you said, we should do it in
right way.
Currently logic is confused, there are two path to communicate with server and
statistics information store in PB result, it is not reasonable. I think we
should make it clear.
After HBASE-14693, there are more statistics we need to collect, i think we
should support an unify way to do it. wdyt?
As for AbstractMultiCallable, if you think it has too much levels. Maybe we
can unify it with PayloadCarryingServerCallable
was (Author: chenheng):
{quote}
It makes the code change we are putting in bigger, but doesn't really add any
more clarity
{quote}
I don't think so. Indeed, if we just want to fix this issue as title, it is
easy and no need for this big change. But as you said, we should do it in
right way.
Currently logic is confused, there are two path to communicate with server and
statistics information store in PB result, it is not reasonable. I think we
should make it clear.
After HBASE-14693, there are more statistics we need to collect, i think we
should support an unify way to do it. wdyt?
> not collect stats when call HTable.mutateRow
> -
>
> Key: HBASE-14703
> URL: https://issues.apache.org/jira/browse/HBASE-14703
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
>Assignee: Heng Chen
> Fix For: 2.0.0
>
> Attachments: HBASE-14702_v5.2_addendum-addendum.patch,
> HBASE-14703-5.2-addendum.patch, HBASE-14703-async.patch,
> HBASE-14703-start.patch, HBASE-14703-v4.1.patch, HBASE-14703-v4.patch,
> HBASE-14703-v6_with-check-and-mutate.patch, HBASE-14703.patch,
> HBASE-14703_v1.patch, HBASE-14703_v2.patch, HBASE-14703_v3.patch,
> HBASE-14703_v5.1.patch, HBASE-14703_v5.2.patch, HBASE-14703_v5.patch,
> HBASE-14703_v6-addendum.patch, HBASE-14703_v6.patch, HBASE-14703_v7.patch
>
>
> In {{AsyncProcess.SingleServerRequestRunnable}}, it seems we update
> serverStatistics twice.
> The first one is that we wrapper {{RetryingCallable}} by
> {{StatsTrackingRpcRetryingCaller}}, and do serverStatistics update when we
> call {{callWithRetries}} and {{callWithoutRetries}}. Relates code like below:
> {code}
> @Override
> public T callWithRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> @Override
> public T callWithoutRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> {code}
> The secondary one is after we get response, in {{receiveMultiAction}}, we do
> update again.
> {code}
> // update the stats about the region, if its a user table. We don't want to
> slow down
> // updates to meta tables, especially from internal updates (master, etc).
> if (AsyncProcess.this.connection.getStatisticsTracker() != null) {
> result = ResultStatsUtil.updateStats(result,
> AsyncProcess.this.connection.getStatisticsTracker(), server, regionName);
> }
> {code}
> It seems that {{StatsTrackingRpcRetryingCaller}} is NOT necessary, remove it?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14703) not collect stats when call HTable.mutateRow
[
https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026271#comment-15026271
]
Heng Chen edited comment on HBASE-14703 at 11/25/15 6:50 AM:
-
{quote}
It makes the code change we are putting in bigger, but doesn't really add any
more clarity
{quote}
As for AbstractMultiCallable, if you think it has too much levels. Maybe we
can unify it with PayloadCarryingServerCallable
was (Author: chenheng):
{quote}
It makes the code change we are putting in bigger, but doesn't really add any
more clarity
{quote}
I don't think so. Indeed, if we just want to fix this issue as title, it is
easy and no need for this big change. But as you said, we should do it in
right way.
Currently logic is confused, there are two path to communicate with server and
statistics information store in PB result, it is not reasonable. I think we
should make it clear.
After HBASE-14693, there are more statistics we need to collect, i think we
should support an unify way to do it. wdyt?
As for AbstractMultiCallable, if you think it has too much levels. Maybe we
can unify it with PayloadCarryingServerCallable
> not collect stats when call HTable.mutateRow
> -
>
> Key: HBASE-14703
> URL: https://issues.apache.org/jira/browse/HBASE-14703
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
>Assignee: Heng Chen
> Fix For: 2.0.0
>
> Attachments: HBASE-14702_v5.2_addendum-addendum.patch,
> HBASE-14703-5.2-addendum.patch, HBASE-14703-async.patch,
> HBASE-14703-start.patch, HBASE-14703-v4.1.patch, HBASE-14703-v4.patch,
> HBASE-14703-v6_with-check-and-mutate.patch, HBASE-14703.patch,
> HBASE-14703_v1.patch, HBASE-14703_v2.patch, HBASE-14703_v3.patch,
> HBASE-14703_v5.1.patch, HBASE-14703_v5.2.patch, HBASE-14703_v5.patch,
> HBASE-14703_v6-addendum.patch, HBASE-14703_v6.patch, HBASE-14703_v7.patch
>
>
> In {{AsyncProcess.SingleServerRequestRunnable}}, it seems we update
> serverStatistics twice.
> The first one is that we wrapper {{RetryingCallable}} by
> {{StatsTrackingRpcRetryingCaller}}, and do serverStatistics update when we
> call {{callWithRetries}} and {{callWithoutRetries}}. Relates code like below:
> {code}
> @Override
> public T callWithRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> @Override
> public T callWithoutRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> {code}
> The secondary one is after we get response, in {{receiveMultiAction}}, we do
> update again.
> {code}
> // update the stats about the region, if its a user table. We don't want to
> slow down
> // updates to meta tables, especially from internal updates (master, etc).
> if (AsyncProcess.this.connection.getStatisticsTracker() != null) {
> result = ResultStatsUtil.updateStats(result,
> AsyncProcess.this.connection.getStatisticsTracker(), server, regionName);
> }
> {code}
> It seems that {{StatsTrackingRpcRetryingCaller}} is NOT necessary, remove it?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14207) Region was hijacked and remained in transition when RS failed to open a region and later regionplan changed to new RS on retry
[
https://issues.apache.org/jira/browse/HBASE-14207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025557#comment-15025557
]
Stephen Yuan Jiang commented on HBASE-14207:
I am surprised that this change is not in branch-1, which is still relevant.
> Region was hijacked and remained in transition when RS failed to open a
> region and later regionplan changed to new RS on retry
> --
>
> Key: HBASE-14207
> URL: https://issues.apache.org/jira/browse/HBASE-14207
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 0.98.6
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 0.98.15
>
> Attachments: HBASE-14207-0.98-V2.patch, HBASE-14207-0.98-V2.patch,
> HBASE-14207-0.98.patch
>
>
> On production environment, following events happened
> 1. Master is trying to assign a region to RS, but due to
> KeeperException$SessionExpiredException RS failed to open the region.
> In RS log, saw multiple WARN log related to
> KeeperException$SessionExpiredException
> > KeeperErrorCode = Session expired for
> /hbase/region-in-transition/08f1935d652e5dbdac09b423b8f9401b
> > Unable to get data of znode
> /hbase/region-in-transition/08f1935d652e5dbdac09b423b8f9401b
> 2. Master retried to assign the region to same RS, but RS again failed.
> 3. On second retry new plan formed and this time plan destination (RS) is
> different, so master send the request to new RS to open the region. But new
> RS failed to open the region as there was server mismatch in ZNODE than the
> expected current server name.
> Logs Snippet:
> {noformat}
> HM
> 2015-07-14 03:50:29,759 | INFO | master:T101PC03VM13:21300 | Processing
> 08f1935d652e5dbdac09b423b8f9401b in state: M_ZK_REGION_OFFLINE |
> org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:644)
> 2015-07-14 03:50:29,759 | INFO | master:T101PC03VM13:21300 | Transitioned
> {08f1935d652e5dbdac09b423b8f9401b state=OFFLINE, ts=1436817029679,
> server=null} to {08f1935d652e5dbdac09b423b8f9401b state=PENDING_OPEN,
> ts=1436817029759, server=T101PC03VM13,21302,1436816690692} |
> org.apache.hadoop.hbase.master.RegionStates.updateRegionState(RegionStates.java:327)
> 2015-07-14 03:50:29,760 | INFO | master:T101PC03VM13:21300 | Processed
> region 08f1935d652e5dbdac09b423b8f9401b in state M_ZK_REGION_OFFLINE, on
> server: T101PC03VM13,21302,1436816690692 |
> org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:768)
> 2015-07-14 03:50:29,800 | INFO |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Assigning
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> T101PC03VM13,21302,1436816690692 |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1983)
> 2015-07-14 03:50:29,801 | WARN |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Failed assignment of
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> T101PC03VM13,21302,1436816690692, trying to assign elsewhere instead; try=1
> of 10 |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2077)
> 2015-07-14 03:50:29,802 | INFO |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Trying to re-assign
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> the same failed server. |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2123)
> 2015-07-14 03:50:31,804 | INFO |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Assigning
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> T101PC03VM13,21302,1436816690692 |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1983)
> 2015-07-14 03:50:31,806 | WARN |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Failed assignment of
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> T101PC03VM13,21302,1436816690692, trying to assign elsewhere instead; try=2
> of 10 |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2077)
> 2015-07-14 03:50:31,807 | INFO |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Transitioned
> {08f1935d652e5dbdac09b423b8f9401b state=PENDING_OPEN, ts=1436817031804,
> server=T101PC03VM13,21302,1436816690692} to {08f1935d652e5dbdac09b423b8f9401b
> state=OFFLINE, ts=1436817031807, server=T101PC03VM13,21302,1436816690692} |
> org.apache.hadoop.hbase.master.RegionStates.updateRegionState(RegionStates.java:327)
> 2015-07-14 03:50:31,807 | INFO |
>
[jira] [Commented] (HBASE-14769) Remove unused functions and duplicate javadocs from HBaseAdmin
[ https://issues.apache.org/jira/browse/HBASE-14769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025588#comment-15025588 ] Sean Busbey commented on HBASE-14769: - HBTU should be returning Admin (but I agree we ought not change it to do so until 2.0+). I think a note that only the Admin interface is actually supported on the returned object would be good addition. > Remove unused functions and duplicate javadocs from HBaseAdmin > --- > > Key: HBASE-14769 > URL: https://issues.apache.org/jira/browse/HBASE-14769 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: HBASE-14769-master-v2.patch, > HBASE-14769-master-v3.patch, HBASE-14769-master-v4.patch, > HBASE-14769-master.patch > > > HBaseAdmin is marked private, so removing the functions not being used > anywhere. > Also, the javadocs of overridden functions are same as corresponding ones in > Admin.java. Since javadocs are automatically inherited from the interface > class, we can remove these redundant 100s of lines. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14821) CopyTable should allow overriding more config properties for peer cluster
[
https://issues.apache.org/jira/browse/HBASE-14821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025691#comment-15025691
]
Hudson commented on HBASE-14821:
SUCCESS: Integrated in HBase-1.3 #395 (See
[https://builds.apache.org/job/HBase-1.3/395/])
HBASE-14821 Allow configuration overrides in TableOutputFormat (garyh: rev
07b07300b86385e33f3ab83c2930886935ad882c)
* hbase-common/src/test/java/org/apache/hadoop/hbase/TestHBaseConfiguration.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableOutputFormat.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java
> CopyTable should allow overriding more config properties for peer cluster
> -
>
> Key: HBASE-14821
> URL: https://issues.apache.org/jira/browse/HBASE-14821
> Project: HBase
> Issue Type: Improvement
> Components: mapreduce
>Reporter: Gary Helmling
>Assignee: Gary Helmling
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14821.patch
>
>
> When using CopyTable across two separate clusters, you can specify the ZK
> quorum for the destination cluster, but not much else in configuration
> overrides. This can be a problem when the cluster configurations differ,
> such as when using security with different configurations for server
> principals.
> We should provide a general way to override configuration properties for the
> peer / destination cluster. One option would be to allow use of a prefix for
> command line properties ("peer.property."). Properties matching this prefix
> will be stripped and merged to the peer configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14207) Region was hijacked and remained in transition when RS failed to open a region and later regionplan changed to new RS on retry
[
https://issues.apache.org/jira/browse/HBASE-14207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025728#comment-15025728
]
Hudson commented on HBASE-14207:
SUCCESS: Integrated in HBase-1.3-IT #335 (See
[https://builds.apache.org/job/HBase-1.3-IT/335/])
HBASE-14875 Forward port HBASE-14207 'Region was hijacked and remained (tedyu:
rev 1917516ffd7a5d5c9f4402c820e7b1ce8811e716)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
> Region was hijacked and remained in transition when RS failed to open a
> region and later regionplan changed to new RS on retry
> --
>
> Key: HBASE-14207
> URL: https://issues.apache.org/jira/browse/HBASE-14207
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 0.98.6
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 0.98.15
>
> Attachments: HBASE-14207-0.98-V2.patch, HBASE-14207-0.98-V2.patch,
> HBASE-14207-0.98.patch
>
>
> On production environment, following events happened
> 1. Master is trying to assign a region to RS, but due to
> KeeperException$SessionExpiredException RS failed to open the region.
> In RS log, saw multiple WARN log related to
> KeeperException$SessionExpiredException
> > KeeperErrorCode = Session expired for
> /hbase/region-in-transition/08f1935d652e5dbdac09b423b8f9401b
> > Unable to get data of znode
> /hbase/region-in-transition/08f1935d652e5dbdac09b423b8f9401b
> 2. Master retried to assign the region to same RS, but RS again failed.
> 3. On second retry new plan formed and this time plan destination (RS) is
> different, so master send the request to new RS to open the region. But new
> RS failed to open the region as there was server mismatch in ZNODE than the
> expected current server name.
> Logs Snippet:
> {noformat}
> HM
> 2015-07-14 03:50:29,759 | INFO | master:T101PC03VM13:21300 | Processing
> 08f1935d652e5dbdac09b423b8f9401b in state: M_ZK_REGION_OFFLINE |
> org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:644)
> 2015-07-14 03:50:29,759 | INFO | master:T101PC03VM13:21300 | Transitioned
> {08f1935d652e5dbdac09b423b8f9401b state=OFFLINE, ts=1436817029679,
> server=null} to {08f1935d652e5dbdac09b423b8f9401b state=PENDING_OPEN,
> ts=1436817029759, server=T101PC03VM13,21302,1436816690692} |
> org.apache.hadoop.hbase.master.RegionStates.updateRegionState(RegionStates.java:327)
> 2015-07-14 03:50:29,760 | INFO | master:T101PC03VM13:21300 | Processed
> region 08f1935d652e5dbdac09b423b8f9401b in state M_ZK_REGION_OFFLINE, on
> server: T101PC03VM13,21302,1436816690692 |
> org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:768)
> 2015-07-14 03:50:29,800 | INFO |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Assigning
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> T101PC03VM13,21302,1436816690692 |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1983)
> 2015-07-14 03:50:29,801 | WARN |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Failed assignment of
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> T101PC03VM13,21302,1436816690692, trying to assign elsewhere instead; try=1
> of 10 |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2077)
> 2015-07-14 03:50:29,802 | INFO |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Trying to re-assign
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> the same failed server. |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2123)
> 2015-07-14 03:50:31,804 | INFO |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Assigning
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> T101PC03VM13,21302,1436816690692 |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1983)
> 2015-07-14 03:50:31,806 | WARN |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Failed assignment of
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> T101PC03VM13,21302,1436816690692, trying to assign elsewhere instead; try=2
> of 10 |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2077)
> 2015-07-14 03:50:31,807 | INFO |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Transitioned
> {08f1935d652e5dbdac09b423b8f9401b state=PENDING_OPEN, ts=1436817031804,
> server=T101PC03VM13,21302,1436816690692} to {08f1935d652e5dbdac09b423b8f9401b
>
[jira] [Commented] (HBASE-14875) Forward port HBASE-14207 'Region was hijacked and remained in transition when RS failed to open a region and later regionplan changed to new RS on retry'
[ https://issues.apache.org/jira/browse/HBASE-14875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025726#comment-15025726 ] Hudson commented on HBASE-14875: SUCCESS: Integrated in HBase-1.3-IT #335 (See [https://builds.apache.org/job/HBase-1.3-IT/335/]) HBASE-14875 Forward port HBASE-14207 'Region was hijacked and remained (tedyu: rev 1917516ffd7a5d5c9f4402c820e7b1ce8811e716) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java > Forward port HBASE-14207 'Region was hijacked and remained in transition when > RS failed to open a region and later regionplan changed to new RS on retry' > - > > Key: HBASE-14875 > URL: https://issues.apache.org/jira/browse/HBASE-14875 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu > Fix For: 1.2.0, 1.3.0, 1.1.3 > > Attachments: 14875-branch-1.txt > > > HBASE-14207 was integrated to 0.98 > However, for hbase 1.x where zookeeper is involved in region assignment, the > fix from HBASE-14207 applies. > HBASE-14207 has been closed. So opening a new JIRA for forward porting. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14821) CopyTable should allow overriding more config properties for peer cluster
[
https://issues.apache.org/jira/browse/HBASE-14821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025727#comment-15025727
]
Hudson commented on HBASE-14821:
SUCCESS: Integrated in HBase-1.3-IT #335 (See
[https://builds.apache.org/job/HBase-1.3-IT/335/])
HBASE-14821 Allow configuration overrides in TableOutputFormat (garyh: rev
07b07300b86385e33f3ab83c2930886935ad882c)
* hbase-common/src/test/java/org/apache/hadoop/hbase/TestHBaseConfiguration.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/HBaseConfiguration.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableOutputFormat.java
> CopyTable should allow overriding more config properties for peer cluster
> -
>
> Key: HBASE-14821
> URL: https://issues.apache.org/jira/browse/HBASE-14821
> Project: HBase
> Issue Type: Improvement
> Components: mapreduce
>Reporter: Gary Helmling
>Assignee: Gary Helmling
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14821.patch
>
>
> When using CopyTable across two separate clusters, you can specify the ZK
> quorum for the destination cluster, but not much else in configuration
> overrides. This can be a problem when the cluster configurations differ,
> such as when using security with different configurations for server
> principals.
> We should provide a general way to override configuration properties for the
> peer / destination cluster. One option would be to allow use of a prefix for
> command line properties ("peer.property."). Properties matching this prefix
> will be stripped and merged to the peer configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14865) Support passing multiple QOPs to SaslClient/Server via hbase.rpc.protection
[
https://issues.apache.org/jira/browse/HBASE-14865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1502#comment-1502
]
Hadoop QA commented on HBASE-14865:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12774135/HBASE-14865-master-v3.patch
against master branch at commit 4cc341b9c23183fe12225fb03d30ac975a87d07c.
ATTACHMENT ID: 12774135
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 27 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
18687 checkstyle errors (more than the master's current 18686 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16657//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16657//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16657//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16657//console
This message is automatically generated.
> Support passing multiple QOPs to SaslClient/Server via hbase.rpc.protection
> ---
>
> Key: HBASE-14865
> URL: https://issues.apache.org/jira/browse/HBASE-14865
> Project: HBase
> Issue Type: Improvement
> Components: security
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-14865-master-v2.patch,
> HBASE-14865-master-v3.patch, HBASE-14865-master.patch
>
>
> Currently, we can set the value of hbase.rpc.protection to one of
> authentication/integrity/privacy. It is the used to set
> {{javax.security.sasl.qop}} in SaslUtil.java.
> The problem is, if a cluster wants to switch from one qop to another, it'll
> have to take a downtime. Rolling upgrade will create a situation where some
> nodes have old value and some have new, which'll prevent any communication
> between them. There will be similar issue when clients will try to connect.
> {{javax.security.sasl.qop}} can take in a list of QOP in preferences order.
> So a transition from qop1 to qop2 can be easily done like this
> "qop1" --> "qop2,qop1" --> rolling restart --> "qop2" --> rolling restart
> Need to change hbase.rpc.protection to accept a list too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14769) Remove unused functions and duplicate javadocs from HBaseAdmin
[ https://issues.apache.org/jira/browse/HBASE-14769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025649#comment-15025649 ] Appy commented on HBASE-14769: -- Yeah, that's an issue. Users getting HBaseAdmin and using public functions being removed here will break. On other side, considering InterfaceAudience scope transitively will make things very difficult. Even though there is a function to return HBaseAdmin, users should be aware that it's a private and evolving class. It's hard to decide what's right in such case. But given the functions being removed are long dead (we moved to TableName two years ago?) and the class itself is private, it should be alright. I agree with Sean though, mentioning clearly in HBTU that it only supports Admin interface. In addition i'll deprecate that function and add a function to return Admin object. > Remove unused functions and duplicate javadocs from HBaseAdmin > --- > > Key: HBASE-14769 > URL: https://issues.apache.org/jira/browse/HBASE-14769 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0 > > Attachments: HBASE-14769-master-v2.patch, > HBASE-14769-master-v3.patch, HBASE-14769-master-v4.patch, > HBASE-14769-master.patch > > > HBaseAdmin is marked private, so removing the functions not being used > anywhere. > Also, the javadocs of overridden functions are same as corresponding ones in > Admin.java. Since javadocs are automatically inherited from the interface > class, we can remove these redundant 100s of lines. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14769) Remove unused functions and duplicate javadocs from HBaseAdmin
[ https://issues.apache.org/jira/browse/HBASE-14769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-14769: - Attachment: HBASE-14769-master-v5.patch v5 patch rebasing, fixing checkstyle, deprecate HBTU.getHBaseAdmin, etc > Remove unused functions and duplicate javadocs from HBaseAdmin > --- > > Key: HBASE-14769 > URL: https://issues.apache.org/jira/browse/HBASE-14769 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0 > > Attachments: HBASE-14769-master-v2.patch, > HBASE-14769-master-v3.patch, HBASE-14769-master-v4.patch, > HBASE-14769-master-v5.patch, HBASE-14769-master.patch > > > HBaseAdmin is marked private, so removing the functions not being used > anywhere. > Also, the javadocs of overridden functions are same as corresponding ones in > Admin.java. Since javadocs are automatically inherited from the interface > class, we can remove these redundant 100s of lines. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14875) Forward port HBASE-14207 'Region was hijacked and remained in transition when RS failed to open a region and later regionplan changed to new RS on retry'
[ https://issues.apache.org/jira/browse/HBASE-14875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025573#comment-15025573 ] Ted Yu commented on HBASE-14875: HBASE-14207 was closed - there is no way to reuse it for porting. > Forward port HBASE-14207 'Region was hijacked and remained in transition when > RS failed to open a region and later regionplan changed to new RS on retry' > - > > Key: HBASE-14875 > URL: https://issues.apache.org/jira/browse/HBASE-14875 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu > Fix For: 1.2.0, 1.3.0, 1.1.3 > > Attachments: 14875-branch-1.txt > > > HBASE-14207 was integrated to 0.98 > However, for hbase 1.x where zookeeper is involved in region assignment, the > fix from HBASE-14207 applies. > HBASE-14207 has been closed. So opening a new JIRA for forward porting. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14769) Remove unused functions and duplicate javadocs from HBaseAdmin
[ https://issues.apache.org/jira/browse/HBASE-14769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-14769: - Fix Version/s: 2.0.0 > Remove unused functions and duplicate javadocs from HBaseAdmin > --- > > Key: HBASE-14769 > URL: https://issues.apache.org/jira/browse/HBASE-14769 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Fix For: 2.0.0 > > Attachments: HBASE-14769-master-v2.patch, > HBASE-14769-master-v3.patch, HBASE-14769-master-v4.patch, > HBASE-14769-master.patch > > > HBaseAdmin is marked private, so removing the functions not being used > anywhere. > Also, the javadocs of overridden functions are same as corresponding ones in > Admin.java. Since javadocs are automatically inherited from the interface > class, we can remove these redundant 100s of lines. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14769) Remove unused functions and duplicate javadocs from HBaseAdmin
[
https://issues.apache.org/jira/browse/HBASE-14769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025602#comment-15025602
]
Appy commented on HBASE-14769:
--
{quote}
So you keep this because it was not deprecated though it should have been?
public Pair getAlterStatus(final byte[] tableName) throws
IOException
Want to add @deprecated as part of this patch or do you want to do that in new
issue?
{quote}
Yup, I couldn't remove it because it's also present in Admin.java. Other
"byte[] tableName" functions are not there in Admin.java (Audience.Public) so
they can be removed directly. Added @deprecated to declaration in Admin.java .
{quote}
What is parent doc in below?
// See parent doc for deprecation timeline.
{quote}
It referred to doc of the parent function. Nevermind it, had to change to get
rid of checkstyle error.
{quote}
Ok to remove this one?
3290public void snapshot(final String snapshotName,
3291final String tableName) throws IOException,
3292SnapshotCreationException, IllegalArgumentException \{
3293snapshot(snapshotName, TableName.valueOf(tableName)
3294SnapshotDescription.Type.FLUSH);
3295}
and a few of the other snapshot methods being removed?
{quote}
There are no functions corresponding to these in Admin.java. So there's nothing
keeping them from being removed from HBaseAdmin.
> Remove unused functions and duplicate javadocs from HBaseAdmin
> ---
>
> Key: HBASE-14769
> URL: https://issues.apache.org/jira/browse/HBASE-14769
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-14769-master-v2.patch,
> HBASE-14769-master-v3.patch, HBASE-14769-master-v4.patch,
> HBASE-14769-master.patch
>
>
> HBaseAdmin is marked private, so removing the functions not being used
> anywhere.
> Also, the javadocs of overridden functions are same as corresponding ones in
> Admin.java. Since javadocs are automatically inherited from the interface
> class, we can remove these redundant 100s of lines.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14875) Forward port HBASE-14207 'Region was hijacked and remained in transition when RS failed to open a region and later regionplan changed to new RS on retry'
[ https://issues.apache.org/jira/browse/HBASE-14875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025570#comment-15025570 ] Stephen Yuan Jiang commented on HBASE-14875: +1 LGTM (straightforward port). In the future, we really should just use one JIRA for all releases (FYI - [~pankaj2461]); this would be easier to track issues. > Forward port HBASE-14207 'Region was hijacked and remained in transition when > RS failed to open a region and later regionplan changed to new RS on retry' > - > > Key: HBASE-14875 > URL: https://issues.apache.org/jira/browse/HBASE-14875 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu > Fix For: 1.2.0, 1.3.0 > > Attachments: 14875-branch-1.txt > > > HBASE-14207 was integrated to 0.98 > However, for hbase 1.x where zookeeper is involved in region assignment, the > fix from HBASE-14207 applies. > HBASE-14207 has been closed. So opening a new JIRA for forward porting. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14861) HBASE_ZNODE_FILE on master server is overwritten by regionserver process in case of master-rs collocation
[
https://issues.apache.org/jira/browse/HBASE-14861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025625#comment-15025625
]
Hadoop QA commented on HBASE-14861:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12774133/HBASE-14861v3.patch
against master branch at commit 4a60c25c702a57d043ca3fd3a9996f9fe63f9343.
ATTACHMENT ID: 12774133
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16656//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16656//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16656//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16656//console
This message is automatically generated.
> HBASE_ZNODE_FILE on master server is overwritten by regionserver process in
> case of master-rs collocation
> --
>
> Key: HBASE-14861
> URL: https://issues.apache.org/jira/browse/HBASE-14861
> Project: HBase
> Issue Type: Bug
> Components: Operability
>Affects Versions: 2.0.0
>Reporter: Samir Ahmic
>Assignee: Samir Ahmic
> Fix For: 2.0.0
>
> Attachments: HBASE-14861.patch, HBASE-14861v2.patch,
> HBASE-14861v3.patch
>
>
> In case of master-rs collocation HBASE_ZNODE_FILE is overwritten by
> regionserver process in HRegionServer#handleReportForDutyResponse() here is
> how it looks on master server:
> {code}
> [hbase@hnode2 hbase]$ cat hbase-hbase-master.znode
> /hbase/rs/hnode2,16000,1448022074888
> {code}
> it contains regionserver znode path instead of String value of master's
> ServerName. This affects ZNodeClearer#clear() in way that will not clear
> master znode in case we detect master crash. At end this will extend
> failover time until master znode expires configured in zookeeper by
> maxSessionTimeout parameter (40s in my case).
> I have notice this on mater branch but it can be case in other branches where
> we are collocating master and rs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14799) Commons-collections object deserialization remote command execution vulnerability
[ https://issues.apache.org/jira/browse/HBASE-14799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025723#comment-15025723 ] Hudson commented on HBASE-14799: FAILURE: Integrated in HBase-0.94-security #586 (See [https://builds.apache.org/job/HBase-0.94-security/586/]) HBASE-14799 Commons-collections object deserialization remote command (apurtell: rev 509131608210166b664a46c827a9e99f9873d18a) * pom.xml > Commons-collections object deserialization remote command execution > vulnerability > -- > > Key: HBASE-14799 > URL: https://issues.apache.org/jira/browse/HBASE-14799 > Project: HBase > Issue Type: Bug >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Critical > Fix For: 2.0.0, 0.94.28, 1.2.0, 1.3.0, 1.1.3, 0.98.17, 1.0.4 > > Attachments: HBASE-14799-0.94.patch, HBASE-14799-0.94.patch, > HBASE-14799-0.94.patch, HBASE-14799-0.94.patch, HBASE-14799-0.94.patch, > HBASE-14799-0.98.patch, HBASE-14799-0.98.patch, HBASE-14799-0.98.patch, > HBASE-14799.patch, HBASE-14799.patch > > > Read: > http://foxglovesecurity.com/2015/11/06/what-do-weblogic-websphere-jboss-jenkins-opennms-and-your-application-have-in-common-this-vulnerability/ > TL;DR: If you have commons-collections on your classpath and accept and > process Java object serialization data, then you probably have an exploitable > remote command execution vulnerability. > 0.94 and earlier HBase releases are vulnerable because we might read in and > rehydrate serialized Java objects out of RPC packet data in > HbaseObjectWritable using ObjectInputStream#readObject (see > https://hbase.apache.org/0.94/xref/org/apache/hadoop/hbase/io/HbaseObjectWritable.html#714) > and we have commons-collections on the classpath on the server. > 0.98 also carries some limited exposure to this problem through inclusion of > backwards compatible deserialization code in > HbaseObjectWritableFor96Migration. This is used by the 0.94-to-0.98 migration > utility, and by the AccessController when reading permissions from the ACL > table serialized in legacy format by 0.94. Unprivileged users cannot run the > tool nor access the ACL table. > Unprivileged users can however attack a 0.94 installation. An attacker might > be able to use the method discussed on that blog post to capture valid HBase > RPC payloads for 0.94 and prior versions, rewrite them to embed an exploit, > and replay them to trigger a remote command execution with the privileges of > the account under which the HBase RegionServer daemon is running. > We need to make a patch release of 0.94 that changes HbaseObjectWritable to > disallow processing of random Java object serializations. This will be a > compatibility break that might affect old style coprocessors, which quite > possibly may rely on this catch-all in HbaseObjectWritable for custom object > (de)serialization. We can introduce a new configuration setting, > "hbase.allow.legacy.object.serialization", defaulting to false. > To be thorough, we can also use the new configuration setting > "hbase.allow.legacy.object.serialization" (defaulting to false) in 0.98 to > prevent the AccessController from falling back to the vulnerable legacy code. > This turns out to not affect the ability to migrate permissions because > TablePermission implements Writable, which is safe, not Serializable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14777) Fix Inter Cluster Replication Future ordering issues
[
https://issues.apache.org/jira/browse/HBASE-14777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025078#comment-15025078
]
Lars Hofhansl edited comment on HBASE-14777 at 11/24/15 7:45 PM:
-
Any comments [~ashu210890], [~stack], [~eclark]? I'd like to commit my
addendum, with the "real" fix.
was (Author: lhofhansl):
Any comments [~ashu210890], [~stack], [~eclark]? I'd like to commit my
addendum, with the real fix.
> Fix Inter Cluster Replication Future ordering issues
>
>
> Key: HBASE-14777
> URL: https://issues.apache.org/jira/browse/HBASE-14777
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Bhupendra Kumar Jain
>Assignee: Ashu Pachauri
>Priority: Critical
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14777-alternative.txt, HBASE-14777-1.patch,
> HBASE-14777-2.patch, HBASE-14777-3.patch, HBASE-14777-4.patch,
> HBASE-14777-5.patch, HBASE-14777-6.patch, HBASE-14777-addendum.patch,
> HBASE-14777.patch
>
>
> Replication fails with IndexOutOfBoundsException
> {code}
> regionserver.ReplicationSource$ReplicationSourceWorkerThread(939):
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint
> threw unknown exception:java.lang.IndexOutOfBoundsException: Index: 1, Size:
> 1
> at java.util.ArrayList.rangeCheck(Unknown Source)
> at java.util.ArrayList.remove(Unknown Source)
> at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint.replicate(HBaseInterClusterReplicationEndpoint.java:222)
> {code}
> Its happening due to incorrect removal of entries from the replication
> entries list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14777) Fix Inter Cluster Replication Future ordering issues
[
https://issues.apache.org/jira/browse/HBASE-14777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025222#comment-15025222
]
Ashu Pachauri commented on HBASE-14777:
---
Looks good to me. +!
> Fix Inter Cluster Replication Future ordering issues
>
>
> Key: HBASE-14777
> URL: https://issues.apache.org/jira/browse/HBASE-14777
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Bhupendra Kumar Jain
>Assignee: Ashu Pachauri
>Priority: Critical
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14777-alternative.txt, HBASE-14777-1.patch,
> HBASE-14777-2.patch, HBASE-14777-3.patch, HBASE-14777-4.patch,
> HBASE-14777-5.patch, HBASE-14777-6.patch, HBASE-14777-addendum.patch,
> HBASE-14777.patch
>
>
> Replication fails with IndexOutOfBoundsException
> {code}
> regionserver.ReplicationSource$ReplicationSourceWorkerThread(939):
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint
> threw unknown exception:java.lang.IndexOutOfBoundsException: Index: 1, Size:
> 1
> at java.util.ArrayList.rangeCheck(Unknown Source)
> at java.util.ArrayList.remove(Unknown Source)
> at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint.replicate(HBaseInterClusterReplicationEndpoint.java:222)
> {code}
> Its happening due to incorrect removal of entries from the replication
> entries list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14821) CopyTable should allow overriding more config properties for peer cluster
[
https://issues.apache.org/jira/browse/HBASE-14821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Helmling updated HBASE-14821:
--
Resolution: Fixed
Release Note: Configuration properties for
org.apache.hadoop.hbase.mapreduce.TableOutputFormat can now be overridden by
prefixing the property keys with "hbase.mapred.output.". When the
configuration is applied to TableOutputFormat, these entries will be rewritten
with the prefix removed -- ie.
"hbase.mapred.output.hbase.security.authentication" becomes
"hbase.security.authentication". This can be useful when directing output to a
peer cluster with different security configuration, for example.
Status: Resolved (was: Patch Available)
Thanks for reviews [~chenheng] and [~eclark]!
Committed to master, branch-1, and branch-1.2.
> CopyTable should allow overriding more config properties for peer cluster
> -
>
> Key: HBASE-14821
> URL: https://issues.apache.org/jira/browse/HBASE-14821
> Project: HBase
> Issue Type: Improvement
> Components: mapreduce
>Reporter: Gary Helmling
>Assignee: Gary Helmling
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14821.patch
>
>
> When using CopyTable across two separate clusters, you can specify the ZK
> quorum for the destination cluster, but not much else in configuration
> overrides. This can be a problem when the cluster configurations differ,
> such as when using security with different configurations for server
> principals.
> We should provide a general way to override configuration properties for the
> peer / destination cluster. One option would be to allow use of a prefix for
> command line properties ("peer.property."). Properties matching this prefix
> will be stripped and merged to the peer configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13408) HBase In-Memory Memstore Compaction
[ https://issues.apache.org/jira/browse/HBASE-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025386#comment-15025386 ] Eshcar Hillel commented on HBASE-13408: --- Latency spikes are from the evaluation we did with the 0.98 branch (posted July 14). In the more recent evaluation we did we were able to identify the cause of the spikes and avoid it (anyway thanks for suggesting to help :) ). Without the spikes it is easier to see the benefit of the new feature. > HBase In-Memory Memstore Compaction > --- > > Key: HBASE-13408 > URL: https://issues.apache.org/jira/browse/HBASE-13408 > Project: HBase > Issue Type: New Feature >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Fix For: 2.0.0 > > Attachments: HBASE-13408-trunk-v01.patch, > HBASE-13408-trunk-v02.patch, HBASE-13408-trunk-v03.patch, > HBASE-13408-trunk-v04.patch, HBASE-13408-trunk-v05.patch, > HBASE-13408-trunk-v06.patch, HBASE-13408-trunk-v07.patch, > HBASE-13408-trunk-v08.patch, HBASE-13408-trunk-v09.patch, > HBASE-13408-trunk-v10.patch, > HBaseIn-MemoryMemstoreCompactionDesignDocument-ver02.pdf, > HBaseIn-MemoryMemstoreCompactionDesignDocument-ver03.pdf, > HBaseIn-MemoryMemstoreCompactionDesignDocument-ver04.pdf, > HBaseIn-MemoryMemstoreCompactionDesignDocument.pdf, > InMemoryMemstoreCompactionEvaluationResults.pdf, > InMemoryMemstoreCompactionMasterEvaluationResults.pdf, > InMemoryMemstoreCompactionScansEvaluationResults.pdf, > StoreSegmentandStoreSegmentScannerClassHierarchies.pdf > > > A store unit holds a column family in a region, where the memstore is its > in-memory component. The memstore absorbs all updates to the store; from time > to time these updates are flushed to a file on disk, where they are > compacted. Unlike disk components, the memstore is not compacted until it is > written to the filesystem and optionally to block-cache. This may result in > underutilization of the memory due to duplicate entries per row, for example, > when hot data is continuously updated. > Generally, the faster the data is accumulated in memory, more flushes are > triggered, the data sinks to disk more frequently, slowing down retrieval of > data, even if very recent. > In high-churn workloads, compacting the memstore can help maintain the data > in memory, and thereby speed up data retrieval. > We suggest a new compacted memstore with the following principles: > 1.The data is kept in memory for as long as possible > 2.Memstore data is either compacted or in process of being compacted > 3.Allow a panic mode, which may interrupt an in-progress compaction and > force a flush of part of the memstore. > We suggest applying this optimization only to in-memory column families. > A design document is attached. > This feature was previously discussed in HBASE-5311. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14825) HBase Ref Guide corrections of typos/misspellings
[
https://issues.apache.org/jira/browse/HBASE-14825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misty Stanley-Jones updated HBASE-14825:
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Pushed to master. Thanks for your contribution, [~daniel_vimont]!
By the way, if the Hadoop bot tells you that the only thing wrong with your
patch is line lengths, and it's a docs patch, we sometimes let that ride.
Sometimes you just have to use a long line.
> HBase Ref Guide corrections of typos/misspellings
> -
>
> Key: HBASE-14825
> URL: https://issues.apache.org/jira/browse/HBASE-14825
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.0.0
>Reporter: Daniel Vimont
>Assignee: Daniel Vimont
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-14825-v2.patch, HBASE-14825-v3.patch,
> HBASE-14825-v4.patch, HBASE-14825-v5-test.patch, HBASE-14825-v6.patch,
> HBASE-14825.patch, HBASE-14825_misty_example.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Found the following list of typos/misspellings on the book.html page, and
> thought I would make corrections to the appropriate src/main/asciidoc files
> in which they are located. (This is just a good opportunity for me to become
> familiar with submission of fixes/patches as a prelude to beginning to make
> some coding contributions. This is also my first submission to the JIRA
> system, so corrections to content/conventions are welcome!)
> [Note: I see that [~misty] may be in the midst of a reformatting task --
> HBASE-14823 -- that might involve these same asciidoc files. Please advise
> if I should wait on this task to avoid a possibly cumbersome Git
> reconciliation mess. (?)]
> Here is the list of typos/misspellings. The format of each item is (a) the
> problem is presented in brackets on the first line, and (b) the phrase (as it
> currently appears in the text) is on the second line.
> ===
> ["you" should be "your", and "Kimballs'" should be "Kimball's" (move the
> apostrophe) in the following:]
> A useful read setting config on you hadoop cluster is Aaron Kimballs'
> Configuration Parameters: What can you just ignore?
> [Period needed after "a"]
> a.k.a pseudo-distributed
> ["empty" is misspelled]
> The default value in this configuration has been intentionally left emtpy in
> order to honor the old hbase.regionserver.global.memstore.upperLimit property
> if present.
> [All occurrences of "a HBase" should be changed to "an HBase" -- 15
> occurrences found]
> ["file path are" should be "file paths are"]
> By default, all of HBase's ZooKeeper file path are configured with a relative
> path, so they will all go under this directory unless changed.
> ["times" -- plural required]
> How many time to retry attempting to write a version file before just
> aborting.
> ["separated" is misspelled]
> Each attempt is seperated by the hbase.server.thread.wakefrequency
> milliseconds.
> [space needed after quotation mark (include"limit)]
> Because this limit represents the "automatic include"limit...
> [space needed ("ashbase:metadata" should be "as hbase:metadata")]
> This helps to keep compaction of lean tables (such ashbase:meta) fast.
> [Acronym "ide" should be capitalized for clarity: IDE]
> Setting this to true can be useful in contexts other than the other side of a
> maven generation; i.e. running in an ide.
> [RuntimeException missing an "e"]
> You'll want to set this boolean to true to avoid seeing the RuntimException
> complaint:
> [Space missing after "secure"]
> FS Permissions for the root directory in a secure(kerberos) setup.
> ["mutations" misspelled]
> ...will be created which will tail the logs and replicate the mutatations to
> region replicas for tables that have region replication > 1.
> ["it such that" should be "is such that"]
> If your working set it such that block cache does you no good...
> ["an" should be "and"]
> See the Deveraj Das an Nicolas Liochon blog post...
> [Tag "" should be ""]
> hbase.coprocessor.master.classes
> [Misspelling of "implementations"]
> Those consumers are coprocessors, phoenix, replication endpoint
> implemnetations or similar.
> [Misspelling of "cluster"]
> On upgrade, before running a rolling restart over the cluser...
> ["effect" should be "affect"]
> If NOT using BucketCache, this change does not effect you.
> [Need space after "throw"]
> This will throw`java.lang.NoSuchMethodError...
> ["habasee" should be "hbase"]
> You can pass commands to the HBase Shell in non-interactive mode (see
> hbasee.shell.noninteractive)...
> ["ie" should be "i.e."]
> Restrict the amount of resources (ie regions, tables) a namespace can consume.
> ["an"
[jira] [Commented] (HBASE-14689) Addendum and unit test for HBASE-13471
[ https://issues.apache.org/jira/browse/HBASE-14689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025464#comment-15025464 ] Stephen Yuan Jiang commented on HBASE-14689: +1 LGTM (removing the assert - a valid scenario to return null) > Addendum and unit test for HBASE-13471 > -- > > Key: HBASE-14689 > URL: https://issues.apache.org/jira/browse/HBASE-14689 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16 > > Attachments: hbase-14689-after-revert.patch, > hbase-14689-after-revert.patch, hbase-14689_v1-branch-1.1.patch, > hbase-14689_v1-branch-1.1.patch, hbase-14689_v1.patch > > > One of our customers ran into HBASE-13471, which resulted in all the handlers > getting blocked and various other issues. While backporting the issue, I > noticed that there is one more case where we might go into infinite loop. In > case a row lock cannot be acquired (due to a previous leak for example which > we have seen in Phoenix before) this will cause similar infinite loop. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14865) Support passing multiple QOPs to SaslClient/Server via hbase.rpc.protection
[
https://issues.apache.org/jira/browse/HBASE-14865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Appy updated HBASE-14865:
-
Attachment: HBASE-14865-master-v3.patch
fixing checkstyle errors.
> Support passing multiple QOPs to SaslClient/Server via hbase.rpc.protection
> ---
>
> Key: HBASE-14865
> URL: https://issues.apache.org/jira/browse/HBASE-14865
> Project: HBase
> Issue Type: Improvement
> Components: security
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-14865-master-v2.patch,
> HBASE-14865-master-v3.patch, HBASE-14865-master.patch
>
>
> Currently, we can set the value of hbase.rpc.protection to one of
> authentication/integrity/privacy. It is the used to set
> {{javax.security.sasl.qop}} in SaslUtil.java.
> The problem is, if a cluster wants to switch from one qop to another, it'll
> have to take a downtime. Rolling upgrade will create a situation where some
> nodes have old value and some have new, which'll prevent any communication
> between them. There will be similar issue when clients will try to connect.
> {{javax.security.sasl.qop}} can take in a list of QOP in preferences order.
> So a transition from qop1 to qop2 can be easily done like this
> "qop1" --> "qop2,qop1" --> rolling restart --> "qop2" --> rolling restart
> Need to change hbase.rpc.protection to accept a list too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14875) Forward port HBASE-14207 'Region was hijacked and remained in transition when RS failed to open a region and later regionplan changed to new RS on retry'
[
https://issues.apache.org/jira/browse/HBASE-14875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025238#comment-15025238
]
Hadoop QA commented on HBASE-14875:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12774096/14875-branch-1.txt
against branch-1 branch at commit 4a60c25c702a57d043ca3fd3a9996f9fe63f9343.
ATTACHMENT ID: 12774096
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16654//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16654//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16654//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16654//console
This message is automatically generated.
> Forward port HBASE-14207 'Region was hijacked and remained in transition when
> RS failed to open a region and later regionplan changed to new RS on retry'
> -
>
> Key: HBASE-14875
> URL: https://issues.apache.org/jira/browse/HBASE-14875
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
> Fix For: 1.2.0, 1.3.0
>
> Attachments: 14875-branch-1.txt
>
>
> HBASE-14207 was integrated to 0.98
> However, for hbase 1.x where zookeeper is involved in region assignment, the
> fix from HBASE-14207 applies.
> HBASE-14207 has been closed. So opening a new JIRA for forward porting.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13408) HBase In-Memory Memstore Compaction
[ https://issues.apache.org/jira/browse/HBASE-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025324#comment-15025324 ] Eshcar Hillel commented on HBASE-13408: --- [~stack] we'll get back with answers to your comments/suggestions/review later on, just wanted to say that the evaluation results for the patch is attached (posted October 26), please take a look. > HBase In-Memory Memstore Compaction > --- > > Key: HBASE-13408 > URL: https://issues.apache.org/jira/browse/HBASE-13408 > Project: HBase > Issue Type: New Feature >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Fix For: 2.0.0 > > Attachments: HBASE-13408-trunk-v01.patch, > HBASE-13408-trunk-v02.patch, HBASE-13408-trunk-v03.patch, > HBASE-13408-trunk-v04.patch, HBASE-13408-trunk-v05.patch, > HBASE-13408-trunk-v06.patch, HBASE-13408-trunk-v07.patch, > HBASE-13408-trunk-v08.patch, HBASE-13408-trunk-v09.patch, > HBASE-13408-trunk-v10.patch, > HBaseIn-MemoryMemstoreCompactionDesignDocument-ver02.pdf, > HBaseIn-MemoryMemstoreCompactionDesignDocument-ver03.pdf, > HBaseIn-MemoryMemstoreCompactionDesignDocument-ver04.pdf, > HBaseIn-MemoryMemstoreCompactionDesignDocument.pdf, > InMemoryMemstoreCompactionEvaluationResults.pdf, > InMemoryMemstoreCompactionMasterEvaluationResults.pdf, > InMemoryMemstoreCompactionScansEvaluationResults.pdf, > StoreSegmentandStoreSegmentScannerClassHierarchies.pdf > > > A store unit holds a column family in a region, where the memstore is its > in-memory component. The memstore absorbs all updates to the store; from time > to time these updates are flushed to a file on disk, where they are > compacted. Unlike disk components, the memstore is not compacted until it is > written to the filesystem and optionally to block-cache. This may result in > underutilization of the memory due to duplicate entries per row, for example, > when hot data is continuously updated. > Generally, the faster the data is accumulated in memory, more flushes are > triggered, the data sinks to disk more frequently, slowing down retrieval of > data, even if very recent. > In high-churn workloads, compacting the memstore can help maintain the data > in memory, and thereby speed up data retrieval. > We suggest a new compacted memstore with the following principles: > 1.The data is kept in memory for as long as possible > 2.Memstore data is either compacted or in process of being compacted > 3.Allow a panic mode, which may interrupt an in-progress compaction and > force a flush of part of the memstore. > We suggest applying this optimization only to in-memory column families. > A design document is attached. > This feature was previously discussed in HBASE-5311. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14866) VerifyReplication should use peer configuration in peer connection
[
https://issues.apache.org/jira/browse/HBASE-14866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025375#comment-15025375
]
Gary Helmling commented on HBASE-14866:
---
[~chenheng] no worries! The rest of your patch looked good. I'll merge it
with the approach I described.
> VerifyReplication should use peer configuration in peer connection
> --
>
> Key: HBASE-14866
> URL: https://issues.apache.org/jira/browse/HBASE-14866
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Reporter: Gary Helmling
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14866.patch, HBASE-14866_v1.patch
>
>
> VerifyReplication uses the replication peer's configuration to construct the
> ZooKeeper quorum address for the peer connection. However, other
> configuration properties in the peer's configuration are dropped. It should
> merge all configuration properties from the {{ReplicationPeerConfig}} when
> creating the peer connection and obtaining a credentials for the peer cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14872) Scan different timeRange per column family doesn't percolate down to the memstore
[ https://issues.apache.org/jira/browse/HBASE-14872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025475#comment-15025475 ] churro morales commented on HBASE-14872: [~anoop.hbase] and [~stack] noticed the ScanQueryMatcher also needs to have this change. Aside from that looks like ScannerModel is the last reference for scan.getTimeRange() that needs to be updated. Looks like this is used for the rest API, do we want to go ahead and make the changes there too? The ScanQueryMatcher is pretty easy to fix, but I don't know a lot about the ScannerModel and its use cases. > Scan different timeRange per column family doesn't percolate down to the > memstore > -- > > Key: HBASE-14872 > URL: https://issues.apache.org/jira/browse/HBASE-14872 > Project: HBase > Issue Type: Bug > Components: Client, regionserver, Scanners >Affects Versions: 2.0.0, 1.3.0 >Reporter: churro morales >Assignee: churro morales > Fix For: 2.0.0, 1.3.0, 0.98.17 > > Attachments: HBASE-14872.patch > > > HBASE-14355 The scan different time range for column family feature was not > applied to the memstore it was only done for the store files. This breaks > the contract. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14819) hbase-it tests failing with OOME; permgen
[
https://issues.apache.org/jira/browse/HBASE-14819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025189#comment-15025189
]
Hudson commented on HBASE-14819:
FAILURE: Integrated in HBase-Trunk_matrix #496 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/496/])
HBASE-14819 hbase-it tests failing with OOME; permgen -- DEBUGGING (stack: rev
4a60c25c702a57d043ca3fd3a9996f9fe63f9343)
* hbase-it/pom.xml
> hbase-it tests failing with OOME; permgen
> -
>
> Key: HBASE-14819
> URL: https://issues.apache.org/jira/browse/HBASE-14819
> Project: HBase
> Issue Type: Sub-task
> Components: test
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14819.addendum.patch, 14819v2.txt, Screen Shot
> 2015-11-16 at 11.37.41 PM.png, itag.txt
>
>
> Let me up the heap used when failsafe forks.
> Here is example OOME doing ITBLL:
> {code}
> 2015-11-16 03:09:15,073 INFO [Thread-694] actions.BatchRestartRsAction(69):
> Starting region server:asf905.gq1.ygridcore.net
> 2015-11-16 03:09:15,099 INFO [Thread-694] client.ConnectionUtils(104):
> regionserver/asf905.gq1.ygridcore.net/67.195.81.149:0 server-side HConnection
> retries=350
> 2015-11-16 03:09:15,099 INFO [Thread-694] ipc.SimpleRpcScheduler(128): Using
> deadline as user call queue, count=1
> 2015-11-16 03:09:15,101 INFO [Thread-694] ipc.RpcServer$Listener(607):
> regionserver/asf905.gq1.ygridcore.net/67.195.81.149:0: started 3 reader(s)
> listening on port=36114
> 2015-11-16 03:09:15,103 INFO [Thread-694] fs.HFileSystem(252): Added
> intercepting call to namenode#getBlockLocations so can do block reordering
> using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks
> 2015-11-16 03:09:15,104 INFO [Thread-694]
> zookeeper.RecoverableZooKeeper(120): Process identifier=regionserver:36114
> connecting to ZooKeeper ensemble=localhost:50139
> 2015-11-16 03:09:15,117 DEBUG [Thread-694-EventThread]
> zookeeper.ZooKeeperWatcher(554): regionserver:361140x0,
> quorum=localhost:50139, baseZNode=/hbase Received ZooKeeper Event, type=None,
> state=SyncConnected, path=null
> 2015-11-16 03:09:15,118 DEBUG [Thread-694] zookeeper.ZKUtil(492):
> regionserver:361140x0, quorum=localhost:50139, baseZNode=/hbase Set watcher
> on existing znode=/hbase/master
> 2015-11-16 03:09:15,119 DEBUG [Thread-694] zookeeper.ZKUtil(492):
> regionserver:361140x0, quorum=localhost:50139, baseZNode=/hbase Set watcher
> on existing znode=/hbase/running
> 2015-11-16 03:09:15,119 DEBUG [Thread-694-EventThread]
> zookeeper.ZooKeeperWatcher(638): regionserver:36114-0x1510e2c6f1d0029
> connected
> 2015-11-16 03:09:15,120 INFO [RpcServer.responder]
> ipc.RpcServer$Responder(926): RpcServer.responder: starting
> 2015-11-16 03:09:15,121 INFO [RpcServer.listener,port=36114]
> ipc.RpcServer$Listener(738): RpcServer.listener,port=36114: starting
> 2015-11-16 03:09:15,121 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=0 queue=0
> 2015-11-16 03:09:15,121 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=1 queue=0
> 2015-11-16 03:09:15,121 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=2 queue=0
> 2015-11-16 03:09:15,122 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=3 queue=0
> 2015-11-16 03:09:15,122 DEBUG [Thread-694] ipc.RpcExecutor(115): B.default
> Start Handler index=4 queue=0
> 2015-11-16 03:09:15,122 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=0 queue=0
> 2015-11-16 03:09:15,123 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=1 queue=1
> 2015-11-16 03:09:15,123 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=2 queue=0
> 2015-11-16 03:09:15,123 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=3 queue=1
> 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Priority
> Start Handler index=4 queue=0
> 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Replication
> Start Handler index=0 queue=0
> 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Replication
> Start Handler index=1 queue=0
> 2015-11-16 03:09:15,124 DEBUG [Thread-694] ipc.RpcExecutor(115): Replication
> Start Handler index=2 queue=0
> 2015-11-16 03:09:15,761 DEBUG [RS:0;asf905:36114]
> client.ConnectionManager$HConnectionImplementation(715): connection
> construction failed
> java.io.IOException: java.lang.OutOfMemoryError: PermGen space
> at
> org.apache.hadoop.hbase.client.RegistryFactory.getRegistry(RegistryFactory.java:43)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.setupRegistry(ConnectionManager.java:886)
> at
>
[jira] [Commented] (HBASE-13408) HBase In-Memory Memstore Compaction
[ https://issues.apache.org/jira/browse/HBASE-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025366#comment-15025366 ] stack commented on HBASE-13408: --- [~eshcar] Thanks. I missed that. There is at least one case where there is a clear benefit (I can help w/ the latency spike). Hopefully we can have it so this feature is near always beneficial and if not that, that it is an enabler of other features that will show general benefit (We are on that path I believe). > HBase In-Memory Memstore Compaction > --- > > Key: HBASE-13408 > URL: https://issues.apache.org/jira/browse/HBASE-13408 > Project: HBase > Issue Type: New Feature >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Fix For: 2.0.0 > > Attachments: HBASE-13408-trunk-v01.patch, > HBASE-13408-trunk-v02.patch, HBASE-13408-trunk-v03.patch, > HBASE-13408-trunk-v04.patch, HBASE-13408-trunk-v05.patch, > HBASE-13408-trunk-v06.patch, HBASE-13408-trunk-v07.patch, > HBASE-13408-trunk-v08.patch, HBASE-13408-trunk-v09.patch, > HBASE-13408-trunk-v10.patch, > HBaseIn-MemoryMemstoreCompactionDesignDocument-ver02.pdf, > HBaseIn-MemoryMemstoreCompactionDesignDocument-ver03.pdf, > HBaseIn-MemoryMemstoreCompactionDesignDocument-ver04.pdf, > HBaseIn-MemoryMemstoreCompactionDesignDocument.pdf, > InMemoryMemstoreCompactionEvaluationResults.pdf, > InMemoryMemstoreCompactionMasterEvaluationResults.pdf, > InMemoryMemstoreCompactionScansEvaluationResults.pdf, > StoreSegmentandStoreSegmentScannerClassHierarchies.pdf > > > A store unit holds a column family in a region, where the memstore is its > in-memory component. The memstore absorbs all updates to the store; from time > to time these updates are flushed to a file on disk, where they are > compacted. Unlike disk components, the memstore is not compacted until it is > written to the filesystem and optionally to block-cache. This may result in > underutilization of the memory due to duplicate entries per row, for example, > when hot data is continuously updated. > Generally, the faster the data is accumulated in memory, more flushes are > triggered, the data sinks to disk more frequently, slowing down retrieval of > data, even if very recent. > In high-churn workloads, compacting the memstore can help maintain the data > in memory, and thereby speed up data retrieval. > We suggest a new compacted memstore with the following principles: > 1.The data is kept in memory for as long as possible > 2.Memstore data is either compacted or in process of being compacted > 3.Allow a panic mode, which may interrupt an in-progress compaction and > force a flush of part of the memstore. > We suggest applying this optimization only to in-memory column families. > A design document is attached. > This feature was previously discussed in HBASE-5311. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-14871) Allow specifying the base branch for make_patch
[ https://issues.apache.org/jira/browse/HBASE-14871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark resolved HBASE-14871. --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.0.0 > Allow specifying the base branch for make_patch > --- > > Key: HBASE-14871 > URL: https://issues.apache.org/jira/browse/HBASE-14871 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0 > > Attachments: HBASE-14871-v2.patch, HBASE-14871-v3.patch, > HBASE-14871.patch > > > Not all branches will be based off of origin/*. Lets allow the user to > specify which branch to base the patch off of. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14875) Forward port HBASE-14207 'Region was hijacked and remained in transition when RS failed to open a region and later regionplan changed to new RS on retry'
[ https://issues.apache.org/jira/browse/HBASE-14875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026346#comment-15026346 ] Hudson commented on HBASE-14875: FAILURE: Integrated in HBase-1.1-JDK8 #1694 (See [https://builds.apache.org/job/HBase-1.1-JDK8/1694/]) HBASE-14875 Forward port HBASE-14207 'Region was hijacked and remained (tedyu: rev a75d2aac4f48b7c0b2d679416f081a05242d4378) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java > Forward port HBASE-14207 'Region was hijacked and remained in transition when > RS failed to open a region and later regionplan changed to new RS on retry' > - > > Key: HBASE-14875 > URL: https://issues.apache.org/jira/browse/HBASE-14875 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu > Fix For: 1.2.0, 1.3.0, 1.1.3 > > Attachments: 14875-branch-1.txt > > > HBASE-14207 was integrated to 0.98 > However, for hbase 1.x where zookeeper is involved in region assignment, the > fix from HBASE-14207 applies. > HBASE-14207 has been closed. So opening a new JIRA for forward porting. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14207) Region was hijacked and remained in transition when RS failed to open a region and later regionplan changed to new RS on retry
[
https://issues.apache.org/jira/browse/HBASE-14207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026348#comment-15026348
]
Hudson commented on HBASE-14207:
FAILURE: Integrated in HBase-1.1-JDK8 #1694 (See
[https://builds.apache.org/job/HBase-1.1-JDK8/1694/])
HBASE-14875 Forward port HBASE-14207 'Region was hijacked and remained (tedyu:
rev a75d2aac4f48b7c0b2d679416f081a05242d4378)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
> Region was hijacked and remained in transition when RS failed to open a
> region and later regionplan changed to new RS on retry
> --
>
> Key: HBASE-14207
> URL: https://issues.apache.org/jira/browse/HBASE-14207
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 0.98.6
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 0.98.15
>
> Attachments: HBASE-14207-0.98-V2.patch, HBASE-14207-0.98-V2.patch,
> HBASE-14207-0.98.patch
>
>
> On production environment, following events happened
> 1. Master is trying to assign a region to RS, but due to
> KeeperException$SessionExpiredException RS failed to open the region.
> In RS log, saw multiple WARN log related to
> KeeperException$SessionExpiredException
> > KeeperErrorCode = Session expired for
> /hbase/region-in-transition/08f1935d652e5dbdac09b423b8f9401b
> > Unable to get data of znode
> /hbase/region-in-transition/08f1935d652e5dbdac09b423b8f9401b
> 2. Master retried to assign the region to same RS, but RS again failed.
> 3. On second retry new plan formed and this time plan destination (RS) is
> different, so master send the request to new RS to open the region. But new
> RS failed to open the region as there was server mismatch in ZNODE than the
> expected current server name.
> Logs Snippet:
> {noformat}
> HM
> 2015-07-14 03:50:29,759 | INFO | master:T101PC03VM13:21300 | Processing
> 08f1935d652e5dbdac09b423b8f9401b in state: M_ZK_REGION_OFFLINE |
> org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:644)
> 2015-07-14 03:50:29,759 | INFO | master:T101PC03VM13:21300 | Transitioned
> {08f1935d652e5dbdac09b423b8f9401b state=OFFLINE, ts=1436817029679,
> server=null} to {08f1935d652e5dbdac09b423b8f9401b state=PENDING_OPEN,
> ts=1436817029759, server=T101PC03VM13,21302,1436816690692} |
> org.apache.hadoop.hbase.master.RegionStates.updateRegionState(RegionStates.java:327)
> 2015-07-14 03:50:29,760 | INFO | master:T101PC03VM13:21300 | Processed
> region 08f1935d652e5dbdac09b423b8f9401b in state M_ZK_REGION_OFFLINE, on
> server: T101PC03VM13,21302,1436816690692 |
> org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:768)
> 2015-07-14 03:50:29,800 | INFO |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Assigning
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> T101PC03VM13,21302,1436816690692 |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1983)
> 2015-07-14 03:50:29,801 | WARN |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Failed assignment of
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> T101PC03VM13,21302,1436816690692, trying to assign elsewhere instead; try=1
> of 10 |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2077)
> 2015-07-14 03:50:29,802 | INFO |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Trying to re-assign
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> the same failed server. |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2123)
> 2015-07-14 03:50:31,804 | INFO |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Assigning
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> T101PC03VM13,21302,1436816690692 |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1983)
> 2015-07-14 03:50:31,806 | WARN |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Failed assignment of
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> T101PC03VM13,21302,1436816690692, trying to assign elsewhere instead; try=2
> of 10 |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2077)
> 2015-07-14 03:50:31,807 | INFO |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Transitioned
> {08f1935d652e5dbdac09b423b8f9401b state=PENDING_OPEN, ts=1436817031804,
> server=T101PC03VM13,21302,1436816690692} to {08f1935d652e5dbdac09b423b8f9401b
>
[jira] [Commented] (HBASE-13471) Fix a possible infinite loop in doMiniBatchMutation
[
https://issues.apache.org/jira/browse/HBASE-13471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026349#comment-15026349
]
Hudson commented on HBASE-13471:
FAILURE: Integrated in HBase-1.1-JDK8 #1694 (See
[https://builds.apache.org/job/HBase-1.1-JDK8/1694/])
HBASE-14689 Addendum and unit test for HBASE-13471 (enis: rev
f28cd4aa23196cd5466af84285db813beee0ec33)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> Fix a possible infinite loop in doMiniBatchMutation
> ---
>
> Key: HBASE-13471
> URL: https://issues.apache.org/jira/browse/HBASE-13471
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.0.0, 2.0.0, 1.1.0, 0.98.13
>Reporter: Elliott Clark
>Assignee: Rajesh Nishtala
> Fix For: 2.0.0, 1.1.0, 0.98.13, 1.0.2
>
> Attachments: HBASE-13471-v1.patch, HBASE-13471.patch
>
>
> {code}
> Thread 4139
> (regionserver/hbase412.example.com/10.158.6.53:60020-splits-1429003183537):
> State: WAITING
> Blocked count: 131
> Waited count: 228
> Waiting on
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@50714dc3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
>
> java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
> org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1371)
> org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1325)
>
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.stepsBeforePONR(SplitTransactionImpl.java:352)
>
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.createDaughters(SplitTransactionImpl.java:252)
>
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.execute(SplitTransactionImpl.java:509)
>
> org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:84)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14689) Addendum and unit test for HBASE-13471
[ https://issues.apache.org/jira/browse/HBASE-14689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026347#comment-15026347 ] Hudson commented on HBASE-14689: FAILURE: Integrated in HBase-1.1-JDK8 #1694 (See [https://builds.apache.org/job/HBase-1.1-JDK8/1694/]) HBASE-14689 Addendum and unit test for HBASE-13471 (enis: rev f28cd4aa23196cd5466af84285db813beee0ec33) * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > Addendum and unit test for HBASE-13471 > -- > > Key: HBASE-14689 > URL: https://issues.apache.org/jira/browse/HBASE-14689 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16, 0.98.17 > > Attachments: hbase-14689-after-revert.patch, > hbase-14689-after-revert.patch, hbase-14689_v1-branch-1.1.patch, > hbase-14689_v1-branch-1.1.patch, hbase-14689_v1.patch > > > One of our customers ran into HBASE-13471, which resulted in all the handlers > getting blocked and various other issues. While backporting the issue, I > noticed that there is one more case where we might go into infinite loop. In > case a row lock cannot be acquired (due to a previous leak for example which > we have seen in Phoenix before) this will cause similar infinite loop. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14875) Forward port HBASE-14207 'Region was hijacked and remained in transition when RS failed to open a region and later regionplan changed to new RS on retry'
[ https://issues.apache.org/jira/browse/HBASE-14875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026375#comment-15026375 ] Hudson commented on HBASE-14875: FAILURE: Integrated in HBase-1.1-JDK7 #1606 (See [https://builds.apache.org/job/HBase-1.1-JDK7/1606/]) HBASE-14875 Forward port HBASE-14207 'Region was hijacked and remained (tedyu: rev a75d2aac4f48b7c0b2d679416f081a05242d4378) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java > Forward port HBASE-14207 'Region was hijacked and remained in transition when > RS failed to open a region and later regionplan changed to new RS on retry' > - > > Key: HBASE-14875 > URL: https://issues.apache.org/jira/browse/HBASE-14875 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu > Fix For: 1.2.0, 1.3.0, 1.1.3 > > Attachments: 14875-branch-1.txt > > > HBASE-14207 was integrated to 0.98 > However, for hbase 1.x where zookeeper is involved in region assignment, the > fix from HBASE-14207 applies. > HBASE-14207 has been closed. So opening a new JIRA for forward porting. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14207) Region was hijacked and remained in transition when RS failed to open a region and later regionplan changed to new RS on retry
[
https://issues.apache.org/jira/browse/HBASE-14207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026377#comment-15026377
]
Hudson commented on HBASE-14207:
FAILURE: Integrated in HBase-1.1-JDK7 #1606 (See
[https://builds.apache.org/job/HBase-1.1-JDK7/1606/])
HBASE-14875 Forward port HBASE-14207 'Region was hijacked and remained (tedyu:
rev a75d2aac4f48b7c0b2d679416f081a05242d4378)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
> Region was hijacked and remained in transition when RS failed to open a
> region and later regionplan changed to new RS on retry
> --
>
> Key: HBASE-14207
> URL: https://issues.apache.org/jira/browse/HBASE-14207
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 0.98.6
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 0.98.15
>
> Attachments: HBASE-14207-0.98-V2.patch, HBASE-14207-0.98-V2.patch,
> HBASE-14207-0.98.patch
>
>
> On production environment, following events happened
> 1. Master is trying to assign a region to RS, but due to
> KeeperException$SessionExpiredException RS failed to open the region.
> In RS log, saw multiple WARN log related to
> KeeperException$SessionExpiredException
> > KeeperErrorCode = Session expired for
> /hbase/region-in-transition/08f1935d652e5dbdac09b423b8f9401b
> > Unable to get data of znode
> /hbase/region-in-transition/08f1935d652e5dbdac09b423b8f9401b
> 2. Master retried to assign the region to same RS, but RS again failed.
> 3. On second retry new plan formed and this time plan destination (RS) is
> different, so master send the request to new RS to open the region. But new
> RS failed to open the region as there was server mismatch in ZNODE than the
> expected current server name.
> Logs Snippet:
> {noformat}
> HM
> 2015-07-14 03:50:29,759 | INFO | master:T101PC03VM13:21300 | Processing
> 08f1935d652e5dbdac09b423b8f9401b in state: M_ZK_REGION_OFFLINE |
> org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:644)
> 2015-07-14 03:50:29,759 | INFO | master:T101PC03VM13:21300 | Transitioned
> {08f1935d652e5dbdac09b423b8f9401b state=OFFLINE, ts=1436817029679,
> server=null} to {08f1935d652e5dbdac09b423b8f9401b state=PENDING_OPEN,
> ts=1436817029759, server=T101PC03VM13,21302,1436816690692} |
> org.apache.hadoop.hbase.master.RegionStates.updateRegionState(RegionStates.java:327)
> 2015-07-14 03:50:29,760 | INFO | master:T101PC03VM13:21300 | Processed
> region 08f1935d652e5dbdac09b423b8f9401b in state M_ZK_REGION_OFFLINE, on
> server: T101PC03VM13,21302,1436816690692 |
> org.apache.hadoop.hbase.master.AssignmentManager.processRegionsInTransition(AssignmentManager.java:768)
> 2015-07-14 03:50:29,800 | INFO |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Assigning
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> T101PC03VM13,21302,1436816690692 |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1983)
> 2015-07-14 03:50:29,801 | WARN |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Failed assignment of
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> T101PC03VM13,21302,1436816690692, trying to assign elsewhere instead; try=1
> of 10 |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2077)
> 2015-07-14 03:50:29,802 | INFO |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Trying to re-assign
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> the same failed server. |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2123)
> 2015-07-14 03:50:31,804 | INFO |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Assigning
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> T101PC03VM13,21302,1436816690692 |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1983)
> 2015-07-14 03:50:31,806 | WARN |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Failed assignment of
> INTER_CONCURRENCY_SETTING,,1436596137981.08f1935d652e5dbdac09b423b8f9401b. to
> T101PC03VM13,21302,1436816690692, trying to assign elsewhere instead; try=2
> of 10 |
> org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:2077)
> 2015-07-14 03:50:31,807 | INFO |
> MASTER_SERVER_OPERATIONS-T101PC03VM13:21300-3 | Transitioned
> {08f1935d652e5dbdac09b423b8f9401b state=PENDING_OPEN, ts=1436817031804,
> server=T101PC03VM13,21302,1436816690692} to {08f1935d652e5dbdac09b423b8f9401b
>
[jira] [Commented] (HBASE-13471) Fix a possible infinite loop in doMiniBatchMutation
[
https://issues.apache.org/jira/browse/HBASE-13471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026327#comment-15026327
]
Hudson commented on HBASE-13471:
SUCCESS: Integrated in HBase-1.3-IT #336 (See
[https://builds.apache.org/job/HBase-1.3-IT/336/])
HBASE-14689 Addendum and unit test for HBASE-13471 (enis: rev
e80face0e95f9f194d6b6d48c0a9169d9016c809)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> Fix a possible infinite loop in doMiniBatchMutation
> ---
>
> Key: HBASE-13471
> URL: https://issues.apache.org/jira/browse/HBASE-13471
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.0.0, 2.0.0, 1.1.0, 0.98.13
>Reporter: Elliott Clark
>Assignee: Rajesh Nishtala
> Fix For: 2.0.0, 1.1.0, 0.98.13, 1.0.2
>
> Attachments: HBASE-13471-v1.patch, HBASE-13471.patch
>
>
> {code}
> Thread 4139
> (regionserver/hbase412.example.com/10.158.6.53:60020-splits-1429003183537):
> State: WAITING
> Blocked count: 131
> Waited count: 228
> Waiting on
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@50714dc3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
>
> java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
> org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1371)
> org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1325)
>
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.stepsBeforePONR(SplitTransactionImpl.java:352)
>
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.createDaughters(SplitTransactionImpl.java:252)
>
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.execute(SplitTransactionImpl.java:509)
>
> org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:84)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14689) Addendum and unit test for HBASE-13471
[ https://issues.apache.org/jira/browse/HBASE-14689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026326#comment-15026326 ] Hudson commented on HBASE-14689: SUCCESS: Integrated in HBase-1.3-IT #336 (See [https://builds.apache.org/job/HBase-1.3-IT/336/]) HBASE-14689 Addendum and unit test for HBASE-13471 (enis: rev e80face0e95f9f194d6b6d48c0a9169d9016c809) * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > Addendum and unit test for HBASE-13471 > -- > > Key: HBASE-14689 > URL: https://issues.apache.org/jira/browse/HBASE-14689 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16, 0.98.17 > > Attachments: hbase-14689-after-revert.patch, > hbase-14689-after-revert.patch, hbase-14689_v1-branch-1.1.patch, > hbase-14689_v1-branch-1.1.patch, hbase-14689_v1.patch > > > One of our customers ran into HBASE-13471, which resulted in all the handlers > getting blocked and various other issues. While backporting the issue, I > noticed that there is one more case where we might go into infinite loop. In > case a row lock cannot be acquired (due to a previous leak for example which > we have seen in Phoenix before) this will cause similar infinite loop. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14734) TestGenerateDelegationToken fails with BindAddress clash
[ https://issues.apache.org/jira/browse/HBASE-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026379#comment-15026379 ] stack commented on HBASE-14734: --- Failed again here: https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/lastCompletedBuild/jdk=latest1.8,label=Hadoop/testReport/org.apache.hadoop.hbase.security.token/TestGenerateDelegationToken/org_apache_hadoop_hbase_security_token_TestGenerateDelegationToken/ > TestGenerateDelegationToken fails with BindAddress clash > > > Key: HBASE-14734 > URL: https://issues.apache.org/jira/browse/HBASE-14734 > Project: HBase > Issue Type: Sub-task > Components: flakey, test >Reporter: stack > > From > https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/330/jdk=latest1.7,label=Hadoop/testReport/junit/org.apache.hadoop.hbase.security.token/TestGenerateDelegationToken/org_apache_hadoop_hbase_security_token_TestGenerateDelegationToken/ > Error Message > Address already in use > Stacktrace > java.net.BindException: Address already in use > at sun.nio.ch.Net.bind0(Native Method) > at sun.nio.ch.Net.bind(Net.java:444) > at sun.nio.ch.Net.bind(Net.java:436) > at > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214) > at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) > at > org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:198) > at > org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:51) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor.registerHandles(AbstractPollingIoAcceptor.java:547) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor.access$400(AbstractPollingIoAcceptor.java:68) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:422) > at > org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Can this utility be made to not fail if address taken? Try another? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14689) Addendum and unit test for HBASE-13471
[ https://issues.apache.org/jira/browse/HBASE-14689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026376#comment-15026376 ] Hudson commented on HBASE-14689: FAILURE: Integrated in HBase-1.1-JDK7 #1606 (See [https://builds.apache.org/job/HBase-1.1-JDK7/1606/]) HBASE-14689 Addendum and unit test for HBASE-13471 (enis: rev f28cd4aa23196cd5466af84285db813beee0ec33) * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java > Addendum and unit test for HBASE-13471 > -- > > Key: HBASE-14689 > URL: https://issues.apache.org/jira/browse/HBASE-14689 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16, 0.98.17 > > Attachments: hbase-14689-after-revert.patch, > hbase-14689-after-revert.patch, hbase-14689_v1-branch-1.1.patch, > hbase-14689_v1-branch-1.1.patch, hbase-14689_v1.patch > > > One of our customers ran into HBASE-13471, which resulted in all the handlers > getting blocked and various other issues. While backporting the issue, I > noticed that there is one more case where we might go into infinite loop. In > case a row lock cannot be acquired (due to a previous leak for example which > we have seen in Phoenix before) this will cause similar infinite loop. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13471) Fix a possible infinite loop in doMiniBatchMutation
[
https://issues.apache.org/jira/browse/HBASE-13471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026378#comment-15026378
]
Hudson commented on HBASE-13471:
FAILURE: Integrated in HBase-1.1-JDK7 #1606 (See
[https://builds.apache.org/job/HBase-1.1-JDK7/1606/])
HBASE-14689 Addendum and unit test for HBASE-13471 (enis: rev
f28cd4aa23196cd5466af84285db813beee0ec33)
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
> Fix a possible infinite loop in doMiniBatchMutation
> ---
>
> Key: HBASE-13471
> URL: https://issues.apache.org/jira/browse/HBASE-13471
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.0.0, 2.0.0, 1.1.0, 0.98.13
>Reporter: Elliott Clark
>Assignee: Rajesh Nishtala
> Fix For: 2.0.0, 1.1.0, 0.98.13, 1.0.2
>
> Attachments: HBASE-13471-v1.patch, HBASE-13471.patch
>
>
> {code}
> Thread 4139
> (regionserver/hbase412.example.com/10.158.6.53:60020-splits-1429003183537):
> State: WAITING
> Blocked count: 131
> Waited count: 228
> Waiting on
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@50714dc3
> Stack:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
>
> java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
> org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1371)
> org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1325)
>
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.stepsBeforePONR(SplitTransactionImpl.java:352)
>
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.createDaughters(SplitTransactionImpl.java:252)
>
> org.apache.hadoop.hbase.regionserver.SplitTransactionImpl.execute(SplitTransactionImpl.java:509)
>
> org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:84)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13408) HBase In-Memory Memstore Compaction
[ https://issues.apache.org/jira/browse/HBASE-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15024910#comment-15024910 ] stack commented on HBASE-13408: --- Thanks for the design doc update. What do you lot think of the new 'principals' (am asking the authors). We go from "The data is kept in memory for as long as possible" to instead, "...[u]se the inÂmemory space effectively, by periodically compacting the memstore content." We also talk of 'compaction'. What do we mean by compaction? The removal of data that has been overwritten? Or making the data take up smaller space in RAM? The latter is a fine objective but what are you thinking? At first there will be no compaction. We'll just introduce the segments and pipeline. Later we'll want to add in 'compaction'. We'll expend CPU to change skip list to a more compact format. What should it be. We posit hfile or the blocks that will go into hfiles. Does that makes sense as an in memory data structure? If it does, good. If not, what should the in memory compacted format be? Have you done any exploration here? Do we have a sense of how much advantage there is to be had 'compacting' segments in the pipeline? How do we ensure this feature is of benefit 90% of the time and not for some exotic use case where most of the data is being overwritten and the column families are 'in memory'. Even then, do we have measure to see the improvement to be had? Let me look at the patch (smile). In fact, the above questions come of my trying to look at the patch. Thanks. > HBase In-Memory Memstore Compaction > --- > > Key: HBASE-13408 > URL: https://issues.apache.org/jira/browse/HBASE-13408 > Project: HBase > Issue Type: New Feature >Reporter: Eshcar Hillel >Assignee: Eshcar Hillel > Fix For: 2.0.0 > > Attachments: HBASE-13408-trunk-v01.patch, > HBASE-13408-trunk-v02.patch, HBASE-13408-trunk-v03.patch, > HBASE-13408-trunk-v04.patch, HBASE-13408-trunk-v05.patch, > HBASE-13408-trunk-v06.patch, HBASE-13408-trunk-v07.patch, > HBASE-13408-trunk-v08.patch, HBASE-13408-trunk-v09.patch, > HBASE-13408-trunk-v10.patch, > HBaseIn-MemoryMemstoreCompactionDesignDocument-ver02.pdf, > HBaseIn-MemoryMemstoreCompactionDesignDocument-ver03.pdf, > HBaseIn-MemoryMemstoreCompactionDesignDocument-ver04.pdf, > HBaseIn-MemoryMemstoreCompactionDesignDocument.pdf, > InMemoryMemstoreCompactionEvaluationResults.pdf, > InMemoryMemstoreCompactionMasterEvaluationResults.pdf, > InMemoryMemstoreCompactionScansEvaluationResults.pdf, > StoreSegmentandStoreSegmentScannerClassHierarchies.pdf > > > A store unit holds a column family in a region, where the memstore is its > in-memory component. The memstore absorbs all updates to the store; from time > to time these updates are flushed to a file on disk, where they are > compacted. Unlike disk components, the memstore is not compacted until it is > written to the filesystem and optionally to block-cache. This may result in > underutilization of the memory due to duplicate entries per row, for example, > when hot data is continuously updated. > Generally, the faster the data is accumulated in memory, more flushes are > triggered, the data sinks to disk more frequently, slowing down retrieval of > data, even if very recent. > In high-churn workloads, compacting the memstore can help maintain the data > in memory, and thereby speed up data retrieval. > We suggest a new compacted memstore with the following principles: > 1.The data is kept in memory for as long as possible > 2.Memstore data is either compacted or in process of being compacted > 3.Allow a panic mode, which may interrupt an in-progress compaction and > force a flush of part of the memstore. > We suggest applying this optimization only to in-memory column families. > A design document is attached. > This feature was previously discussed in HBASE-5311. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14871) Allow specifying the base branch for make_patch
[ https://issues.apache.org/jira/browse/HBASE-14871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-14871: -- Attachment: HBASE-14871-v3.patch Patch with some more explanation. > Allow specifying the base branch for make_patch > --- > > Key: HBASE-14871 > URL: https://issues.apache.org/jira/browse/HBASE-14871 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-14871-v2.patch, HBASE-14871-v3.patch, > HBASE-14871.patch > > > Not all branches will be based off of origin/*. Lets allow the user to > specify which branch to base the patch off of. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14871) Allow specifying the base branch for make_patch
[ https://issues.apache.org/jira/browse/HBASE-14871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15024946#comment-15024946 ] stack commented on HBASE-14871: --- +1 Change dthat to on commit. > Allow specifying the base branch for make_patch > --- > > Key: HBASE-14871 > URL: https://issues.apache.org/jira/browse/HBASE-14871 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Clark >Assignee: Elliott Clark > Attachments: HBASE-14871-v2.patch, HBASE-14871-v3.patch, > HBASE-14871.patch > > > Not all branches will be based off of origin/*. Lets allow the user to > specify which branch to base the patch off of. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14703) not collect stats when call HTable.mutateRow
[
https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesse Yates updated HBASE-14703:
Attachment: HBASE-14703-v6_with-check-and-mutate.patch
Attaching a patch on top of v6 to support checkAndMutate with the new style.
Takes advantage of the 'exists' flag in Result to track processed state.
Haven't tested against the whole test suite, but TestCheckAndMutate passes for
me locally.
WDYT [~chenheng]?
> not collect stats when call HTable.mutateRow
> -
>
> Key: HBASE-14703
> URL: https://issues.apache.org/jira/browse/HBASE-14703
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
>Assignee: Heng Chen
> Fix For: 2.0.0
>
> Attachments: HBASE-14702_v5.2_addendum-addendum.patch,
> HBASE-14703-5.2-addendum.patch, HBASE-14703-async.patch,
> HBASE-14703-start.patch, HBASE-14703-v4.1.patch, HBASE-14703-v4.patch,
> HBASE-14703-v6_with-check-and-mutate.patch, HBASE-14703.patch,
> HBASE-14703_v1.patch, HBASE-14703_v2.patch, HBASE-14703_v3.patch,
> HBASE-14703_v5.1.patch, HBASE-14703_v5.2.patch, HBASE-14703_v5.patch,
> HBASE-14703_v6.patch
>
>
> In {{AsyncProcess.SingleServerRequestRunnable}}, it seems we update
> serverStatistics twice.
> The first one is that we wrapper {{RetryingCallable}} by
> {{StatsTrackingRpcRetryingCaller}}, and do serverStatistics update when we
> call {{callWithRetries}} and {{callWithoutRetries}}. Relates code like below:
> {code}
> @Override
> public T callWithRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> @Override
> public T callWithoutRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> {code}
> The secondary one is after we get response, in {{receiveMultiAction}}, we do
> update again.
> {code}
> // update the stats about the region, if its a user table. We don't want to
> slow down
> // updates to meta tables, especially from internal updates (master, etc).
> if (AsyncProcess.this.connection.getStatisticsTracker() != null) {
> result = ResultStatsUtil.updateStats(result,
> AsyncProcess.this.connection.getStatisticsTracker(), server, regionName);
> }
> {code}
> It seems that {{StatsTrackingRpcRetryingCaller}} is NOT necessary, remove it?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14703) not collect stats when call HTable.mutateRow
[
https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025023#comment-15025023
]
Jesse Yates commented on HBASE-14703:
-
With the amount of copied code, we will definitely want to consider making a
new callable class to that does a lot of the work, but that's for a future
patch. FWIW, this wasn't too complicated once I figured out how manage the
processed state, which is a testament to this new style.
> not collect stats when call HTable.mutateRow
> -
>
> Key: HBASE-14703
> URL: https://issues.apache.org/jira/browse/HBASE-14703
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
>Assignee: Heng Chen
> Fix For: 2.0.0
>
> Attachments: HBASE-14702_v5.2_addendum-addendum.patch,
> HBASE-14703-5.2-addendum.patch, HBASE-14703-async.patch,
> HBASE-14703-start.patch, HBASE-14703-v4.1.patch, HBASE-14703-v4.patch,
> HBASE-14703-v6_with-check-and-mutate.patch, HBASE-14703.patch,
> HBASE-14703_v1.patch, HBASE-14703_v2.patch, HBASE-14703_v3.patch,
> HBASE-14703_v5.1.patch, HBASE-14703_v5.2.patch, HBASE-14703_v5.patch,
> HBASE-14703_v6.patch
>
>
> In {{AsyncProcess.SingleServerRequestRunnable}}, it seems we update
> serverStatistics twice.
> The first one is that we wrapper {{RetryingCallable}} by
> {{StatsTrackingRpcRetryingCaller}}, and do serverStatistics update when we
> call {{callWithRetries}} and {{callWithoutRetries}}. Relates code like below:
> {code}
> @Override
> public T callWithRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> @Override
> public T callWithoutRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> {code}
> The secondary one is after we get response, in {{receiveMultiAction}}, we do
> update again.
> {code}
> // update the stats about the region, if its a user table. We don't want to
> slow down
> // updates to meta tables, especially from internal updates (master, etc).
> if (AsyncProcess.this.connection.getStatisticsTracker() != null) {
> result = ResultStatsUtil.updateStats(result,
> AsyncProcess.this.connection.getStatisticsTracker(), server, regionName);
> }
> {code}
> It seems that {{StatsTrackingRpcRetryingCaller}} is NOT necessary, remove it?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14860) Improve BoundedByteBufferPool; make lockless
[ https://issues.apache.org/jira/browse/HBASE-14860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025017#comment-15025017 ] ramkrishna.s.vasudevan commented on HBASE-14860: The scan failed with OOME. Thanks [~ikeda]. > Improve BoundedByteBufferPool; make lockless > > > Key: HBASE-14860 > URL: https://issues.apache.org/jira/browse/HBASE-14860 > Project: HBase > Issue Type: Improvement >Reporter: Hiroshi Ikeda >Assignee: Hiroshi Ikeda >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14860.patch > > > Make it unblocking. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14860) Improve BoundedByteBufferPool; make lockless
[ https://issues.apache.org/jira/browse/HBASE-14860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025026#comment-15025026 ] stack commented on HBASE-14860: --- bq. I'll fix it tomorrow. Sounds good. Thanks [~ikeda] (I should have caught that in review) > Improve BoundedByteBufferPool; make lockless > > > Key: HBASE-14860 > URL: https://issues.apache.org/jira/browse/HBASE-14860 > Project: HBase > Issue Type: Improvement >Reporter: Hiroshi Ikeda >Assignee: Hiroshi Ikeda >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14860.patch > > > Make it unblocking. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14030) HBase Backup/Restore Phase 1
[ https://issues.apache.org/jira/browse/HBASE-14030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025064#comment-15025064 ] Jerry He commented on HBASE-14030: -- Okay. bq. In case of a whole cluster crash, it is supposed to be restored first. It will be delicate. Most of the info in the backup system table is probably not relevant anymore on a new instance. For example, all the kept filename, timestamps, etc. If the backup sessions are still wanted, the other info probably should be cleaned. The info on filename and timestamps are instance dependent and time sensitive (follow a time series). > HBase Backup/Restore Phase 1 > > > Key: HBASE-14030 > URL: https://issues.apache.org/jira/browse/HBASE-14030 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-14030-v0.patch, HBASE-14030-v1.patch, > HBASE-14030-v10.patch, HBASE-14030-v11.patch, HBASE-14030-v12.patch, > HBASE-14030-v13.patch, HBASE-14030-v14.patch, HBASE-14030-v15.patch, > HBASE-14030-v17.patch, HBASE-14030-v18.patch, HBASE-14030-v2.patch, > HBASE-14030-v3.patch, HBASE-14030-v4.patch, HBASE-14030-v5.patch, > HBASE-14030-v6.patch, HBASE-14030-v7.patch, HBASE-14030-v8.patch > > > This is the umbrella ticket for Backup/Restore Phase 1. See HBASE-7912 design > doc for the phase description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14703) not collect stats when call HTable.mutateRow
[
https://issues.apache.org/jira/browse/HBASE-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025063#comment-15025063
]
Hadoop QA commented on HBASE-14703:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12774115/HBASE-14703-v6_with-check-and-mutate.patch
against master branch at commit 4a60c25c702a57d043ca3fd3a9996f9fe63f9343.
ATTACHMENT ID: 12774115
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16655//console
This message is automatically generated.
> not collect stats when call HTable.mutateRow
> -
>
> Key: HBASE-14703
> URL: https://issues.apache.org/jira/browse/HBASE-14703
> Project: HBase
> Issue Type: Bug
>Reporter: Heng Chen
>Assignee: Heng Chen
> Fix For: 2.0.0
>
> Attachments: HBASE-14702_v5.2_addendum-addendum.patch,
> HBASE-14703-5.2-addendum.patch, HBASE-14703-async.patch,
> HBASE-14703-start.patch, HBASE-14703-v4.1.patch, HBASE-14703-v4.patch,
> HBASE-14703-v6_with-check-and-mutate.patch, HBASE-14703.patch,
> HBASE-14703_v1.patch, HBASE-14703_v2.patch, HBASE-14703_v3.patch,
> HBASE-14703_v5.1.patch, HBASE-14703_v5.2.patch, HBASE-14703_v5.patch,
> HBASE-14703_v6.patch
>
>
> In {{AsyncProcess.SingleServerRequestRunnable}}, it seems we update
> serverStatistics twice.
> The first one is that we wrapper {{RetryingCallable}} by
> {{StatsTrackingRpcRetryingCaller}}, and do serverStatistics update when we
> call {{callWithRetries}} and {{callWithoutRetries}}. Relates code like below:
> {code}
> @Override
> public T callWithRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> @Override
> public T callWithoutRetries(RetryingCallable callable, int callTimeout)
> throws IOException, RuntimeException {
> T result = delegate.callWithRetries(callable, callTimeout);
> return updateStatsAndUnwrap(result, callable);
> }
> {code}
> The secondary one is after we get response, in {{receiveMultiAction}}, we do
> update again.
> {code}
> // update the stats about the region, if its a user table. We don't want to
> slow down
> // updates to meta tables, especially from internal updates (master, etc).
> if (AsyncProcess.this.connection.getStatisticsTracker() != null) {
> result = ResultStatsUtil.updateStats(result,
> AsyncProcess.this.connection.getStatisticsTracker(), server, regionName);
> }
> {code}
> It seems that {{StatsTrackingRpcRetryingCaller}} is NOT necessary, remove it?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14777) Fix Inter Cluster Replication Future ordering issues
[
https://issues.apache.org/jira/browse/HBASE-14777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15025078#comment-15025078
]
Lars Hofhansl commented on HBASE-14777:
---
Any comments [~ashu210890], [~stack], [~eclark]? I'd like to commit my
addendum, with the real fix.
> Fix Inter Cluster Replication Future ordering issues
>
>
> Key: HBASE-14777
> URL: https://issues.apache.org/jira/browse/HBASE-14777
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Bhupendra Kumar Jain
>Assignee: Ashu Pachauri
>Priority: Critical
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: 14777-alternative.txt, HBASE-14777-1.patch,
> HBASE-14777-2.patch, HBASE-14777-3.patch, HBASE-14777-4.patch,
> HBASE-14777-5.patch, HBASE-14777-6.patch, HBASE-14777-addendum.patch,
> HBASE-14777.patch
>
>
> Replication fails with IndexOutOfBoundsException
> {code}
> regionserver.ReplicationSource$ReplicationSourceWorkerThread(939):
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint
> threw unknown exception:java.lang.IndexOutOfBoundsException: Index: 1, Size:
> 1
> at java.util.ArrayList.rangeCheck(Unknown Source)
> at java.util.ArrayList.remove(Unknown Source)
> at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint.replicate(HBaseInterClusterReplicationEndpoint.java:222)
> {code}
> Its happening due to incorrect removal of entries from the replication
> entries list.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
