[jira] [Commented] (HBASE-17125) Inconsistent result when use filter to read data
[
https://issues.apache.org/jira/browse/HBASE-17125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16058872#comment-16058872
]
Guanghao Zhang commented on HBASE-17125:
Take a summary about the key points of this fix.
1. Easy to use for user, no need extra work.
2. No more complicated work for ColumnTracker. Only fix this when use filter.
3. Accurate javadoc. And mark scan's setMaxVersions as Deprecated.
Attach a 012 patch and upload to review board, too.
The new patch add two new SQM: UserScanWithFilterQueryMatcher and
RawScanWithFilterQueryMatcher. Fix this bug in these SQM internal. If no
filter, it should same with previous implementation. Thanks.
> Inconsistent result when use filter to read data
>
>
> Key: HBASE-17125
> URL: https://issues.apache.org/jira/browse/HBASE-17125
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: example.diff, HBASE-17125.master.001.patch,
> HBASE-17125.master.002.patch, HBASE-17125.master.002.patch,
> HBASE-17125.master.003.patch, HBASE-17125.master.004.patch,
> HBASE-17125.master.005.patch, HBASE-17125.master.006.patch,
> HBASE-17125.master.007.patch, HBASE-17125.master.008.patch,
> HBASE-17125.master.009.patch, HBASE-17125.master.009.patch,
> HBASE-17125.master.010.patch, HBASE-17125.master.011.patch,
> HBASE-17125.master.011.patch, HBASE-17125.master.012.patch,
> HBASE-17125.master.checkReturnedVersions.patch,
> HBASE-17125.master.no-specified-filter.patch
>
>
> Assume a cloumn's max versions is 3, then we write 4 versions of this column.
> The oldest version doesn't remove immediately. But from the user view, the
> oldest version has gone. When user use a filter to query, if the filter skip
> a new version, then the oldest version will be seen again. But after compact
> the region, then the oldest version will never been seen. So it is weird for
> user. The query will get inconsistent result before and after region
> compaction.
> The reason is matchColumn method of UserScanQueryMatcher. It first check the
> cell by filter, then check the number of versions needed. So if the filter
> skip the new version, then the oldest version will be seen again when it is
> not removed.
> Have a discussion offline with [~Apache9] and [~fenghh], now we have two
> solution for this problem. The first idea is check the number of versions
> first, then check the cell by filter. As the comment of setFilter, the filter
> is called after all tests for ttl, column match, deletes and max versions
> have been run.
> {code}
> /**
>* Apply the specified server-side filter when performing the Query.
>* Only {@link Filter#filterKeyValue(Cell)} is called AFTER all tests
>* for ttl, column match, deletes and max versions have been run.
>* @param filter filter to run on the server
>* @return this for invocation chaining
>*/
> public Query setFilter(Filter filter) {
> this.filter = filter;
> return this;
> }
> {code}
> But this idea has another problem, if a column's max version is 5 and the
> user query only need 3 versions. It first check the version's number, then
> check the cell by filter. So the cells number of the result may less than 3.
> But there are 2 versions which don't read anymore.
> So the second idea has three steps.
> 1. check by the max versions of this column
> 2. check the kv by filter
> 3. check the versions which user need.
> But this will lead the ScanQueryMatcher more complicated. And this will break
> the javadoc of Query.setFilter.
> Now we don't have a final solution for this problem. Suggestions are welcomed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18254) ServerCrashProcedure checks and waits for meta initialized, instead should check and wait for meta loaded
[ https://issues.apache.org/jira/browse/HBASE-18254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16058933#comment-16058933 ] Hudson commented on HBASE-18254: FAILURE: Integrated in Jenkins build HBase-2.0 #87 (See [https://builds.apache.org/job/HBase-2.0/87/]) HBASE-18254 ServerCrashProcedure checks and waits for meta initialized, (stack: rev 0357c2266a306ffea6c1c2f98a022d116b4f8c56) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestServerCrashProcedure.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java > ServerCrashProcedure checks and waits for meta initialized, instead should > check and wait for meta loaded > - > > Key: HBASE-18254 > URL: https://issues.apache.org/jira/browse/HBASE-18254 > Project: HBase > Issue Type: Bug > Components: amv2 >Affects Versions: 2.0.0-alpha-1 >Reporter: Umesh Agashe >Assignee: Umesh Agashe > Fix For: 2.0.0 > > Attachments: HBASE-18254.master.001.patch > > > After enabling test > hbase.master.procedure.TestServerCrashProcedure#testRecoveryAndDoubleExecutionOnRsWithMeta, > this bug is found in ServerCrashProcedure. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18167) OfflineMetaRepair tool may cause HMaster abort always
[
https://issues.apache.org/jira/browse/HBASE-18167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16058968#comment-16058968
]
Pankaj Kumar commented on HBASE-18167:
--
Ted:
All test cases are successful except
TestSnapshotHFileCleaner#testMissedTmpSnapshot and integration test cases.
1. TestSnapshotHFileCleaner#testMissedTmpSnapshot failing in branch-1 also when
executed locally.
2. Facing error while running Integration test cases locally.
> OfflineMetaRepair tool may cause HMaster abort always
> -
>
> Key: HBASE-18167
> URL: https://issues.apache.org/jira/browse/HBASE-18167
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 1.4.0, 1.3.1, 1.3.2
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 1.4.0, 1.3.2
>
> Attachments: HBASE-18167-branch-1.3.v2.patch,
> HBASE-18167-branch-1.patch, HBASE-18167-branch-1-V2.patch
>
>
> In the production environment, we met a weird scenario where some Meta table
> HFile blocks were missing due to some reason.
> To recover the environment we tried to rebuild the meta using
> OfflineMetaRepair tool and restart the cluster, but HMaster couldn't finish
> it's initialization. It always timed out as namespace table region was never
> assigned.
> Steps to reproduce
> ==
> 1. Assign meta table region to HMaster (it can be on any RS, just to
> reproduce the scenario)
> {noformat}
>
> hbase.balancer.tablesOnMaster
> hbase:meta
>
> {noformat}
> 2. Start HMaster and RegionServer
> 2. Create two namespace, say "ns1" & "ns2"
> 3. Create two tables "ns1:t1' & "ns2:t1'
> 4. flush 'hbase:meta"
> 5. Stop HMaster (graceful shutdown)
> 6. Kill -9 RegionServer (Abnormal shutdown)
> 7. Run OfflineMetaRepair as follows,
> {noformat}
> hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair -fix
> {noformat}
> 8. Restart HMaster and RegionServer
> 9. HMaster will never be able to finish its initialization and abort always
> with below message,
> {code}
> 2017-06-06 15:11:07,582 FATAL [Hostname:16000.activeMasterManager]
> master.HMaster: Unhandled exception. Starting shutdown.
> java.io.IOException: Timedout 12ms waiting for namespace table to be
> assigned
> at
> org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:98)
> at
> org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:1054)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:848)
> at org.apache.hadoop.hbase.master.HMaster.access$600(HMaster.java:199)
> at org.apache.hadoop.hbase.master.HMaster$2.run(HMaster.java:1871)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Root cause
> ==
> 1. During HM start up AM assumes that it's a failover scenario based on the
> existing old WAL files, so SSH/SCP will split WAL files and assign the
> holding regions.
> 2. During SSH/SCP it retrieves the server holding regions from meta/AM's
> in-memory-state, but meta only had "regioninfo" entry (as already rebuild by
> OfflineMetaRepair). So empty region will be returned and it wont trigger any
> assignment.
> 3. HMaster which is waiting for namespace table to be assigned will timeout
> and abort always.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18254) ServerCrashProcedure checks and waits for meta initialized, instead should check and wait for meta loaded
[ https://issues.apache.org/jira/browse/HBASE-18254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16058970#comment-16058970 ] Hudson commented on HBASE-18254: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #3239 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3239/]) HBASE-18254 ServerCrashProcedure checks and waits for meta initialized, (stack: rev 7cc458e129cf7c55910c9985d0824a72189a1a73) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestServerCrashProcedure.java > ServerCrashProcedure checks and waits for meta initialized, instead should > check and wait for meta loaded > - > > Key: HBASE-18254 > URL: https://issues.apache.org/jira/browse/HBASE-18254 > Project: HBase > Issue Type: Bug > Components: amv2 >Affects Versions: 2.0.0-alpha-1 >Reporter: Umesh Agashe >Assignee: Umesh Agashe > Fix For: 2.0.0 > > Attachments: HBASE-18254.master.001.patch > > > After enabling test > hbase.master.procedure.TestServerCrashProcedure#testRecoveryAndDoubleExecutionOnRsWithMeta, > this bug is found in ServerCrashProcedure. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18258) Take down hbasecon2017 logo from hbase.apache.org and put up hbaseconasia2017 instead.
[ https://issues.apache.org/jira/browse/HBASE-18258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16058969#comment-16058969 ] Hudson commented on HBASE-18258: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #3239 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3239/]) HBASE-18258 Take down hbasecon2017 logo from hbase.apache.org and put up (stack: rev 9cb5fc9825ed74bb37dd8def9ea981c114d32f4b) * (edit) src/main/site/site.xml * (add) src/main/site/resources/images/hbaseconasia2017.png > Take down hbasecon2017 logo from hbase.apache.org and put up hbaseconasia2017 > instead. > -- > > Key: HBASE-18258 > URL: https://issues.apache.org/jira/browse/HBASE-18258 > Project: HBase > Issue Type: Task >Reporter: stack >Assignee: stack > Fix For: 3.0.0 > > > Change logo on home page. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17125) Inconsistent result when use filter to read data
[
https://issues.apache.org/jira/browse/HBASE-17125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-17125:
---
Attachment: HBASE-17125.master.013.patch
> Inconsistent result when use filter to read data
>
>
> Key: HBASE-17125
> URL: https://issues.apache.org/jira/browse/HBASE-17125
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: example.diff, HBASE-17125.master.001.patch,
> HBASE-17125.master.002.patch, HBASE-17125.master.002.patch,
> HBASE-17125.master.003.patch, HBASE-17125.master.004.patch,
> HBASE-17125.master.005.patch, HBASE-17125.master.006.patch,
> HBASE-17125.master.007.patch, HBASE-17125.master.008.patch,
> HBASE-17125.master.009.patch, HBASE-17125.master.009.patch,
> HBASE-17125.master.010.patch, HBASE-17125.master.011.patch,
> HBASE-17125.master.011.patch, HBASE-17125.master.012.patch,
> HBASE-17125.master.013.patch, HBASE-17125.master.checkReturnedVersions.patch,
> HBASE-17125.master.no-specified-filter.patch
>
>
> Assume a cloumn's max versions is 3, then we write 4 versions of this column.
> The oldest version doesn't remove immediately. But from the user view, the
> oldest version has gone. When user use a filter to query, if the filter skip
> a new version, then the oldest version will be seen again. But after compact
> the region, then the oldest version will never been seen. So it is weird for
> user. The query will get inconsistent result before and after region
> compaction.
> The reason is matchColumn method of UserScanQueryMatcher. It first check the
> cell by filter, then check the number of versions needed. So if the filter
> skip the new version, then the oldest version will be seen again when it is
> not removed.
> Have a discussion offline with [~Apache9] and [~fenghh], now we have two
> solution for this problem. The first idea is check the number of versions
> first, then check the cell by filter. As the comment of setFilter, the filter
> is called after all tests for ttl, column match, deletes and max versions
> have been run.
> {code}
> /**
>* Apply the specified server-side filter when performing the Query.
>* Only {@link Filter#filterKeyValue(Cell)} is called AFTER all tests
>* for ttl, column match, deletes and max versions have been run.
>* @param filter filter to run on the server
>* @return this for invocation chaining
>*/
> public Query setFilter(Filter filter) {
> this.filter = filter;
> return this;
> }
> {code}
> But this idea has another problem, if a column's max version is 5 and the
> user query only need 3 versions. It first check the version's number, then
> check the cell by filter. So the cells number of the result may less than 3.
> But there are 2 versions which don't read anymore.
> So the second idea has three steps.
> 1. check by the max versions of this column
> 2. check the kv by filter
> 3. check the versions which user need.
> But this will lead the ScanQueryMatcher more complicated. And this will break
> the javadoc of Query.setFilter.
> Now we don't have a final solution for this problem. Suggestions are welcomed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-17125) Inconsistent result when use filter to read data
[
https://issues.apache.org/jira/browse/HBASE-17125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059000#comment-16059000
]
Guanghao Zhang commented on HBASE-17125:
Attach 013 patch which remove the two new SQM to make code easy to read. And
fix the bug in UserScanQueryMatcher internal.
> Inconsistent result when use filter to read data
>
>
> Key: HBASE-17125
> URL: https://issues.apache.org/jira/browse/HBASE-17125
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: example.diff, HBASE-17125.master.001.patch,
> HBASE-17125.master.002.patch, HBASE-17125.master.002.patch,
> HBASE-17125.master.003.patch, HBASE-17125.master.004.patch,
> HBASE-17125.master.005.patch, HBASE-17125.master.006.patch,
> HBASE-17125.master.007.patch, HBASE-17125.master.008.patch,
> HBASE-17125.master.009.patch, HBASE-17125.master.009.patch,
> HBASE-17125.master.010.patch, HBASE-17125.master.011.patch,
> HBASE-17125.master.011.patch, HBASE-17125.master.012.patch,
> HBASE-17125.master.013.patch, HBASE-17125.master.checkReturnedVersions.patch,
> HBASE-17125.master.no-specified-filter.patch
>
>
> Assume a cloumn's max versions is 3, then we write 4 versions of this column.
> The oldest version doesn't remove immediately. But from the user view, the
> oldest version has gone. When user use a filter to query, if the filter skip
> a new version, then the oldest version will be seen again. But after compact
> the region, then the oldest version will never been seen. So it is weird for
> user. The query will get inconsistent result before and after region
> compaction.
> The reason is matchColumn method of UserScanQueryMatcher. It first check the
> cell by filter, then check the number of versions needed. So if the filter
> skip the new version, then the oldest version will be seen again when it is
> not removed.
> Have a discussion offline with [~Apache9] and [~fenghh], now we have two
> solution for this problem. The first idea is check the number of versions
> first, then check the cell by filter. As the comment of setFilter, the filter
> is called after all tests for ttl, column match, deletes and max versions
> have been run.
> {code}
> /**
>* Apply the specified server-side filter when performing the Query.
>* Only {@link Filter#filterKeyValue(Cell)} is called AFTER all tests
>* for ttl, column match, deletes and max versions have been run.
>* @param filter filter to run on the server
>* @return this for invocation chaining
>*/
> public Query setFilter(Filter filter) {
> this.filter = filter;
> return this;
> }
> {code}
> But this idea has another problem, if a column's max version is 5 and the
> user query only need 3 versions. It first check the version's number, then
> check the cell by filter. So the cells number of the result may less than 3.
> But there are 2 versions which don't read anymore.
> So the second idea has three steps.
> 1. check by the max versions of this column
> 2. check the kv by filter
> 3. check the versions which user need.
> But this will lead the ScanQueryMatcher more complicated. And this will break
> the javadoc of Query.setFilter.
> Now we don't have a final solution for this problem. Suggestions are welcomed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-17125) Inconsistent result when use filter to read data
[
https://issues.apache.org/jira/browse/HBASE-17125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-17125:
---
Fix Version/s: (was: 2.0.0)
2.0.0-alpha-2
3.0.0
> Inconsistent result when use filter to read data
>
>
> Key: HBASE-17125
> URL: https://issues.apache.org/jira/browse/HBASE-17125
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: example.diff, HBASE-17125.master.001.patch,
> HBASE-17125.master.002.patch, HBASE-17125.master.002.patch,
> HBASE-17125.master.003.patch, HBASE-17125.master.004.patch,
> HBASE-17125.master.005.patch, HBASE-17125.master.006.patch,
> HBASE-17125.master.007.patch, HBASE-17125.master.008.patch,
> HBASE-17125.master.009.patch, HBASE-17125.master.009.patch,
> HBASE-17125.master.010.patch, HBASE-17125.master.011.patch,
> HBASE-17125.master.011.patch, HBASE-17125.master.012.patch,
> HBASE-17125.master.013.patch, HBASE-17125.master.checkReturnedVersions.patch,
> HBASE-17125.master.no-specified-filter.patch
>
>
> Assume a cloumn's max versions is 3, then we write 4 versions of this column.
> The oldest version doesn't remove immediately. But from the user view, the
> oldest version has gone. When user use a filter to query, if the filter skip
> a new version, then the oldest version will be seen again. But after compact
> the region, then the oldest version will never been seen. So it is weird for
> user. The query will get inconsistent result before and after region
> compaction.
> The reason is matchColumn method of UserScanQueryMatcher. It first check the
> cell by filter, then check the number of versions needed. So if the filter
> skip the new version, then the oldest version will be seen again when it is
> not removed.
> Have a discussion offline with [~Apache9] and [~fenghh], now we have two
> solution for this problem. The first idea is check the number of versions
> first, then check the cell by filter. As the comment of setFilter, the filter
> is called after all tests for ttl, column match, deletes and max versions
> have been run.
> {code}
> /**
>* Apply the specified server-side filter when performing the Query.
>* Only {@link Filter#filterKeyValue(Cell)} is called AFTER all tests
>* for ttl, column match, deletes and max versions have been run.
>* @param filter filter to run on the server
>* @return this for invocation chaining
>*/
> public Query setFilter(Filter filter) {
> this.filter = filter;
> return this;
> }
> {code}
> But this idea has another problem, if a column's max version is 5 and the
> user query only need 3 versions. It first check the version's number, then
> check the cell by filter. So the cells number of the result may less than 3.
> But there are 2 versions which don't read anymore.
> So the second idea has three steps.
> 1. check by the max versions of this column
> 2. check the kv by filter
> 3. check the versions which user need.
> But this will lead the ScanQueryMatcher more complicated. And this will break
> the javadoc of Query.setFilter.
> Now we don't have a final solution for this problem. Suggestions are welcomed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18010) Connect CellChunkMap to be used for flattening in CompactingMemStore
[ https://issues.apache.org/jira/browse/HBASE-18010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anastasia Braginsky updated HBASE-18010: Attachment: HBASE-18010-V07.patch > Connect CellChunkMap to be used for flattening in CompactingMemStore > > > Key: HBASE-18010 > URL: https://issues.apache.org/jira/browse/HBASE-18010 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Attachments: HBASE-18010-V04.patch, HBASE-18010-V06.patch, > HBASE-18010-V07.patch > > > The CellChunkMap helps to create a new type of ImmutableSegment, where the > index (CellSet's delegatee) is going to be CellChunkMap. No big cells or > upserted cells are going to be supported here. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18010) Connect CellChunkMap to be used for flattening in CompactingMemStore
[ https://issues.apache.org/jira/browse/HBASE-18010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059008#comment-16059008 ] Anastasia Braginsky commented on HBASE-18010: - Publishing here and on the RB, the patch including referring to all review comments. I have also answered all the comments on the review, so please take a look there. [~anoop.hbase], [~ram_krish], [~eshcar], [~ebortnik], please take one more moment and look on the last variant of the patch. Your comments are very valuable! > Connect CellChunkMap to be used for flattening in CompactingMemStore > > > Key: HBASE-18010 > URL: https://issues.apache.org/jira/browse/HBASE-18010 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Attachments: HBASE-18010-V04.patch, HBASE-18010-V06.patch, > HBASE-18010-V07.patch > > > The CellChunkMap helps to create a new type of ImmutableSegment, where the > index (CellSet's delegatee) is going to be CellChunkMap. No big cells or > upserted cells are going to be supported here. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18167) OfflineMetaRepair tool may cause HMaster abort always
[
https://issues.apache.org/jira/browse/HBASE-18167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059028#comment-16059028
]
Hudson commented on HBASE-18167:
FAILURE: Integrated in Jenkins build HBase-1.4 #786 (See
[https://builds.apache.org/job/HBase-1.4/786/])
HBASE-18167 OfflineMetaRepair tool may cause HMaster to abort always - (tedyu:
rev 3d9c54442865521ab7c7f923e0c475a7ced95035)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildBase.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
> OfflineMetaRepair tool may cause HMaster abort always
> -
>
> Key: HBASE-18167
> URL: https://issues.apache.org/jira/browse/HBASE-18167
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 1.4.0, 1.3.1, 1.3.2
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 1.4.0, 1.3.2
>
> Attachments: HBASE-18167-branch-1.3.v2.patch,
> HBASE-18167-branch-1.patch, HBASE-18167-branch-1-V2.patch
>
>
> In the production environment, we met a weird scenario where some Meta table
> HFile blocks were missing due to some reason.
> To recover the environment we tried to rebuild the meta using
> OfflineMetaRepair tool and restart the cluster, but HMaster couldn't finish
> it's initialization. It always timed out as namespace table region was never
> assigned.
> Steps to reproduce
> ==
> 1. Assign meta table region to HMaster (it can be on any RS, just to
> reproduce the scenario)
> {noformat}
>
> hbase.balancer.tablesOnMaster
> hbase:meta
>
> {noformat}
> 2. Start HMaster and RegionServer
> 2. Create two namespace, say "ns1" & "ns2"
> 3. Create two tables "ns1:t1' & "ns2:t1'
> 4. flush 'hbase:meta"
> 5. Stop HMaster (graceful shutdown)
> 6. Kill -9 RegionServer (Abnormal shutdown)
> 7. Run OfflineMetaRepair as follows,
> {noformat}
> hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair -fix
> {noformat}
> 8. Restart HMaster and RegionServer
> 9. HMaster will never be able to finish its initialization and abort always
> with below message,
> {code}
> 2017-06-06 15:11:07,582 FATAL [Hostname:16000.activeMasterManager]
> master.HMaster: Unhandled exception. Starting shutdown.
> java.io.IOException: Timedout 12ms waiting for namespace table to be
> assigned
> at
> org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:98)
> at
> org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:1054)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:848)
> at org.apache.hadoop.hbase.master.HMaster.access$600(HMaster.java:199)
> at org.apache.hadoop.hbase.master.HMaster$2.run(HMaster.java:1871)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Root cause
> ==
> 1. During HM start up AM assumes that it's a failover scenario based on the
> existing old WAL files, so SSH/SCP will split WAL files and assign the
> holding regions.
> 2. During SSH/SCP it retrieves the server holding regions from meta/AM's
> in-memory-state, but meta only had "regioninfo" entry (as already rebuild by
> OfflineMetaRepair). So empty region will be returned and it wont trigger any
> assignment.
> 3. HMaster which is waiting for namespace table to be assigned will timeout
> and abort always.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18167) OfflineMetaRepair tool may cause HMaster abort always
[
https://issues.apache.org/jira/browse/HBASE-18167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059031#comment-16059031
]
Hudson commented on HBASE-18167:
FAILURE: Integrated in Jenkins build HBase-1.4 #787 (See
[https://builds.apache.org/job/HBase-1.4/787/])
HBASE-18167 OfflineMetaRepair tool may cause HMaster to abort always (tedyu:
rev d5bba50794538293e0361b87c9fdaff375ca4177)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildBase.java
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
> OfflineMetaRepair tool may cause HMaster abort always
> -
>
> Key: HBASE-18167
> URL: https://issues.apache.org/jira/browse/HBASE-18167
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 1.4.0, 1.3.1, 1.3.2
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 1.4.0, 1.3.2
>
> Attachments: HBASE-18167-branch-1.3.v2.patch,
> HBASE-18167-branch-1.patch, HBASE-18167-branch-1-V2.patch
>
>
> In the production environment, we met a weird scenario where some Meta table
> HFile blocks were missing due to some reason.
> To recover the environment we tried to rebuild the meta using
> OfflineMetaRepair tool and restart the cluster, but HMaster couldn't finish
> it's initialization. It always timed out as namespace table region was never
> assigned.
> Steps to reproduce
> ==
> 1. Assign meta table region to HMaster (it can be on any RS, just to
> reproduce the scenario)
> {noformat}
>
> hbase.balancer.tablesOnMaster
> hbase:meta
>
> {noformat}
> 2. Start HMaster and RegionServer
> 2. Create two namespace, say "ns1" & "ns2"
> 3. Create two tables "ns1:t1' & "ns2:t1'
> 4. flush 'hbase:meta"
> 5. Stop HMaster (graceful shutdown)
> 6. Kill -9 RegionServer (Abnormal shutdown)
> 7. Run OfflineMetaRepair as follows,
> {noformat}
> hbase org.apache.hadoop.hbase.util.hbck.OfflineMetaRepair -fix
> {noformat}
> 8. Restart HMaster and RegionServer
> 9. HMaster will never be able to finish its initialization and abort always
> with below message,
> {code}
> 2017-06-06 15:11:07,582 FATAL [Hostname:16000.activeMasterManager]
> master.HMaster: Unhandled exception. Starting shutdown.
> java.io.IOException: Timedout 12ms waiting for namespace table to be
> assigned
> at
> org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:98)
> at
> org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:1054)
> at
> org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:848)
> at org.apache.hadoop.hbase.master.HMaster.access$600(HMaster.java:199)
> at org.apache.hadoop.hbase.master.HMaster$2.run(HMaster.java:1871)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Root cause
> ==
> 1. During HM start up AM assumes that it's a failover scenario based on the
> existing old WAL files, so SSH/SCP will split WAL files and assign the
> holding regions.
> 2. During SSH/SCP it retrieves the server holding regions from meta/AM's
> in-memory-state, but meta only had "regioninfo" entry (as already rebuild by
> OfflineMetaRepair). So empty region will be returned and it wont trigger any
> assignment.
> 3. HMaster which is waiting for namespace table to be assigned will timeout
> and abort always.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-17125) Inconsistent result when use filter to read data
[
https://issues.apache.org/jira/browse/HBASE-17125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059035#comment-16059035
]
Hadoop QA commented on HBASE-17125:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 22s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 34s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
49s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 11s
{color} | {color:red} hbase-client in master has 4 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 7m 56s
{color} | {color:red} hbase-server in master has 10 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 59s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 30s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
1s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
59m 29s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha3. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
15s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 33s
{color} | {color:red} hbase-client generated 8 new + 1 unchanged - 0 fixed = 9
total (was 1) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 42s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 35m 17s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 35s
{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 144m 10s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.filter.TestFilter |
| | hadoop.hbase.master.locking.TestLockProcedure |
| | hadoop.hbase.filter.TestInvocationRecordFilter |
| | hadoop.hbase.io.encoding.TestSeekBeforeWithReverseScan |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12874024/HBASE-17125.master.012.patch
|
| JIRA Issue | HBASE-17125 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux bfde76c5c6f2 4.8.3-std-1 #1 SMP Fri Oct 21 11:15:43 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hb
[jira] [Commented] (HBASE-18221) Switch from pread to stream should happen under HStore's reentrant lock
[
https://issues.apache.org/jira/browse/HBASE-18221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059090#comment-16059090
]
ramkrishna.s.vasudevan commented on HBASE-18221:
[~Apache9] - Ping !!
> Switch from pread to stream should happen under HStore's reentrant lock
> ---
>
> Key: HBASE-18221
> URL: https://issues.apache.org/jira/browse/HBASE-18221
> Project: HBase
> Issue Type: Sub-task
> Components: Scanners
>Affects Versions: 2.0.0, 3.0.0, 2.0.0-alpha-1
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-18221_2_fortestcasefailure.patch,
> HBASE-18221_2.patch, HBASE-18221_2.patch
>
>
> Found this while debugging HBASE-18186. When we try to reopen the scanners on
> the storefiles while trying to switch over from pread to stream, we do not
> use the HStore's reentrant lock to get the current Storefiles from the
> StoreFileManager. All the scan APIs are guarded under that and we must do it
> here also other wise the CompactedHfileDischarger may cause race issues with
> the HStore's datastructures like here
> {code}
> 2017-06-14 18:16:17,223 WARN
> [RpcServer.default.FPBQ.Fifo.handler=23,queue=1,port=16020]
> regionserver.StoreScanner: failed to switch to stream read
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.getScannersForStoreFiles(StoreFileScanner.java:133)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanners(HStore.java:1221)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.trySwitchToStreamRead(StoreScanner.java:997)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.shipped(StoreScanner.java:1134)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.shipped(KeyValueHeap.java:445)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.shipped(HRegion.java:6459)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices$RegionScannerShippedCallBack.run(RSRpcServices.java:339)
> at
> org.apache.hadoop.hbase.ipc.ServerCall.setResponse(ServerCall.java:252)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:166)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258)
> {code}
> I have a working patch fixing this problem. Will do some more testing and try
> to upload the patch after I write a test case for this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-17738) BucketCache startup is slow
[ https://issues.apache.org/jira/browse/HBASE-17738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059092#comment-16059092 ] ramkrishna.s.vasudevan commented on HBASE-17738: bq.Say we have 99 BBs to be created and we have 10 threads, make 9 threads to create 10 BBs each and the last one to create 9? Oh missed seeing this latest comment. So you mean we need to split and then send it to every callable? So by default always do a split of 10 each? > BucketCache startup is slow > --- > > Key: HBASE-17738 > URL: https://issues.apache.org/jira/browse/HBASE-17738 > Project: HBase > Issue Type: Sub-task > Components: BucketCache >Affects Versions: 2.0.0 >Reporter: stack >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-17738_2.patch, HBASE-17738_2.patch, > HBASE-17738_3.patch, HBASE-17738_4.patch, HBASE-17738_5_withoutUnsafe.patch, > HBASE-17738.patch > > > If you set bucketcache size at 64G say and then start hbase, it takes a long > time. Can we do the allocations in parallel and not inline with the server > startup? > Related, prefetching on a bucketcache is slow. Speed it up. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18221) Switch from pread to stream should happen under HStore's reentrant lock
[
https://issues.apache.org/jira/browse/HBASE-18221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059103#comment-16059103
]
Duo Zhang commented on HBASE-18221:
---
+1.
> Switch from pread to stream should happen under HStore's reentrant lock
> ---
>
> Key: HBASE-18221
> URL: https://issues.apache.org/jira/browse/HBASE-18221
> Project: HBase
> Issue Type: Sub-task
> Components: Scanners
>Affects Versions: 2.0.0, 3.0.0, 2.0.0-alpha-1
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-18221_2_fortestcasefailure.patch,
> HBASE-18221_2.patch, HBASE-18221_2.patch
>
>
> Found this while debugging HBASE-18186. When we try to reopen the scanners on
> the storefiles while trying to switch over from pread to stream, we do not
> use the HStore's reentrant lock to get the current Storefiles from the
> StoreFileManager. All the scan APIs are guarded under that and we must do it
> here also other wise the CompactedHfileDischarger may cause race issues with
> the HStore's datastructures like here
> {code}
> 2017-06-14 18:16:17,223 WARN
> [RpcServer.default.FPBQ.Fifo.handler=23,queue=1,port=16020]
> regionserver.StoreScanner: failed to switch to stream read
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.getScannersForStoreFiles(StoreFileScanner.java:133)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanners(HStore.java:1221)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.trySwitchToStreamRead(StoreScanner.java:997)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.shipped(StoreScanner.java:1134)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.shipped(KeyValueHeap.java:445)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.shipped(HRegion.java:6459)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices$RegionScannerShippedCallBack.run(RSRpcServices.java:339)
> at
> org.apache.hadoop.hbase.ipc.ServerCall.setResponse(ServerCall.java:252)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:166)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258)
> {code}
> I have a working patch fixing this problem. Will do some more testing and try
> to upload the patch after I write a test case for this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HBASE-18221) Switch from pread to stream should happen under HStore's reentrant lock
[
https://issues.apache.org/jira/browse/HBASE-18221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059103#comment-16059103
]
Duo Zhang edited comment on HBASE-18221 at 6/22/17 10:10 AM:
-
Skimmed. LGTM. +1.
But I think [~anoop.hbase] maybe a little upset since you add new methods to
the Store interface...
was (Author: apache9):
+1.
> Switch from pread to stream should happen under HStore's reentrant lock
> ---
>
> Key: HBASE-18221
> URL: https://issues.apache.org/jira/browse/HBASE-18221
> Project: HBase
> Issue Type: Sub-task
> Components: Scanners
>Affects Versions: 2.0.0, 3.0.0, 2.0.0-alpha-1
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-18221_2_fortestcasefailure.patch,
> HBASE-18221_2.patch, HBASE-18221_2.patch
>
>
> Found this while debugging HBASE-18186. When we try to reopen the scanners on
> the storefiles while trying to switch over from pread to stream, we do not
> use the HStore's reentrant lock to get the current Storefiles from the
> StoreFileManager. All the scan APIs are guarded under that and we must do it
> here also other wise the CompactedHfileDischarger may cause race issues with
> the HStore's datastructures like here
> {code}
> 2017-06-14 18:16:17,223 WARN
> [RpcServer.default.FPBQ.Fifo.handler=23,queue=1,port=16020]
> regionserver.StoreScanner: failed to switch to stream read
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.getScannersForStoreFiles(StoreFileScanner.java:133)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanners(HStore.java:1221)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.trySwitchToStreamRead(StoreScanner.java:997)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.shipped(StoreScanner.java:1134)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.shipped(KeyValueHeap.java:445)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.shipped(HRegion.java:6459)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices$RegionScannerShippedCallBack.run(RSRpcServices.java:339)
> at
> org.apache.hadoop.hbase.ipc.ServerCall.setResponse(ServerCall.java:252)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:166)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258)
> {code}
> I have a working patch fixing this problem. Will do some more testing and try
> to upload the patch after I write a test case for this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-16868) Add a replicate_all flag to avoid misuse the namespaces and table-cfs config of replication peer
[
https://issues.apache.org/jira/browse/HBASE-16868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-16868:
---
Attachment: HBASE-16868.master.003.patch
> Add a replicate_all flag to avoid misuse the namespaces and table-cfs config
> of replication peer
>
>
> Key: HBASE-16868
> URL: https://issues.apache.org/jira/browse/HBASE-16868
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Affects Versions: 2.0.0, 3.0.0
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: HBASE-16868.master.001.patch,
> HBASE-16868.master.002.patch, HBASE-16868.master.003.patch
>
>
> First add a new peer by shell cmd.
> {code}
> add_peer '1', CLUSTER_KEY => "server1.cie.com:2181:/hbase".
> {code}
> If we don't set namespaces and table cfs in peer config. It means replicate
> all tables to the peer cluster.
> Then append a table to the peer config.
> {code}
> append_peer_tableCFs '1', {"table1" => []}
> {code}
> Then this peer will only replicate table1 to the peer cluster. It changes to
> replicate only one table from replicate all tables in the cluster. It is very
> easy to misuse in production cluster. So we should avoid appending table to a
> peer which replicates all table.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18221) Switch from pread to stream should happen under HStore's reentrant lock
[
https://issues.apache.org/jira/browse/HBASE-18221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059104#comment-16059104
]
ramkrishna.s.vasudevan commented on HBASE-18221:
bq.One question is that given the fact we are now trying to avoid unnecessary
APIs from the LimitedPrivate interfaces this method could add on to that. But
as part of clean up this should also be taken into account.
Exactly I have mentioned the same in my above comment too.
> Switch from pread to stream should happen under HStore's reentrant lock
> ---
>
> Key: HBASE-18221
> URL: https://issues.apache.org/jira/browse/HBASE-18221
> Project: HBase
> Issue Type: Sub-task
> Components: Scanners
>Affects Versions: 2.0.0, 3.0.0, 2.0.0-alpha-1
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-18221_2_fortestcasefailure.patch,
> HBASE-18221_2.patch, HBASE-18221_2.patch
>
>
> Found this while debugging HBASE-18186. When we try to reopen the scanners on
> the storefiles while trying to switch over from pread to stream, we do not
> use the HStore's reentrant lock to get the current Storefiles from the
> StoreFileManager. All the scan APIs are guarded under that and we must do it
> here also other wise the CompactedHfileDischarger may cause race issues with
> the HStore's datastructures like here
> {code}
> 2017-06-14 18:16:17,223 WARN
> [RpcServer.default.FPBQ.Fifo.handler=23,queue=1,port=16020]
> regionserver.StoreScanner: failed to switch to stream read
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.getScannersForStoreFiles(StoreFileScanner.java:133)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanners(HStore.java:1221)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.trySwitchToStreamRead(StoreScanner.java:997)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.shipped(StoreScanner.java:1134)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.shipped(KeyValueHeap.java:445)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.shipped(HRegion.java:6459)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices$RegionScannerShippedCallBack.run(RSRpcServices.java:339)
> at
> org.apache.hadoop.hbase.ipc.ServerCall.setResponse(ServerCall.java:252)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:166)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258)
> {code}
> I have a working patch fixing this problem. Will do some more testing and try
> to upload the patch after I write a test case for this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18221) Switch from pread to stream should happen under HStore's reentrant lock
[
https://issues.apache.org/jira/browse/HBASE-18221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059105#comment-16059105
]
ramkrishna.s.vasudevan commented on HBASE-18221:
Do you have any other better to work aroud this?
> Switch from pread to stream should happen under HStore's reentrant lock
> ---
>
> Key: HBASE-18221
> URL: https://issues.apache.org/jira/browse/HBASE-18221
> Project: HBase
> Issue Type: Sub-task
> Components: Scanners
>Affects Versions: 2.0.0, 3.0.0, 2.0.0-alpha-1
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-18221_2_fortestcasefailure.patch,
> HBASE-18221_2.patch, HBASE-18221_2.patch
>
>
> Found this while debugging HBASE-18186. When we try to reopen the scanners on
> the storefiles while trying to switch over from pread to stream, we do not
> use the HStore's reentrant lock to get the current Storefiles from the
> StoreFileManager. All the scan APIs are guarded under that and we must do it
> here also other wise the CompactedHfileDischarger may cause race issues with
> the HStore's datastructures like here
> {code}
> 2017-06-14 18:16:17,223 WARN
> [RpcServer.default.FPBQ.Fifo.handler=23,queue=1,port=16020]
> regionserver.StoreScanner: failed to switch to stream read
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.getScannersForStoreFiles(StoreFileScanner.java:133)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanners(HStore.java:1221)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.trySwitchToStreamRead(StoreScanner.java:997)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.shipped(StoreScanner.java:1134)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.shipped(KeyValueHeap.java:445)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.shipped(HRegion.java:6459)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices$RegionScannerShippedCallBack.run(RSRpcServices.java:339)
> at
> org.apache.hadoop.hbase.ipc.ServerCall.setResponse(ServerCall.java:252)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:166)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258)
> {code}
> I have a working patch fixing this problem. Will do some more testing and try
> to upload the patch after I write a test case for this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18221) Switch from pread to stream should happen under HStore's reentrant lock
[
https://issues.apache.org/jira/browse/HBASE-18221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059117#comment-16059117
]
Duo Zhang commented on HBASE-18221:
---
{quote}
Do you have any other better to work aroud this?
{quote}
I think this is another issue and should not block this issue. [~anoop.hbase]
has already opened several issues to track the problem. We can discuss there.
For me I think we can use HStore directly in our own code to prevent exposing
everything in Store interface.
Thanks.
> Switch from pread to stream should happen under HStore's reentrant lock
> ---
>
> Key: HBASE-18221
> URL: https://issues.apache.org/jira/browse/HBASE-18221
> Project: HBase
> Issue Type: Sub-task
> Components: Scanners
>Affects Versions: 2.0.0, 3.0.0, 2.0.0-alpha-1
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-18221_2_fortestcasefailure.patch,
> HBASE-18221_2.patch, HBASE-18221_2.patch
>
>
> Found this while debugging HBASE-18186. When we try to reopen the scanners on
> the storefiles while trying to switch over from pread to stream, we do not
> use the HStore's reentrant lock to get the current Storefiles from the
> StoreFileManager. All the scan APIs are guarded under that and we must do it
> here also other wise the CompactedHfileDischarger may cause race issues with
> the HStore's datastructures like here
> {code}
> 2017-06-14 18:16:17,223 WARN
> [RpcServer.default.FPBQ.Fifo.handler=23,queue=1,port=16020]
> regionserver.StoreScanner: failed to switch to stream read
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.getScannersForStoreFiles(StoreFileScanner.java:133)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanners(HStore.java:1221)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.trySwitchToStreamRead(StoreScanner.java:997)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.shipped(StoreScanner.java:1134)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.shipped(KeyValueHeap.java:445)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.shipped(HRegion.java:6459)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices$RegionScannerShippedCallBack.run(RSRpcServices.java:339)
> at
> org.apache.hadoop.hbase.ipc.ServerCall.setResponse(ServerCall.java:252)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:166)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258)
> {code}
> I have a working patch fixing this problem. Will do some more testing and try
> to upload the patch after I write a test case for this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-17125) Inconsistent result when use filter to read data
[
https://issues.apache.org/jira/browse/HBASE-17125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059123#comment-16059123
]
Hadoop QA commented on HBASE-17125:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 17m 47s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 32s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 9s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 1s
{color} | {color:red} hbase-client in master has 4 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 44s
{color} | {color:red} hbase-server in master has 10 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
1s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 16s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha3. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m 26s
{color} | {color:red} hbase-server generated 1 new + 10 unchanged - 0 fixed =
11 total (was 10) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 20s
{color} | {color:red} hbase-client generated 8 new + 1 unchanged - 0 fixed = 9
total (was 1) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 48s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 22m 1s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 95m 19s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
| | Should
org.apache.hadoop.hbase.regionserver.querymatcher.UserScanQueryMatcher$SpecifiedNumVersionsColumnFilter
be a _static_ inner class? At UserScanQueryMatcher.java:inner class? At
UserScanQueryMatcher.java:[lines 272-299] |
| Failed junit tests | hadoop.hbase.filter.TestFilter |
| | hadoop.hbase.io.encoding.TestSeekBeforeWithReverseScan |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.13.1 Server=1.13.1 Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12874037/HBASE-17125.master.013.patch
|
| JIRA Issue | HBASE-17125 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux f
[jira] [Commented] (HBASE-18230) Generated LICENSE file includes unsubstituted Velocity variables
[
https://issues.apache.org/jira/browse/HBASE-18230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059307#comment-16059307
]
Sean Busbey commented on HBASE-18230:
-
addendum is fine. curious about the miss, since I thought I had grepped for
"comments" after running through this.
> Generated LICENSE file includes unsubstituted Velocity variables
>
>
> Key: HBASE-18230
> URL: https://issues.apache.org/jira/browse/HBASE-18230
> Project: HBase
> Issue Type: Bug
> Components: build
>Affects Versions: 2.0.0-alpha-1
>Reporter: Mike Drob
>Assignee: Mike Drob
> Fix For: 3.0.0, 1.4.0, 1.3.2, 1.2.7, 2.0.0-alpha-2, 1.1.12
>
> Attachments: HBASE-18230.addendum.patch, HBASE-18230.patch
>
>
> From the release vote:
> {quote}
> we have a ton of places where we have velocity variables instead of
> copyright years, but IIRC that's a problem on branch-1 right now too.
> {quote}
> This is referring to lines like these:
> {noformat}
> * javax.annotation-api, ${dep.licenses[0].comments}
> * javax.servlet-api, ${dep.licenses[0].comments}
> * jetty-schemas, ${dep.licenses[0].comments}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Reopened] (HBASE-18230) Generated LICENSE file includes unsubstituted Velocity variables
[
https://issues.apache.org/jira/browse/HBASE-18230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey reopened HBASE-18230:
-
> Generated LICENSE file includes unsubstituted Velocity variables
>
>
> Key: HBASE-18230
> URL: https://issues.apache.org/jira/browse/HBASE-18230
> Project: HBase
> Issue Type: Bug
> Components: build
>Affects Versions: 2.0.0-alpha-1
>Reporter: Mike Drob
>Assignee: Mike Drob
> Fix For: 3.0.0, 1.4.0, 1.3.2, 1.2.7, 2.0.0-alpha-2, 1.1.12
>
> Attachments: HBASE-18230.addendum.patch, HBASE-18230.patch
>
>
> From the release vote:
> {quote}
> we have a ton of places where we have velocity variables instead of
> copyright years, but IIRC that's a problem on branch-1 right now too.
> {quote}
> This is referring to lines like these:
> {noformat}
> * javax.annotation-api, ${dep.licenses[0].comments}
> * javax.servlet-api, ${dep.licenses[0].comments}
> * jetty-schemas, ${dep.licenses[0].comments}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-16868) Add a replicate_all flag to avoid misuse the namespaces and table-cfs config of replication peer
[
https://issues.apache.org/jira/browse/HBASE-16868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059439#comment-16059439
]
Hadoop QA commented on HBASE-16868:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 22s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} rubocop {color} | {color:blue} 0m 0s
{color} | {color:blue} rubocop was not available. {color} |
| {color:blue}0{color} | {color:blue} ruby-lint {color} | {color:blue} 0m 1s
{color} | {color:blue} Ruby-lint was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 9 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
35s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 40s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 27m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
19s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 45s
{color} | {color:red} hbase-protocol-shaded in master has 27 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 49s
{color} | {color:red} hbase-client in master has 4 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 5m 37s
{color} | {color:red} hbase-server in master has 10 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 54s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 3m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 27m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
27s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
60m 16s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha3. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 2m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 13m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 3s
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 35s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 33m 45s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 15m 32s {color}
| {color:red} hbase-shell in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
56s {color} | {color:green} The patch
[jira] [Commented] (HBASE-17125) Inconsistent result when use filter to read data
[
https://issues.apache.org/jira/browse/HBASE-17125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059450#comment-16059450
]
Ted Yu commented on HBASE-17125:
My aggregate patch v11 has defect in that it doesn't handle
ExplicitColumnTracker#reset() correctly.
ColumnCount should have slack and retract() as well.
ExplicitColumnTracker#retract() delegates to ColumnCount#retract().
When ExplicitColumnTracker#reset() is called, it resets the slack in each
ColumnCount to the initial value.
I have run test suite with updated aggregate patch which passed (other than the
tests flagged by flaky test dashboard).
> Inconsistent result when use filter to read data
>
>
> Key: HBASE-17125
> URL: https://issues.apache.org/jira/browse/HBASE-17125
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: example.diff, HBASE-17125.master.001.patch,
> HBASE-17125.master.002.patch, HBASE-17125.master.002.patch,
> HBASE-17125.master.003.patch, HBASE-17125.master.004.patch,
> HBASE-17125.master.005.patch, HBASE-17125.master.006.patch,
> HBASE-17125.master.007.patch, HBASE-17125.master.008.patch,
> HBASE-17125.master.009.patch, HBASE-17125.master.009.patch,
> HBASE-17125.master.010.patch, HBASE-17125.master.011.patch,
> HBASE-17125.master.011.patch, HBASE-17125.master.012.patch,
> HBASE-17125.master.013.patch, HBASE-17125.master.checkReturnedVersions.patch,
> HBASE-17125.master.no-specified-filter.patch
>
>
> Assume a cloumn's max versions is 3, then we write 4 versions of this column.
> The oldest version doesn't remove immediately. But from the user view, the
> oldest version has gone. When user use a filter to query, if the filter skip
> a new version, then the oldest version will be seen again. But after compact
> the region, then the oldest version will never been seen. So it is weird for
> user. The query will get inconsistent result before and after region
> compaction.
> The reason is matchColumn method of UserScanQueryMatcher. It first check the
> cell by filter, then check the number of versions needed. So if the filter
> skip the new version, then the oldest version will be seen again when it is
> not removed.
> Have a discussion offline with [~Apache9] and [~fenghh], now we have two
> solution for this problem. The first idea is check the number of versions
> first, then check the cell by filter. As the comment of setFilter, the filter
> is called after all tests for ttl, column match, deletes and max versions
> have been run.
> {code}
> /**
>* Apply the specified server-side filter when performing the Query.
>* Only {@link Filter#filterKeyValue(Cell)} is called AFTER all tests
>* for ttl, column match, deletes and max versions have been run.
>* @param filter filter to run on the server
>* @return this for invocation chaining
>*/
> public Query setFilter(Filter filter) {
> this.filter = filter;
> return this;
> }
> {code}
> But this idea has another problem, if a column's max version is 5 and the
> user query only need 3 versions. It first check the version's number, then
> check the cell by filter. So the cells number of the result may less than 3.
> But there are 2 versions which don't read anymore.
> So the second idea has three steps.
> 1. check by the max versions of this column
> 2. check the kv by filter
> 3. check the versions which user need.
> But this will lead the ScanQueryMatcher more complicated. And this will break
> the javadoc of Query.setFilter.
> Now we don't have a final solution for this problem. Suggestions are welcomed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18206) Organize supplemental-models.xml
[ https://issues.apache.org/jira/browse/HBASE-18206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated HBASE-18206: -- Priority: Minor (was: Major) > Organize supplemental-models.xml > > > Key: HBASE-18206 > URL: https://issues.apache.org/jira/browse/HBASE-18206 > Project: HBase > Issue Type: Improvement > Components: dependencies >Reporter: Mike Drob >Priority: Minor > Labels: beginner > > At one point, the supplemental-models.xml file was organized by module and > had comments on what brings in each dependency. > Since then, I think it has drifted away do to folks adding in dependencies in > any which place there. It's probably more maintainable to simply alphabetize > by group. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18259) HBase book link to "beginner" issues includes resolved issues
Mike Drob created HBASE-18259: - Summary: HBase book link to "beginner" issues includes resolved issues Key: HBASE-18259 URL: https://issues.apache.org/jira/browse/HBASE-18259 Project: HBase Issue Type: Bug Components: documentation Reporter: Mike Drob The link at http://hbase.apache.org/book.html#getting.involved for beginner issues is https://issues.apache.org/jira/issues/?jql=project%20%3D%20HBASE%20AND%20labels%20in%20(beginner) but this includes resolved issues as well, which is not useful to folks looking for new issues to cut their teeth on. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18195) Remove redundant single quote from start message for HMaster and HRegionServer
[
https://issues.apache.org/jira/browse/HBASE-18195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Drob updated HBASE-18195:
--
Labels: beginner (was: beginners)
> Remove redundant single quote from start message for HMaster and HRegionServer
> --
>
> Key: HBASE-18195
> URL: https://issues.apache.org/jira/browse/HBASE-18195
> Project: HBase
> Issue Type: Bug
> Components: master, regionserver
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
>Priority: Minor
> Labels: beginner
> Fix For: 2.0.0, 3.0.0
>
> Attachments: HBASE-18195.master.001.patch
>
>
> Message in the log shows up as:
> {code}
> INFO [main] master.HMaster: STARTING service 'HMaster
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18206) Organize supplemental-models.xml
[ https://issues.apache.org/jira/browse/HBASE-18206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059467#comment-16059467 ] Mike Drob commented on HBASE-18206: --- This should likely wait until HBASE-18202 lands to make both tasks easier. > Organize supplemental-models.xml > > > Key: HBASE-18206 > URL: https://issues.apache.org/jira/browse/HBASE-18206 > Project: HBase > Issue Type: Improvement > Components: dependencies >Reporter: Mike Drob >Priority: Minor > Labels: beginner > > At one point, the supplemental-models.xml file was organized by module and > had comments on what brings in each dependency. > Since then, I think it has drifted away do to folks adding in dependencies in > any which place there. It's probably more maintainable to simply alphabetize > by group. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18260) Address new license dependencies from hadoop3-alpha4
Mike Drob created HBASE-18260: - Summary: Address new license dependencies from hadoop3-alpha4 Key: HBASE-18260 URL: https://issues.apache.org/jira/browse/HBASE-18260 Project: HBase Issue Type: Bug Components: dependencies Reporter: Mike Drob hadoop3-alpha4 adds at least one new dependency that requires an update to our supplemental-models.xml use this jira to track the changes for after that is released. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18260) Address new license dependencies from hadoop3-alpha4
[ https://issues.apache.org/jira/browse/HBASE-18260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob updated HBASE-18260: -- Attachment: HBASE-18260.patch Patch that adds stax2-api dependency as BSD licensed. > Address new license dependencies from hadoop3-alpha4 > > > Key: HBASE-18260 > URL: https://issues.apache.org/jira/browse/HBASE-18260 > Project: HBase > Issue Type: Bug > Components: dependencies >Reporter: Mike Drob > Attachments: HBASE-18260.patch > > > hadoop3-alpha4 adds at least one new dependency that requires an update to > our supplemental-models.xml > use this jira to track the changes for after that is released. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-14135) HBase Backup/Restore Phase 3: Merge backup images
[
https://issues.apache.org/jira/browse/HBASE-14135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059512#comment-16059512
]
Ted Yu commented on HBASE-14135:
Since a lot of effort has gone into fault tolerance, I think each backup
improvement should have builtin fault tolerance design.
> HBase Backup/Restore Phase 3: Merge backup images
> -
>
> Key: HBASE-14135
> URL: https://issues.apache.org/jira/browse/HBASE-14135
> Project: HBase
> Issue Type: New Feature
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Blocker
> Labels: backup
> Fix For: HBASE-7912
>
> Attachments: HBASE-14135-v1.patch
>
>
> User can merge incremental backup images into single incremental backup image.
> # Merge supports only incremental images
> # Merge supports only images for the same backup destinations
> Command:
> {code}
> hbase backup merge image1,image2,..imageK
> {code}
> Example:
> {code}
> hbase backup merge backup_143126764557,backup_143126764456
> {code}
> When operation is complete, only the most recent backup image will be kept
> (in above example - backup_143126764557) as a merged backup image, all other
> images will be deleted from both: file system and backup system tables,
> corresponding backup manifest for the merged backup image will be updated to
> remove dependencies from deleted images. Merged backup image will contains
> all the data from original image and from deleted images.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18245) Handle failed server in RpcClient
[ https://issues.apache.org/jira/browse/HBASE-18245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-18245: --- Attachment: 18245.v3.txt Patch v3 adds FailedServerException which is thrown from RpcClient::GetConnection() > Handle failed server in RpcClient > - > > Key: HBASE-18245 > URL: https://issues.apache.org/jira/browse/HBASE-18245 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu > Attachments: 18245.v1.txt, 18245.v2.txt, 18245.v3.txt > > > This task is to add support for failed server in RpcClient::GetConnection(). > FailedServers Java class would be ported to C++. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18245) Handle failed server in RpcClient
[ https://issues.apache.org/jira/browse/HBASE-18245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-18245: --- Attachment: (was: 18245.v3.txt) > Handle failed server in RpcClient > - > > Key: HBASE-18245 > URL: https://issues.apache.org/jira/browse/HBASE-18245 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu > Attachments: 18245.v1.txt, 18245.v2.txt, 18245.v3.txt > > > This task is to add support for failed server in RpcClient::GetConnection(). > FailedServers Java class would be ported to C++. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18245) Handle failed server in RpcClient
[ https://issues.apache.org/jira/browse/HBASE-18245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-18245: --- Attachment: 18245.v3.txt > Handle failed server in RpcClient > - > > Key: HBASE-18245 > URL: https://issues.apache.org/jira/browse/HBASE-18245 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu > Attachments: 18245.v1.txt, 18245.v2.txt, 18245.v3.txt > > > This task is to add support for failed server in RpcClient::GetConnection(). > FailedServers Java class would be ported to C++. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HBASE-18260) Address new license dependencies from hadoop3-alpha4
[ https://issues.apache.org/jira/browse/HBASE-18260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike Drob reassigned HBASE-18260: - Assignee: Mike Drob > Address new license dependencies from hadoop3-alpha4 > > > Key: HBASE-18260 > URL: https://issues.apache.org/jira/browse/HBASE-18260 > Project: HBase > Issue Type: Bug > Components: dependencies >Reporter: Mike Drob >Assignee: Mike Drob > Attachments: HBASE-18260.patch > > > hadoop3-alpha4 adds at least one new dependency that requires an update to > our supplemental-models.xml > use this jira to track the changes for after that is released. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18161) Incremental Load support for Multiple-Table HFileOutputFormat
[
https://issues.apache.org/jira/browse/HBASE-18161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059638#comment-16059638
]
Ted Yu commented on HBASE-18161:
I ran TestSnapshotManifest with patch v9 which passed.
> Incremental Load support for Multiple-Table HFileOutputFormat
> -
>
> Key: HBASE-18161
> URL: https://issues.apache.org/jira/browse/HBASE-18161
> Project: HBase
> Issue Type: New Feature
>Reporter: Densel Santhmayor
>Priority: Minor
> Attachments: MultiHFileOutputFormatSupport_HBASE_18161.patch,
> MultiHFileOutputFormatSupport_HBASE_18161_v2.patch,
> MultiHFileOutputFormatSupport_HBASE_18161_v3.patch,
> MultiHFileOutputFormatSupport_HBASE_18161_v4.patch,
> MultiHFileOutputFormatSupport_HBASE_18161_v5.patch,
> MultiHFileOutputFormatSupport_HBASE_18161_v6.patch,
> MultiHFileOutputFormatSupport_HBASE_18161_v7.patch,
> MultiHFileOutputFormatSupport_HBASE_18161_v8.patch,
> MultiHFileOutputFormatSupport_HBASE_18161_v9.patch
>
>
> h2. Introduction
> MapReduce currently supports the ability to write HBase records in bulk to
> HFiles for a single table. The file(s) can then be uploaded to the relevant
> RegionServers information with reasonable latency. This feature is useful to
> make a large set of data available for queries at the same time as well as
> provides a way to efficiently process very large input into HBase without
> affecting query latencies.
> There is, however, no support to write variations of the same record key to
> HFiles belonging to multiple HBase tables from within the same MapReduce job.
>
> h2. Goal
> The goal of this JIRA is to extend HFileOutputFormat2 to support writing to
> HFiles for different tables within the same MapReduce job while single-table
> HFile features backwards-compatible.
> For our use case, we needed to write a record key to a smaller HBase table
> for quicker access, and the same record key with a date appended to a larger
> table for longer term storage with chronological access. Each of these tables
> would have different TTL and other settings to support their respective
> access patterns. We also needed to be able to bulk write records to multiple
> tables with different subsets of very large input as efficiently as possible.
> Rather than run the MapReduce job multiple times (one for each table or
> record structure), it would be useful to be able to parse the input a single
> time and write to multiple tables simultaneously.
> Additionally, we'd like to maintain backwards compatibility with the existing
> heavily-used HFileOutputFormat2 interface to allow benefits such as locality
> sensitivity (that was introduced long after we implemented support for
> multiple tables) to support both single table and multi table hfile writes.
> h2. Proposal
> * Backwards compatibility for existing single table support in
> HFileOutputFormat2 will be maintained and in this case, mappers will need to
> emit the table rowkey as before. However, a new class -
> MultiHFileOutputFormat - will provide a helper function to generate a rowkey
> for mappers that prefixes the desired tablename to the existing rowkey as
> well as provides configureIncrementalLoad support for multiple tables.
> * HFileOutputFormat2 will be updated in the following way:
> ** configureIncrementalLoad will now accept multiple table descriptor and
> region locator pairs, analogous to the single pair currently accepted by
> HFileOutputFormat2.
> ** Compression, Block Size, Bloom Type and Datablock settings PER column
> family that are set in the Configuration object are now indexed and retrieved
> by tablename AND column family
> ** getRegionStartKeys will now support multiple regionlocators and calculate
> split points and therefore partitions collectively for all tables. Similarly,
> now the eventual number of Reducers will be equal to the total number of
> partitions across all tables.
> ** The RecordWriter class will be able to process rowkeys either with or
> without the tablename prepended depending on how configureIncrementalLoad was
> configured with MultiHFileOutputFormat or HFileOutputFormat2.
> * The use of MultiHFileOutputFormat will write the output into HFiles which
> will match the output format of HFileOutputFormat2. However, while the
> default use case will keep the existing directory structure with column
> family name as the directory and HFiles within that directory, in the case of
> MultiHFileOutputFormat, it will output HFiles in the output directory with
> the following relative paths:
> {noformat}
> --table1
>--family1
> --HFiles
> --table2
>--family1
>--family2
> --HFiles
> {noformat}
> This aims to be a comprehensive solution to the orig
[jira] [Commented] (HBASE-18254) ServerCrashProcedure checks and waits for meta initialized, instead should check and wait for meta loaded
[ https://issues.apache.org/jira/browse/HBASE-18254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059655#comment-16059655 ] Umesh Agashe commented on HBASE-18254: -- Thanks [~stack] for reviewing and committing the changes. > ServerCrashProcedure checks and waits for meta initialized, instead should > check and wait for meta loaded > - > > Key: HBASE-18254 > URL: https://issues.apache.org/jira/browse/HBASE-18254 > Project: HBase > Issue Type: Bug > Components: amv2 >Affects Versions: 2.0.0-alpha-1 >Reporter: Umesh Agashe >Assignee: Umesh Agashe > Fix For: 2.0.0 > > Attachments: HBASE-18254.master.001.patch > > > After enabling test > hbase.master.procedure.TestServerCrashProcedure#testRecoveryAndDoubleExecutionOnRsWithMeta, > this bug is found in ServerCrashProcedure. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18257) Cover exposed guava Service Interface in ReplicationEndpoint
[ https://issues.apache.org/jira/browse/HBASE-18257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059732#comment-16059732 ] stack commented on HBASE-18257: --- We should try and do this for 2.0.0. > Cover exposed guava Service Interface in ReplicationEndpoint > > > Key: HBASE-18257 > URL: https://issues.apache.org/jira/browse/HBASE-18257 > Project: HBase > Issue Type: Task >Reporter: stack > Fix For: 2.0.0 > > > ReplicationEndpoint implements guava Service. It should instead implement an > hbase Service class that covers over guava; lets not have third-party > Interfaces/classes exposed as part of our API. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18257) Cover exposed guava Service Interface in ReplicationEndpoint
[ https://issues.apache.org/jira/browse/HBASE-18257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-18257: -- Fix Version/s: 2.0.0 > Cover exposed guava Service Interface in ReplicationEndpoint > > > Key: HBASE-18257 > URL: https://issues.apache.org/jira/browse/HBASE-18257 > Project: HBase > Issue Type: Task >Reporter: stack > Fix For: 2.0.0 > > > ReplicationEndpoint implements guava Service. It should instead implement an > hbase Service class that covers over guava; lets not have third-party > Interfaces/classes exposed as part of our API. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18260) Address new license dependencies from hadoop3-alpha4
[ https://issues.apache.org/jira/browse/HBASE-18260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059780#comment-16059780 ] Sean Busbey commented on HBASE-18260: - this is for the current 3.0.0-alpha-4 snapshot, right? > Address new license dependencies from hadoop3-alpha4 > > > Key: HBASE-18260 > URL: https://issues.apache.org/jira/browse/HBASE-18260 > Project: HBase > Issue Type: Bug > Components: dependencies >Reporter: Mike Drob >Assignee: Mike Drob > Attachments: HBASE-18260.patch > > > hadoop3-alpha4 adds at least one new dependency that requires an update to > our supplemental-models.xml > use this jira to track the changes for after that is released. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18242) ClientSideRegionScanner#openRegion, deleted files under regionEncodeName/.tmp
[ https://issues.apache.org/jira/browse/HBASE-18242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059808#comment-16059808 ] Enis Soztutar commented on HBASE-18242: --- Is this related to mounting snapshots without restoring them? HRegion already has a "read-only" mode which does not do any action (like replaying recovered edits, removing tmp files etc) that we can make use of. > ClientSideRegionScanner#openRegion, deleted files under regionEncodeName/.tmp > - > > Key: HBASE-18242 > URL: https://issues.apache.org/jira/browse/HBASE-18242 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.10, 1.3.0 >Reporter: Bo Cui >Priority: Minor > > There are two ways to open region: > 1, RS#openRegion:flush, compact, split, merge, scan, get, etc. > 2, Client#openRegion: only scan, get > Either way, all files under > /hbase/data/default/TableName/RegionEncodeName/.tmp will be deleted when > region is initialized > For method 2, this should not happen: the client opening region can only read > data; there is no possibility of compaction or flush; if RS is processing > compact, and client removes the tmp file, the RS needs to be re executed. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18255) Time-Delayed HBase Performance Degradation with Java 7
[
https://issues.apache.org/jira/browse/HBASE-18255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059825#comment-16059825
]
Vladimir Rodionov commented on HBASE-18255:
---
{quote}
Should we specifically check if the JDK is 7 and then apply the fix?
{quote}
We set code cache size to 256MB. This works for JDK7.
In JDK 8 256MB is the default code cache size, so in case of JDK 8 the patch is
noop (no-operation).
> Time-Delayed HBase Performance Degradation with Java 7
> --
>
> Key: HBASE-18255
> URL: https://issues.apache.org/jira/browse/HBASE-18255
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.1, 1.2.6, 1.1.11
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Fix For: 1.3.2, 1.2.7, 1.1.12
>
> Attachments: HBASE-18255-branch-1.x.v1.patch
>
>
> The good summary of the issue and provided resolution can be found in this
> article:
> https://community.hortonworks.com/articles/105802/time-delayed-hbase-performance-degradation-with-ja.html
> In a few words, due to internal JVM 7 bug (which has been addressed only in
> Java 8), HotSpot code cache can become full and after that ALL JIT
> compilations get suspended indefinitely. The default value for code cache
> size in JVM 7 is quite low: 48MB. It is recommended to increase this value at
> least to 256MB (default in JVM 8).
> This BUG affects only 1.x
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18260) Address new license dependencies from hadoop3-alpha4
[
https://issues.apache.org/jira/browse/HBASE-18260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059838#comment-16059838
]
Mike Drob commented on HBASE-18260:
---
yea, they publish snapshots to maven repo at
https://repository.apache.org/content/groups/snapshots and that is what I built
against {{-Dhadoop-three.version=3.0.0-alpha4-SNAPSHOT}}
> Address new license dependencies from hadoop3-alpha4
>
>
> Key: HBASE-18260
> URL: https://issues.apache.org/jira/browse/HBASE-18260
> Project: HBase
> Issue Type: Bug
> Components: dependencies
>Reporter: Mike Drob
>Assignee: Mike Drob
> Attachments: HBASE-18260.patch
>
>
> hadoop3-alpha4 adds at least one new dependency that requires an update to
> our supplemental-models.xml
> use this jira to track the changes for after that is released.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18255) Time-Delayed HBase Performance Degradation with Java 7
[ https://issues.apache.org/jira/browse/HBASE-18255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059879#comment-16059879 ] Enis Soztutar commented on HBASE-18255: --- +1. It took a VERY long time to debug this. > Time-Delayed HBase Performance Degradation with Java 7 > -- > > Key: HBASE-18255 > URL: https://issues.apache.org/jira/browse/HBASE-18255 > Project: HBase > Issue Type: Bug >Affects Versions: 1.3.1, 1.2.6, 1.1.11 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 1.3.2, 1.2.7, 1.1.12 > > Attachments: HBASE-18255-branch-1.x.v1.patch > > > The good summary of the issue and provided resolution can be found in this > article: > https://community.hortonworks.com/articles/105802/time-delayed-hbase-performance-degradation-with-ja.html > In a few words, due to internal JVM 7 bug (which has been addressed only in > Java 8), HotSpot code cache can become full and after that ALL JIT > compilations get suspended indefinitely. The default value for code cache > size in JVM 7 is quite low: 48MB. It is recommended to increase this value at > least to 256MB (default in JVM 8). > This BUG affects only 1.x -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18257) Cover exposed guava Service Interface in ReplicationEndpoint
[ https://issues.apache.org/jira/browse/HBASE-18257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059880#comment-16059880 ] Enis Soztutar commented on HBASE-18257: --- Indeed we talked about this. HBase already has a Service class. I think we need to unify that and Guava's service. > Cover exposed guava Service Interface in ReplicationEndpoint > > > Key: HBASE-18257 > URL: https://issues.apache.org/jira/browse/HBASE-18257 > Project: HBase > Issue Type: Task >Reporter: stack > Fix For: 2.0.0 > > > ReplicationEndpoint implements guava Service. It should instead implement an > hbase Service class that covers over guava; lets not have third-party > Interfaces/classes exposed as part of our API. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18257) Cover exposed guava Service Interface in ReplicationEndpoint
[ https://issues.apache.org/jira/browse/HBASE-18257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059897#comment-16059897 ] stack commented on HBASE-18257: --- Thanks for reminder [~enis] > Cover exposed guava Service Interface in ReplicationEndpoint > > > Key: HBASE-18257 > URL: https://issues.apache.org/jira/browse/HBASE-18257 > Project: HBase > Issue Type: Task >Reporter: stack > Fix For: 2.0.0 > > > ReplicationEndpoint implements guava Service. It should instead implement an > hbase Service class that covers over guava; lets not have third-party > Interfaces/classes exposed as part of our API. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18175) Add hbase-spark integration test into hbase-it
[ https://issues.apache.org/jira/browse/HBASE-18175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059918#comment-16059918 ] Mike Drob commented on HBASE-18175: --- [~easyliangjob], I like the approach of putting hbase-spark-it under hbase-spark module. What kind of problems are you having with the patch? Anything that I can help with? > Add hbase-spark integration test into hbase-it > -- > > Key: HBASE-18175 > URL: https://issues.apache.org/jira/browse/HBASE-18175 > Project: HBase > Issue Type: Test > Components: spark >Reporter: Yi Liang >Assignee: Yi Liang >Priority: Critical > Fix For: 2.0.0 > > Attachments: hbase-18175-v1.patch > > > After HBASE-17574, all test under hbase-spark are regarded as unit test, and > this jira will add integration test of hbase-spark into hbase-it. This patch > run same tests as mapreduce.IntegrationTestBulkLoad, just change mapreduce to > spark. > test in Maven: > mvn verify -Dit.test=IntegrationTestSparkBulkLoad > test on cluster: > spark-submit --class > org.apache.hadoop.hbase.spark.IntegrationTestSparkBulkLoad > HBASE_HOME/lib/hbase-it-2.0.0-SNAPSHOT-tests.jar > -Dhbase.spark.bulkload.chainlength=50 -m slowDeterministic -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18248) Warn if monitored task has been tied up beyond a configurable threshold
[ https://issues.apache.org/jira/browse/HBASE-18248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059928#comment-16059928 ] Andrew Purtell commented on HBASE-18248: Sure, that's fine. I'm off for a few days. Back with an updated patch next week. > Warn if monitored task has been tied up beyond a configurable threshold > --- > > Key: HBASE-18248 > URL: https://issues.apache.org/jira/browse/HBASE-18248 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 3.0.0, 1.4.0, 1.3.2 > > Attachments: HBASE-18248-branch-1.3.patch, > HBASE-18248-branch-1.patch, HBASE-18248-branch-2.patch, HBASE-18248.patch > > > Warn if monitored task has been tied up beyond a configurable threshold. We > especially want to do this for RPC tasks. Use a separate threshold for > warning about stuck RPC tasks versus other types of tasks. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18054) log when we add/remove failed servers in client
[
https://issues.apache.org/jira/browse/HBASE-18054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ali updated HBASE-18054:
Status: Patch Available (was: Open)
> log when we add/remove failed servers in client
> ---
>
> Key: HBASE-18054
> URL: https://issues.apache.org/jira/browse/HBASE-18054
> Project: HBase
> Issue Type: Bug
> Components: Client, Operability
>Affects Versions: 1.3.0
>Reporter: Sean Busbey
>Assignee: Ali
> Attachments: HBASE-18054.patch, HBASE-18054.v2.master.patch,
> HBASE-18054.v3.master.patch, HBASE-18054.v4.master.patch
>
>
> Currently we log if a server is in the failed server list when we go to
> connect to it, but we don't log anything about when the server got into the
> list.
> This means we have to search the log for errors involving the same server
> name that (hopefully) managed to get into the log within
> {{FAILED_SERVER_EXPIRY_KEY}} milliseconds earlier (default 2 seconds).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18054) log when we add/remove failed servers in client
[
https://issues.apache.org/jira/browse/HBASE-18054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ali updated HBASE-18054:
Status: Open (was: Patch Available)
> log when we add/remove failed servers in client
> ---
>
> Key: HBASE-18054
> URL: https://issues.apache.org/jira/browse/HBASE-18054
> Project: HBase
> Issue Type: Bug
> Components: Client, Operability
>Affects Versions: 1.3.0
>Reporter: Sean Busbey
>Assignee: Ali
> Attachments: HBASE-18054.patch, HBASE-18054.v2.master.patch,
> HBASE-18054.v3.master.patch, HBASE-18054.v4.master.patch
>
>
> Currently we log if a server is in the failed server list when we go to
> connect to it, but we don't log anything about when the server got into the
> list.
> This means we have to search the log for errors involving the same server
> name that (hopefully) managed to get into the log within
> {{FAILED_SERVER_EXPIRY_KEY}} milliseconds earlier (default 2 seconds).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18175) Add hbase-spark integration test into hbase-it
[ https://issues.apache.org/jira/browse/HBASE-18175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059959#comment-16059959 ] Yi Liang commented on HBASE-18175: -- Hi Mike, Thanks for asking, i am now waiting for the final layout of hbase-spark module. Since right now, if I add hbase-spark-it under hbase spark, it has below layout. hbase-spark ---src ---pom ---hbase-spark-it ---src ---pom And if we have layout above, there is no hbase-spark.jar created in hbase-spark/target after I mvn package, so the unit test and it test can not find related jars. if we have something below, actually a place to store hbase-spark.jar, like(hbase-spark-2.0/target). The tests then can run successfully. hbase-spark ---src ---pom ---hbase-spark-2.0 ---hbase-spark-it ---src ---pom So I am still waiting for the final layout, then I can decide to have a new hbase-spark-it module or just directly put it into hbase-spark module > Add hbase-spark integration test into hbase-it > -- > > Key: HBASE-18175 > URL: https://issues.apache.org/jira/browse/HBASE-18175 > Project: HBase > Issue Type: Test > Components: spark >Reporter: Yi Liang >Assignee: Yi Liang >Priority: Critical > Fix For: 2.0.0 > > Attachments: hbase-18175-v1.patch > > > After HBASE-17574, all test under hbase-spark are regarded as unit test, and > this jira will add integration test of hbase-spark into hbase-it. This patch > run same tests as mapreduce.IntegrationTestBulkLoad, just change mapreduce to > spark. > test in Maven: > mvn verify -Dit.test=IntegrationTestSparkBulkLoad > test on cluster: > spark-submit --class > org.apache.hadoop.hbase.spark.IntegrationTestSparkBulkLoad > HBASE_HOME/lib/hbase-it-2.0.0-SNAPSHOT-tests.jar > -Dhbase.spark.bulkload.chainlength=50 -m slowDeterministic -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18023) Log multi-* requests for more than threshold number of rows
[ https://issues.apache.org/jira/browse/HBASE-18023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Harju updated HBASE-18023: Attachment: HBASE-18023.master.004.patch > Log multi-* requests for more than threshold number of rows > --- > > Key: HBASE-18023 > URL: https://issues.apache.org/jira/browse/HBASE-18023 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Clay B. >Assignee: David Harju >Priority: Minor > Attachments: HBASE-18023.master.001.patch, > HBASE-18023.master.002.patch, HBASE-18023.master.003.patch, > HBASE-18023.master.004.patch > > > Today, if a user happens to do something like a large multi-put, they can get > through request throttling (e.g. it is one request) but still crash a region > server with a garbage storm. We have seen regionservers hit this issue and it > is silent and deadly. The RS will report nothing more than a mysterious > garbage collection and exit out. > Ideally, we could report a large multi-* request before starting it, in case > it happens to be deadly. Knowing the client, user and how many rows are > affected would be a good start to tracking down painful users. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18023) Log multi-* requests for more than threshold number of rows
[ https://issues.apache.org/jira/browse/HBASE-18023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Harju updated HBASE-18023: Status: Open (was: Patch Available) > Log multi-* requests for more than threshold number of rows > --- > > Key: HBASE-18023 > URL: https://issues.apache.org/jira/browse/HBASE-18023 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Clay B. >Assignee: David Harju >Priority: Minor > Attachments: HBASE-18023.master.001.patch, > HBASE-18023.master.002.patch, HBASE-18023.master.003.patch, > HBASE-18023.master.004.patch > > > Today, if a user happens to do something like a large multi-put, they can get > through request throttling (e.g. it is one request) but still crash a region > server with a garbage storm. We have seen regionservers hit this issue and it > is silent and deadly. The RS will report nothing more than a mysterious > garbage collection and exit out. > Ideally, we could report a large multi-* request before starting it, in case > it happens to be deadly. Knowing the client, user and how many rows are > affected would be a good start to tracking down painful users. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18023) Log multi-* requests for more than threshold number of rows
[ https://issues.apache.org/jira/browse/HBASE-18023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Harju updated HBASE-18023: Status: Patch Available (was: Open) Removed TestHRegion.java changes > Log multi-* requests for more than threshold number of rows > --- > > Key: HBASE-18023 > URL: https://issues.apache.org/jira/browse/HBASE-18023 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Clay B. >Assignee: David Harju >Priority: Minor > Attachments: HBASE-18023.master.001.patch, > HBASE-18023.master.002.patch, HBASE-18023.master.003.patch, > HBASE-18023.master.004.patch > > > Today, if a user happens to do something like a large multi-put, they can get > through request throttling (e.g. it is one request) but still crash a region > server with a garbage storm. We have seen regionservers hit this issue and it > is silent and deadly. The RS will report nothing more than a mysterious > garbage collection and exit out. > Ideally, we could report a large multi-* request before starting it, in case > it happens to be deadly. Knowing the client, user and how many rows are > affected would be a good start to tracking down painful users. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18023) Log multi-* requests for more than threshold number of rows
[ https://issues.apache.org/jira/browse/HBASE-18023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059971#comment-16059971 ] David Harju commented on HBASE-18023: - [~elserj], oops, sorry I missed that. Removed those changes. > Log multi-* requests for more than threshold number of rows > --- > > Key: HBASE-18023 > URL: https://issues.apache.org/jira/browse/HBASE-18023 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Clay B. >Assignee: David Harju >Priority: Minor > Attachments: HBASE-18023.master.001.patch, > HBASE-18023.master.002.patch, HBASE-18023.master.003.patch, > HBASE-18023.master.004.patch > > > Today, if a user happens to do something like a large multi-put, they can get > through request throttling (e.g. it is one request) but still crash a region > server with a garbage storm. We have seen regionservers hit this issue and it > is silent and deadly. The RS will report nothing more than a mysterious > garbage collection and exit out. > Ideally, we could report a large multi-* request before starting it, in case > it happens to be deadly. Knowing the client, user and how many rows are > affected would be a good start to tracking down painful users. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HBASE-18261) [AMv2] Create new RecoverMetaProcedure and use it from ServerCrashProcedure and HMaster.finishActiveMasterInitialization()
Umesh Agashe created HBASE-18261: Summary: [AMv2] Create new RecoverMetaProcedure and use it from ServerCrashProcedure and HMaster.finishActiveMasterInitialization() Key: HBASE-18261 URL: https://issues.apache.org/jira/browse/HBASE-18261 Project: HBase Issue Type: Improvement Components: amv2 Affects Versions: 2.0.0-alpha-1 Reporter: Umesh Agashe Assignee: Umesh Agashe Fix For: 2.0.0-alpha-2 When unit test hbase.master.procedure.TestServerCrashProcedure#testRecoveryAndDoubleExecutionOnRsWithMeta() is enabled and run several times, it fails intermittently. Cause is meta recovery is done at two different places: * ServerCrashProcedure.processMeta() * HMaster.finishActiveMasterInitialization() and its not coordinated. When HMaster.finishActiveMasterInitialization() gets to submit splitMetaLog() first and while its running call from ServerCrashProcedure.processMeta() fails causing step to be retried again in a loop. When ServerCrashProcedure.processMeta() submits splitMetaLog after splitMetaLog from HMaster.finishActiveMasterInitialization() is finished, success is returned without doing any work. But if ServerCrashProcedure.processMeta() submits splitMetaLog request and while its going HMaster.finishActiveMasterInitialization() submits it test fails with exception. [~stack] and I discussed the possible solution: Create RecoverMetaProcedure and call it where required. Procedure framework provides mutual exclusion and requires idempotence, which should fix the problem. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18218) List replication peers for the cluster
[ https://issues.apache.org/jira/browse/HBASE-18218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ali updated HBASE-18218: Attachment: HBASE-18218.v3.master.patch > List replication peers for the cluster > -- > > Key: HBASE-18218 > URL: https://issues.apache.org/jira/browse/HBASE-18218 > Project: HBase > Issue Type: New Feature >Reporter: Ali >Assignee: Ali > Attachments: HBASE-18218.v1.branch-1.2.patch, > HBASE-18218.v2.branch-1.patch, HBASE-18218.v2.master.patch, > HBASE-18218.v3.master.patch, screenshot-1.png, screenshot-2.png > > > HBase Master page that listed all the replication peers for a cluster, with > their associated metadata -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18218) List replication peers for the cluster
[ https://issues.apache.org/jira/browse/HBASE-18218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ali updated HBASE-18218: Attachment: screenshot-3.png > List replication peers for the cluster > -- > > Key: HBASE-18218 > URL: https://issues.apache.org/jira/browse/HBASE-18218 > Project: HBase > Issue Type: New Feature >Reporter: Ali >Assignee: Ali > Attachments: HBASE-18218.v1.branch-1.2.patch, > HBASE-18218.v2.branch-1.patch, HBASE-18218.v2.master.patch, > HBASE-18218.v3.master.patch, screenshot-1.png, screenshot-2.png, > screenshot-3.png > > > HBase Master page that listed all the replication peers for a cluster, with > their associated metadata -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18054) log when we add/remove failed servers in client
[
https://issues.apache.org/jira/browse/HBASE-18054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060014#comment-16060014
]
Mike Drob commented on HBASE-18054:
---
{code}
+ private static Log LOG = LogFactory.getLog(FailedServers.class);
{code}
Nit: should be declared final.
{code}
- public synchronized void addToFailedServers(InetSocketAddress address) {
+ public synchronized void addToFailedServers(InetSocketAddress address,
Throwable throwable) {
{code}
Should we have multiple methods instead to allow a gradual migration? I guess
it's {{@IA.Private}}, so maybe it doesn't matter.
{code}
+assertEquals(nullException.toString(), "java.lang.NullPointerException");
+assertEquals(loggingEvent.getRenderedMessage(), "Added failed server with
address "
++ addr.toString() + " to list caused by " + nullException.toString());
+assertNotEquals(loggingEvent.getRenderedMessage(), "Added failed server
with address "
++ addr.toString() + " to list caused by " + iOException.toString());
{code}
I don't think we need to verify the behaviour of NPE.toString. And I don't
think we need to check that the message is both equal to and not equal to
things. The equality check strongly implies the non-equality check.
Nit: Argument order for JUnit is (expected, actual), you have it reversed.
I really like the use of Mock here, I'm going to have to keep that in mind for
when I'm doing something similar in the future.
> log when we add/remove failed servers in client
> ---
>
> Key: HBASE-18054
> URL: https://issues.apache.org/jira/browse/HBASE-18054
> Project: HBase
> Issue Type: Bug
> Components: Client, Operability
>Affects Versions: 1.3.0
>Reporter: Sean Busbey
>Assignee: Ali
> Attachments: HBASE-18054.patch, HBASE-18054.v2.master.patch,
> HBASE-18054.v3.master.patch, HBASE-18054.v4.master.patch
>
>
> Currently we log if a server is in the failed server list when we go to
> connect to it, but we don't log anything about when the server got into the
> list.
> This means we have to search the log for errors involving the same server
> name that (hopefully) managed to get into the log within
> {{FAILED_SERVER_EXPIRY_KEY}} milliseconds earlier (default 2 seconds).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18218) List replication peers for the cluster
[ https://issues.apache.org/jira/browse/HBASE-18218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060015#comment-16060015 ] Ali commented on HBASE-18218: - [~yuzhih...@gmail.com] I have attached screenshot-3 which is in case there are too many pairs in config. The columns resize by themselves which kind of seems fine. let me know if you'd prefer fixed column sizes > List replication peers for the cluster > -- > > Key: HBASE-18218 > URL: https://issues.apache.org/jira/browse/HBASE-18218 > Project: HBase > Issue Type: New Feature >Reporter: Ali >Assignee: Ali > Attachments: HBASE-18218.v1.branch-1.2.patch, > HBASE-18218.v2.branch-1.patch, HBASE-18218.v2.master.patch, > HBASE-18218.v3.master.patch, screenshot-1.png, screenshot-2.png, > screenshot-3.png > > > HBase Master page that listed all the replication peers for a cluster, with > their associated metadata -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18218) List replication peers for the cluster
[ https://issues.apache.org/jira/browse/HBASE-18218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ali updated HBASE-18218: Status: Patch Available (was: Open) > List replication peers for the cluster > -- > > Key: HBASE-18218 > URL: https://issues.apache.org/jira/browse/HBASE-18218 > Project: HBase > Issue Type: New Feature >Reporter: Ali >Assignee: Ali > Attachments: HBASE-18218.v1.branch-1.2.patch, > HBASE-18218.v2.branch-1.patch, HBASE-18218.v2.master.patch, > HBASE-18218.v3.master.patch, screenshot-1.png, screenshot-2.png, > screenshot-3.png > > > HBase Master page that listed all the replication peers for a cluster, with > their associated metadata -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18175) Add hbase-spark integration test into hbase-it
[ https://issues.apache.org/jira/browse/HBASE-18175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060030#comment-16060030 ] Mike Drob commented on HBASE-18175: --- Ah, yes, you're right, the parent pom in a multi-module project shouldn't have any output artifacts. Is the simplest thing to let you move forward to add a top-level hbase-spark-it module? And then we can move things around once the mailing list discussion reaches consensus. Alternatively, I would support you doing something like: hbase-spark * pom * hbase-spark-core (this is the current hbase-spark dir) ** pom ** src * hbase-spark-it (this is your new module) ** pom ** src But that might require a lot of shuffling of maven parts that are likely to end up redone later anyway. > Add hbase-spark integration test into hbase-it > -- > > Key: HBASE-18175 > URL: https://issues.apache.org/jira/browse/HBASE-18175 > Project: HBase > Issue Type: Test > Components: spark >Reporter: Yi Liang >Assignee: Yi Liang >Priority: Critical > Fix For: 2.0.0 > > Attachments: hbase-18175-v1.patch > > > After HBASE-17574, all test under hbase-spark are regarded as unit test, and > this jira will add integration test of hbase-spark into hbase-it. This patch > run same tests as mapreduce.IntegrationTestBulkLoad, just change mapreduce to > spark. > test in Maven: > mvn verify -Dit.test=IntegrationTestSparkBulkLoad > test on cluster: > spark-submit --class > org.apache.hadoop.hbase.spark.IntegrationTestSparkBulkLoad > HBASE_HOME/lib/hbase-it-2.0.0-SNAPSHOT-tests.jar > -Dhbase.spark.bulkload.chainlength=50 -m slowDeterministic -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17908) Upgrade guava
[
https://issues.apache.org/jira/browse/HBASE-17908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060061#comment-16060061
]
stack commented on HBASE-17908:
---
As part of this exercise, I purged all includes of guava, even transitives from
hadoop. Here are places where hadoop itself depends on guava so we'll have to
let the transitives come in. Seems tough to do any exclusion here if Hadoop
Configuration depends on guava so I ended-up putting back transitive include by
hadoop-guava of whatever its guava is.
{code}
Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.502 sec <<<
FAILURE! - in org.apache.hadoop.hbase.io.crypto.aes.TestAES
testAlternateRNG(org.apache.hadoop.hbase.io.crypto.aes.TestAES) Time elapsed:
0.252 sec <<< ERROR!
java.lang.NoClassDefFoundError: com/google/common/base/Preconditions
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at
org.apache.hadoop.conf.Configuration$DeprecationDelta.(Configuration.java:328)
at
org.apache.hadoop.conf.Configuration$DeprecationDelta.(Configuration.java:341)
at org.apache.hadoop.conf.Configuration.(Configuration.java:423)
at
org.apache.hadoop.hbase.io.crypto.aes.TestAES.testAlternateRNG(TestAES.java:88)
{code}
Let me post a patch that is severe in its guava eliminations and that has us
using guava 22.0 (shaded).
One thing I think I have learned is that shading, relocations need to always be
the same for the case where a shaded lib itself depends on another shaded lib;
e.g. protobuf utils depends on guava. Currently we have our internal relocation
for protobuf3 which adds the prefix org.apache.hadoop.hbase.shaded but our
thirdparty jar has guava offset by org.apache.hbase.thirdparty.shaded; I think
these need to be the same. Let me try moving protobuf to hbase-thirdparty
project and bring in the protobuf util which uses guava and get that working.
Once that is done, I think we have a pattern going forward for removing from
the guava/netty/protobuf versions mash-up.
> Upgrade guava
> -
>
> Key: HBASE-17908
> URL: https://issues.apache.org/jira/browse/HBASE-17908
> Project: HBase
> Issue Type: Sub-task
> Components: dependencies
>Reporter: Balazs Meszaros
>Assignee: stack
>Priority: Critical
> Fix For: 2.0.0
>
>
> Currently we are using guava 12.0.1, but the latest version is 21.0.
> Upgrading guava is always a hassle because it is not always backward
> compatible with itself.
> Currently I think there are to approaches:
> 1. Upgrade guava to the newest version (21.0) and shade it.
> 2. Upgrade guava to a version which does not break or builds (15.0).
> If we can update it, some dependencies should be removed:
> commons-collections, commons-codec, ...
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18218) List replication peers for the cluster
[ https://issues.apache.org/jira/browse/HBASE-18218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060098#comment-16060098 ] Ted Yu commented on HBASE-18218: Screenshot 3 looks good. > List replication peers for the cluster > -- > > Key: HBASE-18218 > URL: https://issues.apache.org/jira/browse/HBASE-18218 > Project: HBase > Issue Type: New Feature >Reporter: Ali >Assignee: Ali > Attachments: HBASE-18218.v1.branch-1.2.patch, > HBASE-18218.v2.branch-1.patch, HBASE-18218.v2.master.patch, > HBASE-18218.v3.master.patch, screenshot-1.png, screenshot-2.png, > screenshot-3.png > > > HBase Master page that listed all the replication peers for a cluster, with > their associated metadata -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18175) Add hbase-spark integration test into hbase-it
[ https://issues.apache.org/jira/browse/HBASE-18175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060106#comment-16060106 ] Yi Liang commented on HBASE-18175: -- I can add a top-level hbase-spark-it module, and then move thing around once discussion reaches consensus. What do you think [~busbey] > Add hbase-spark integration test into hbase-it > -- > > Key: HBASE-18175 > URL: https://issues.apache.org/jira/browse/HBASE-18175 > Project: HBase > Issue Type: Test > Components: spark >Reporter: Yi Liang >Assignee: Yi Liang >Priority: Critical > Fix For: 2.0.0 > > Attachments: hbase-18175-v1.patch > > > After HBASE-17574, all test under hbase-spark are regarded as unit test, and > this jira will add integration test of hbase-spark into hbase-it. This patch > run same tests as mapreduce.IntegrationTestBulkLoad, just change mapreduce to > spark. > test in Maven: > mvn verify -Dit.test=IntegrationTestSparkBulkLoad > test on cluster: > spark-submit --class > org.apache.hadoop.hbase.spark.IntegrationTestSparkBulkLoad > HBASE_HOME/lib/hbase-it-2.0.0-SNAPSHOT-tests.jar > -Dhbase.spark.bulkload.chainlength=50 -m slowDeterministic -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17908) Upgrade guava
[ https://issues.apache.org/jira/browse/HBASE-17908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-17908: -- Attachment: HBASE-17908.master.001.patch > Upgrade guava > - > > Key: HBASE-17908 > URL: https://issues.apache.org/jira/browse/HBASE-17908 > Project: HBase > Issue Type: Sub-task > Components: dependencies >Reporter: Balazs Meszaros >Assignee: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-17908.master.001.patch > > > Currently we are using guava 12.0.1, but the latest version is 21.0. > Upgrading guava is always a hassle because it is not always backward > compatible with itself. > Currently I think there are to approaches: > 1. Upgrade guava to the newest version (21.0) and shade it. > 2. Upgrade guava to a version which does not break or builds (15.0). > If we can update it, some dependencies should be removed: > commons-collections, commons-codec, ... -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17125) Inconsistent result when use filter to read data
[
https://issues.apache.org/jira/browse/HBASE-17125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060137#comment-16060137
]
Guanghao Zhang commented on HBASE-17125:
[~tedyu] your "slack" idea is easy to have bug and has side effects. I try it
in HBASE-17125.master.no-specified-filter.patch.
bq. I have run test suite with updated aggregate patch which passed
As i said in email, "ut passed" doesn't mean that your patch is right... You
send me a v11 patch("ut passed"), I pointed the bug in it. Then you fix it. But
the latest patch which you send me is still wrong.. Please upload your
patch here and review board. And let more people to review to make it right.
Thanks.
> Inconsistent result when use filter to read data
>
>
> Key: HBASE-17125
> URL: https://issues.apache.org/jira/browse/HBASE-17125
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: example.diff, HBASE-17125.master.001.patch,
> HBASE-17125.master.002.patch, HBASE-17125.master.002.patch,
> HBASE-17125.master.003.patch, HBASE-17125.master.004.patch,
> HBASE-17125.master.005.patch, HBASE-17125.master.006.patch,
> HBASE-17125.master.007.patch, HBASE-17125.master.008.patch,
> HBASE-17125.master.009.patch, HBASE-17125.master.009.patch,
> HBASE-17125.master.010.patch, HBASE-17125.master.011.patch,
> HBASE-17125.master.011.patch, HBASE-17125.master.012.patch,
> HBASE-17125.master.013.patch, HBASE-17125.master.checkReturnedVersions.patch,
> HBASE-17125.master.no-specified-filter.patch
>
>
> Assume a cloumn's max versions is 3, then we write 4 versions of this column.
> The oldest version doesn't remove immediately. But from the user view, the
> oldest version has gone. When user use a filter to query, if the filter skip
> a new version, then the oldest version will be seen again. But after compact
> the region, then the oldest version will never been seen. So it is weird for
> user. The query will get inconsistent result before and after region
> compaction.
> The reason is matchColumn method of UserScanQueryMatcher. It first check the
> cell by filter, then check the number of versions needed. So if the filter
> skip the new version, then the oldest version will be seen again when it is
> not removed.
> Have a discussion offline with [~Apache9] and [~fenghh], now we have two
> solution for this problem. The first idea is check the number of versions
> first, then check the cell by filter. As the comment of setFilter, the filter
> is called after all tests for ttl, column match, deletes and max versions
> have been run.
> {code}
> /**
>* Apply the specified server-side filter when performing the Query.
>* Only {@link Filter#filterKeyValue(Cell)} is called AFTER all tests
>* for ttl, column match, deletes and max versions have been run.
>* @param filter filter to run on the server
>* @return this for invocation chaining
>*/
> public Query setFilter(Filter filter) {
> this.filter = filter;
> return this;
> }
> {code}
> But this idea has another problem, if a column's max version is 5 and the
> user query only need 3 versions. It first check the version's number, then
> check the cell by filter. So the cells number of the result may less than 3.
> But there are 2 versions which don't read anymore.
> So the second idea has three steps.
> 1. check by the max versions of this column
> 2. check the kv by filter
> 3. check the versions which user need.
> But this will lead the ScanQueryMatcher more complicated. And this will break
> the javadoc of Query.setFilter.
> Now we don't have a final solution for this problem. Suggestions are welcomed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-17125) Inconsistent result when use filter to read data
[
https://issues.apache.org/jira/browse/HBASE-17125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-17125:
---
Attachment: 17125-slack-13.txt
17125-slack-13.txt is latest incarnation using slack.
Currently uploading to review board gave me 'Unknown error'
> Inconsistent result when use filter to read data
>
>
> Key: HBASE-17125
> URL: https://issues.apache.org/jira/browse/HBASE-17125
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: 17125-slack-13.txt, example.diff,
> HBASE-17125.master.001.patch, HBASE-17125.master.002.patch,
> HBASE-17125.master.002.patch, HBASE-17125.master.003.patch,
> HBASE-17125.master.004.patch, HBASE-17125.master.005.patch,
> HBASE-17125.master.006.patch, HBASE-17125.master.007.patch,
> HBASE-17125.master.008.patch, HBASE-17125.master.009.patch,
> HBASE-17125.master.009.patch, HBASE-17125.master.010.patch,
> HBASE-17125.master.011.patch, HBASE-17125.master.011.patch,
> HBASE-17125.master.012.patch, HBASE-17125.master.013.patch,
> HBASE-17125.master.checkReturnedVersions.patch,
> HBASE-17125.master.no-specified-filter.patch
>
>
> Assume a cloumn's max versions is 3, then we write 4 versions of this column.
> The oldest version doesn't remove immediately. But from the user view, the
> oldest version has gone. When user use a filter to query, if the filter skip
> a new version, then the oldest version will be seen again. But after compact
> the region, then the oldest version will never been seen. So it is weird for
> user. The query will get inconsistent result before and after region
> compaction.
> The reason is matchColumn method of UserScanQueryMatcher. It first check the
> cell by filter, then check the number of versions needed. So if the filter
> skip the new version, then the oldest version will be seen again when it is
> not removed.
> Have a discussion offline with [~Apache9] and [~fenghh], now we have two
> solution for this problem. The first idea is check the number of versions
> first, then check the cell by filter. As the comment of setFilter, the filter
> is called after all tests for ttl, column match, deletes and max versions
> have been run.
> {code}
> /**
>* Apply the specified server-side filter when performing the Query.
>* Only {@link Filter#filterKeyValue(Cell)} is called AFTER all tests
>* for ttl, column match, deletes and max versions have been run.
>* @param filter filter to run on the server
>* @return this for invocation chaining
>*/
> public Query setFilter(Filter filter) {
> this.filter = filter;
> return this;
> }
> {code}
> But this idea has another problem, if a column's max version is 5 and the
> user query only need 3 versions. It first check the version's number, then
> check the cell by filter. So the cells number of the result may less than 3.
> But there are 2 versions which don't read anymore.
> So the second idea has three steps.
> 1. check by the max versions of this column
> 2. check the kv by filter
> 3. check the versions which user need.
> But this will lead the ScanQueryMatcher more complicated. And this will break
> the javadoc of Query.setFilter.
> Now we don't have a final solution for this problem. Suggestions are welcomed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18253) Ability to isolate regions on regionservers through hbase shell
[
https://issues.apache.org/jira/browse/HBASE-18253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060156#comment-16060156
]
Francis Liu commented on HBASE-18253:
-
{quote}
So that begs the question if using regionserver groups is going to be something
we want to suggest.
{quote}
I may be biased here but I'd recommend it to users that are doing serious
workload/scale. Branch-1 backport would be great.
> Ability to isolate regions on regionservers through hbase shell
> ---
>
> Key: HBASE-18253
> URL: https://issues.apache.org/jira/browse/HBASE-18253
> Project: HBase
> Issue Type: Task
>Affects Versions: 2.0.0-alpha-1
>Reporter: churro morales
>Assignee: Chinmay Kulkarni
>Priority: Minor
>
> Now that we have the ability to put regionservers in draining mode through
> the hbase shell. Another tool that would be nice is sometimes certain
> regions need to be isolated from others (temporarily - like META). A shell
> command with the form:
> shell> isolate_regions '', '', ''.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-17908) Upgrade guava
[ https://issues.apache.org/jira/browse/HBASE-17908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-17908: -- Status: Patch Available (was: Open) Lets see how this does. > Upgrade guava > - > > Key: HBASE-17908 > URL: https://issues.apache.org/jira/browse/HBASE-17908 > Project: HBase > Issue Type: Sub-task > Components: dependencies >Reporter: Balazs Meszaros >Assignee: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-17908.master.001.patch > > > Currently we are using guava 12.0.1, but the latest version is 21.0. > Upgrading guava is always a hassle because it is not always backward > compatible with itself. > Currently I think there are to approaches: > 1. Upgrade guava to the newest version (21.0) and shade it. > 2. Upgrade guava to a version which does not break or builds (15.0). > If we can update it, some dependencies should be removed: > commons-collections, commons-codec, ... -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18023) Log multi-* requests for more than threshold number of rows
[
https://issues.apache.org/jira/browse/HBASE-18023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060175#comment-16060175
]
Hadoop QA commented on HBASE-18023:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 30s
{color} | {color:red} hbase-common in master has 2 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 22s
{color} | {color:red} hbase-server in master has 10 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 36s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha3. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 55s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 108m 31s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 155m 33s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.master.procedure.TestMasterProcedureWalLease |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12874132/HBASE-18023.master.004.patch
|
| JIRA Issue | HBASE-18023 |
| Optional Tests | asflicense javac javadoc unit xml findbugs
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 0f3454a916e7 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.
[jira] [Commented] (HBASE-18240) Add hbase-auxillary, a project with hbase utility including an hbase-shaded-thirdparty module with guava, netty, etc.
[ https://issues.apache.org/jira/browse/HBASE-18240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060222#comment-16060222 ] stack commented on HBASE-18240: --- Just pushed to master the addition of protobuf 3.3, protobuf util, and netty 4.1.12. Deployed a snapshot too. Also changed relocation from o.a.h.thirdparty.shaded to o.a.h.h.shaded to match the shading already being done in hbase core. Working on making the patching mechanism we have for protobuf moved to this project. > Add hbase-auxillary, a project with hbase utility including an > hbase-shaded-thirdparty module with guava, netty, etc. > - > > Key: HBASE-18240 > URL: https://issues.apache.org/jira/browse/HBASE-18240 > Project: HBase > Issue Type: Sub-task > Components: dependencies >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: HBASE-18240.master.001.patch, hbase-auxillary.tgz > > > This issue is about adding a new related project to host hbase auxillary > utility. In this new project, the first thing we'd add is a module to host > shaded versions of third party libraries. > This task comes of discussion held here > http://apache-hbase.679495.n3.nabble.com/DISCUSS-More-Shading-td4083025.html > where one conclusion of the DISCUSSION was "... pushing this part forward > with some code is the next logical step. Seems to be consensus about taking > our known internal dependencies and performing this shade magic." -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-18119) Improve HFile readability and modify ChecksumUtil log level
[ https://issues.apache.org/jira/browse/HBASE-18119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16058565#comment-16058565 ] Qilin Cao edited comment on HBASE-18119 at 6/23/17 12:28 AM: - OK, [~zyork] I recreated a patch. was (Author: qilin cao): OK, I recreated a patch. > Improve HFile readability and modify ChecksumUtil log level > --- > > Key: HBASE-18119 > URL: https://issues.apache.org/jira/browse/HBASE-18119 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Qilin Cao >Assignee: Qilin Cao >Priority: Minor > Attachments: HBASE-18119-v1.patch, HBASE-18119-v3.patch, > HBASE-18119-v4.patch, HBASE-18119-v5.patch > > > It is confused when I read the HFile.checkHFileVersion method, so I change > the if expression. Simultaneously, I change ChecksumUtil the info log level > to trace. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18086) Create native client which creates load on selected cluster
[ https://issues.apache.org/jira/browse/HBASE-18086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-18086: --- Attachment: 18086.v3.txt In patch v3, I added: multi-family, multi-column support both scan and get are used in verification Will continue polishing and enriching. Tested using standalone cluster > Create native client which creates load on selected cluster > --- > > Key: HBASE-18086 > URL: https://issues.apache.org/jira/browse/HBASE-18086 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 18086.v1.txt, 18086.v3.txt > > > This task is to create a client which uses multiple threads to conduct Puts > followed by Gets against selected cluster. > Default is to run the tool against local cluster. > This would give us some idea on the characteristics of native client in terms > of handling high load. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17908) Upgrade guava
[
https://issues.apache.org/jira/browse/HBASE-17908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060246#comment-16060246
]
Hadoop QA commented on HBASE-17908:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 205 new or modified
test files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
0s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 55s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 6m
33s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 5m
15s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 1s
{color} | {color:blue} Skipped patched modules with no Java source:
hbase-testing-util hbase-assembly . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 32s
{color} | {color:red} hbase-common in master has 2 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 46s
{color} | {color:red} hbase-client in master has 4 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 28s
{color} | {color:red} hbase-server in master has 10 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 24s
{color} | {color:red} hbase-examples in master has 2 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 31s
{color} | {color:red} hbase-rest in master has 3 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 50s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 14s
{color} | {color:red} hbase-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 8s
{color} | {color:red} hbase-metrics-api in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 10s
{color} | {color:red} hbase-procedure in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 7s
{color} | {color:red} hbase-hadoop-compat in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 8s
{color} | {color:red} hbase-metrics in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 10s
{color} | {color:red} hbase-hadoop2-compat in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 13s
{color} | {color:red} hbase-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 9s
{color} | {color:red} hbase-prefix-tree in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 25s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 17s
{color} | {color:red} hbase-thrift in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 12s
{color} | {color:red} hbase-rsgroup in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 13s
{color} | {color:red} hbase-endpoint in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 14s
{color} | {color:red} hbase-it in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 13s
{color} | {color:red} hbase-examples in the patch failed. {color} |
| {color:red}-1{color} | {color:red
[jira] [Commented] (HBASE-17125) Inconsistent result when use filter to read data
[
https://issues.apache.org/jira/browse/HBASE-17125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060247#comment-16060247
]
Ted Yu commented on HBASE-17125:
Review board:
https://reviews.apache.org/r/60381/
This is just an alternate approach. Guanghao is the owner of this JIRA.
> Inconsistent result when use filter to read data
>
>
> Key: HBASE-17125
> URL: https://issues.apache.org/jira/browse/HBASE-17125
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: 17125-slack-13.txt, example.diff,
> HBASE-17125.master.001.patch, HBASE-17125.master.002.patch,
> HBASE-17125.master.002.patch, HBASE-17125.master.003.patch,
> HBASE-17125.master.004.patch, HBASE-17125.master.005.patch,
> HBASE-17125.master.006.patch, HBASE-17125.master.007.patch,
> HBASE-17125.master.008.patch, HBASE-17125.master.009.patch,
> HBASE-17125.master.009.patch, HBASE-17125.master.010.patch,
> HBASE-17125.master.011.patch, HBASE-17125.master.011.patch,
> HBASE-17125.master.012.patch, HBASE-17125.master.013.patch,
> HBASE-17125.master.checkReturnedVersions.patch,
> HBASE-17125.master.no-specified-filter.patch
>
>
> Assume a cloumn's max versions is 3, then we write 4 versions of this column.
> The oldest version doesn't remove immediately. But from the user view, the
> oldest version has gone. When user use a filter to query, if the filter skip
> a new version, then the oldest version will be seen again. But after compact
> the region, then the oldest version will never been seen. So it is weird for
> user. The query will get inconsistent result before and after region
> compaction.
> The reason is matchColumn method of UserScanQueryMatcher. It first check the
> cell by filter, then check the number of versions needed. So if the filter
> skip the new version, then the oldest version will be seen again when it is
> not removed.
> Have a discussion offline with [~Apache9] and [~fenghh], now we have two
> solution for this problem. The first idea is check the number of versions
> first, then check the cell by filter. As the comment of setFilter, the filter
> is called after all tests for ttl, column match, deletes and max versions
> have been run.
> {code}
> /**
>* Apply the specified server-side filter when performing the Query.
>* Only {@link Filter#filterKeyValue(Cell)} is called AFTER all tests
>* for ttl, column match, deletes and max versions have been run.
>* @param filter filter to run on the server
>* @return this for invocation chaining
>*/
> public Query setFilter(Filter filter) {
> this.filter = filter;
> return this;
> }
> {code}
> But this idea has another problem, if a column's max version is 5 and the
> user query only need 3 versions. It first check the version's number, then
> check the cell by filter. So the cells number of the result may less than 3.
> But there are 2 versions which don't read anymore.
> So the second idea has three steps.
> 1. check by the max versions of this column
> 2. check the kv by filter
> 3. check the versions which user need.
> But this will lead the ScanQueryMatcher more complicated. And this will break
> the javadoc of Query.setFilter.
> Now we don't have a final solution for this problem. Suggestions are welcomed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18253) Ability to isolate regions on regionservers through hbase shell
[ https://issues.apache.org/jira/browse/HBASE-18253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060248#comment-16060248 ] Andrew Purtell commented on HBASE-18253: I'm out for a week but when I get back I'll take over that branch-1 backport JIRA and carry it to (hopefully) commit. > Ability to isolate regions on regionservers through hbase shell > --- > > Key: HBASE-18253 > URL: https://issues.apache.org/jira/browse/HBASE-18253 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0-alpha-1 >Reporter: churro morales >Assignee: Chinmay Kulkarni >Priority: Minor > > Now that we have the ability to put regionservers in draining mode through > the hbase shell. Another tool that would be nice is sometimes certain > regions need to be isolated from others (temporarily - like META). A shell > command with the form: > shell> isolate_regions '', '', ''. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18253) Ability to isolate regions on regionservers through hbase shell
[ https://issues.apache.org/jira/browse/HBASE-18253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060281#comment-16060281 ] Francis Liu commented on HBASE-18253: - Cool, I too have a 1.3 patch. But I'd be happy to review yours. > Ability to isolate regions on regionservers through hbase shell > --- > > Key: HBASE-18253 > URL: https://issues.apache.org/jira/browse/HBASE-18253 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0-alpha-1 >Reporter: churro morales >Assignee: Chinmay Kulkarni >Priority: Minor > > Now that we have the ability to put regionservers in draining mode through > the hbase shell. Another tool that would be nice is sometimes certain > regions need to be isolated from others (temporarily - like META). A shell > command with the form: > shell> isolate_regions '', '', ''. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18177) FanOutOneBlockAsyncDFSOutputHelper fails to compile against Hadoop 3
[ https://issues.apache.org/jira/browse/HBASE-18177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060282#comment-16060282 ] Duo Zhang commented on HBASE-18177: --- [~mdrob] There are bunch of reflection examples in FanOutOneBlockAsyncDFSOutputHelper I think. Usually I prefer to initialize a proxy interface at start to avoid throw and catch NoSuchMethodException every time. Thanks. > FanOutOneBlockAsyncDFSOutputHelper fails to compile against Hadoop 3 > > > Key: HBASE-18177 > URL: https://issues.apache.org/jira/browse/HBASE-18177 > Project: HBase > Issue Type: Bug > Components: wal >Reporter: Esteban Gutierrez > Attachments: HBASE-18177.patch > > > After HDFS-10996 ClientProtocol#create() needs to specify the erasure code > policy to use. In the meantime we should add a workaround to > FanOutOneBlockAsyncDFSOutputHelper to be able to compile against Hadoop 3 and > Hadoop 2. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17125) Inconsistent result when use filter to read data
[
https://issues.apache.org/jira/browse/HBASE-17125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060309#comment-16060309
]
Hadoop QA commented on HBASE-17125:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 3s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 31s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
35s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 12s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
36s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 57s
{color} | {color:red} hbase-client in master has 4 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 54s
{color} | {color:red} hbase-client in master has 4 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 58s
{color} | {color:red} hbase-server in master has 10 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
6s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 55s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 2s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha3. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
44s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 17s
{color} | {color:red} hbase-client generated 2 new + 1 unchanged - 1 fixed = 3
total (was 2) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 24s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 117m 5s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 171m 11s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.master.procedure.TestMasterProcedureWalLease |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12874152/17125-slack-13.txt |
| JIRA Issue | HBASE-17125 |
| Optional Tests | asflicense javac javadoc unit find
[jira] [Commented] (HBASE-18240) Add hbase-auxillary, a project with hbase utility including an hbase-shaded-thirdparty module with guava, netty, etc.
[ https://issues.apache.org/jira/browse/HBASE-18240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060335#comment-16060335 ] Anoop Sam John commented on HBASE-18240: bq.Working on making the patching mechanism we have for protobuf moved to this project. Great!!! > Add hbase-auxillary, a project with hbase utility including an > hbase-shaded-thirdparty module with guava, netty, etc. > - > > Key: HBASE-18240 > URL: https://issues.apache.org/jira/browse/HBASE-18240 > Project: HBase > Issue Type: Sub-task > Components: dependencies >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: HBASE-18240.master.001.patch, hbase-auxillary.tgz > > > This issue is about adding a new related project to host hbase auxillary > utility. In this new project, the first thing we'd add is a module to host > shaded versions of third party libraries. > This task comes of discussion held here > http://apache-hbase.679495.n3.nabble.com/DISCUSS-More-Shading-td4083025.html > where one conclusion of the DISCUSSION was "... pushing this part forward > with some code is the next logical step. Seems to be consensus about taking > our known internal dependencies and performing this shade magic." -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17678) FilterList with MUST_PASS_ONE may lead to redundant cells returned
[
https://issues.apache.org/jira/browse/HBASE-17678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Hu updated HBASE-17678:
-
Attachment: HBASE-17678.branch-1.1.v2.patch
> FilterList with MUST_PASS_ONE may lead to redundant cells returned
> --
>
> Key: HBASE-17678
> URL: https://issues.apache.org/jira/browse/HBASE-17678
> Project: HBase
> Issue Type: Bug
> Components: Filters
>Affects Versions: 2.0.0, 1.3.0, 1.2.1
> Environment: RedHat 7.x
>Reporter: Jason Tokayer
>Assignee: Zheng Hu
> Attachments: HBASE-17678.addendum.patch, HBASE-17678.addendum.patch,
> HBASE-17678.branch-1.1.v1.patch, HBASE-17678.branch-1.1.v2.patch,
> HBASE-17678.branch-1.1.v2.patch, HBASE-17678.branch-1.1.v2.patch,
> HBASE-17678.branch-1.v1.patch, HBASE-17678.branch-1.v1.patch,
> HBASE-17678.branch-1.v2.patch, HBASE-17678.branch-1.v2.patch,
> HBASE-17678.v1.patch, HBASE-17678.v1.rough.patch, HBASE-17678.v2.patch,
> HBASE-17678.v3.patch, HBASE-17678.v4.patch, HBASE-17678.v4.patch,
> HBASE-17678.v5.patch, HBASE-17678.v6.patch, HBASE-17678.v7.patch,
> HBASE-17678.v7.patch, TestColumnPaginationFilterDemo.java
>
>
> When combining ColumnPaginationFilter with a single-element filterList,
> MUST_PASS_ONE and MUST_PASS_ALL give different results when there are
> multiple cells with the same timestamp. This is unexpected since there is
> only a single filter in the list, and I would believe that MUST_PASS_ALL and
> MUST_PASS_ONE should only affect the behavior of the joined filter and not
> the behavior of any one of the individual filters. If this is not a bug then
> it would be nice if the documentation is updated to explain this nuanced
> behavior.
> I know that there was a decision made in an earlier Hbase version to keep
> multiple cells with the same timestamp. This is generally fine but presents
> an issue when using the aforementioned filter combination.
> Steps to reproduce:
> In the shell create a table and insert some data:
> {code:none}
> create 'ns:tbl',{NAME => 'family',VERSIONS => 100}
> put 'ns:tbl','row','family:name','John',1
> put 'ns:tbl','row','family:name','Jane',1
> put 'ns:tbl','row','family:name','Gil',1
> put 'ns:tbl','row','family:name','Jane',1
> {code}
> Then, use a Scala client as:
> {code:none}
> import org.apache.hadoop.hbase.filter._
> import org.apache.hadoop.hbase.util.Bytes
> import org.apache.hadoop.hbase.client._
> import org.apache.hadoop.hbase.{CellUtil, HBaseConfiguration, TableName}
> import scala.collection.mutable._
> val config = HBaseConfiguration.create()
> config.set("hbase.zookeeper.quorum", "localhost")
> config.set("hbase.zookeeper.property.clientPort", "2181")
> val connection = ConnectionFactory.createConnection(config)
> val logicalOp = FilterList.Operator.MUST_PASS_ONE
> val limit = 1
> var resultsList = ListBuffer[String]()
> for (offset <- 0 to 20 by limit) {
> val table = connection.getTable(TableName.valueOf("ns:tbl"))
> val paginationFilter = new ColumnPaginationFilter(limit,offset)
> val filterList: FilterList = new FilterList(logicalOp,paginationFilter)
> println("@ filterList = "+filterList)
> val results = table.get(new
> Get(Bytes.toBytes("row")).setFilter(filterList))
> val cells = results.rawCells()
> if (cells != null) {
> for (cell <- cells) {
> val value = new String(CellUtil.cloneValue(cell))
> val qualifier = new String(CellUtil.cloneQualifier(cell))
> val family = new String(CellUtil.cloneFamily(cell))
> val result = "OFFSET = "+offset+":"+family + "," + qualifier
> + "," + value + "," + cell.getTimestamp()
> resultsList.append(result)
> }
> }
> }
> resultsList.foreach(println)
> {code}
> Here are the results for different limit and logicalOp settings:
> {code:none}
> Limit = 1 & logicalOp = MUST_PASS_ALL:
> scala> resultsList.foreach(println)
> OFFSET = 0:family,name,Jane,1
> Limit = 1 & logicalOp = MUST_PASS_ONE:
> scala> resultsList.foreach(println)
> OFFSET = 0:family,name,Jane,1
> OFFSET = 1:family,name,Gil,1
> OFFSET = 2:family,name,Jane,1
> OFFSET = 3:family,name,John,1
> Limit = 2 & logicalOp = MUST_PASS_ALL:
> scala> resultsList.foreach(println)
> OFFSET = 0:family,name,Jane,1
> Limit = 2 & logicalOp = MUST_PASS_ONE:
> scala> resultsList.foreach(println)
> OFFSET = 0:family,name,Jane,1
> OFFSET = 2:family,name,Jane,1
> {code}
> So, it seems that MUST_PASS_ALL gives the expected behavior, but
> MUST_PASS_ONE does not. Furthermore, MUST_PASS_ONE seems to give only a
> single (not-duplicated) within a page, but not a
[jira] [Updated] (HBASE-18221) Switch from pread to stream should happen under HStore's reentrant lock
[
https://issues.apache.org/jira/browse/HBASE-18221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-18221:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Pushed to branch-2 and master. Thanks for the reviews.
> Switch from pread to stream should happen under HStore's reentrant lock
> ---
>
> Key: HBASE-18221
> URL: https://issues.apache.org/jira/browse/HBASE-18221
> Project: HBase
> Issue Type: Sub-task
> Components: Scanners
>Affects Versions: 2.0.0, 3.0.0, 2.0.0-alpha-1
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-18221_2_fortestcasefailure.patch,
> HBASE-18221_2.patch, HBASE-18221_2.patch
>
>
> Found this while debugging HBASE-18186. When we try to reopen the scanners on
> the storefiles while trying to switch over from pread to stream, we do not
> use the HStore's reentrant lock to get the current Storefiles from the
> StoreFileManager. All the scan APIs are guarded under that and we must do it
> here also other wise the CompactedHfileDischarger may cause race issues with
> the HStore's datastructures like here
> {code}
> 2017-06-14 18:16:17,223 WARN
> [RpcServer.default.FPBQ.Fifo.handler=23,queue=1,port=16020]
> regionserver.StoreScanner: failed to switch to stream read
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.getScannersForStoreFiles(StoreFileScanner.java:133)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanners(HStore.java:1221)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.trySwitchToStreamRead(StoreScanner.java:997)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.shipped(StoreScanner.java:1134)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.shipped(KeyValueHeap.java:445)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.shipped(HRegion.java:6459)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices$RegionScannerShippedCallBack.run(RSRpcServices.java:339)
> at
> org.apache.hadoop.hbase.ipc.ServerCall.setResponse(ServerCall.java:252)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:166)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258)
> {code}
> I have a working patch fixing this problem. Will do some more testing and try
> to upload the patch after I write a test case for this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-17931) Assign system tables to servers with highest version
[ https://issues.apache.org/jira/browse/HBASE-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-17931: -- Attachment: HBASE-17931.branch-1.v03.patch Fix javadoc warning and failed test. TestCompactionInDeadRegionServer failed because it runs a compaction after the region server is regarded as dead in master. But close log file may be delay so we should wait seconds. > Assign system tables to servers with highest version > > > Key: HBASE-17931 > URL: https://issues.apache.org/jira/browse/HBASE-17931 > Project: HBase > Issue Type: Bug > Components: Region Assignment, scan >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17931.branch-1.v01.patch, > HBASE-17931.branch-1.v02.patch, HBASE-17931.branch-1.v03.patch, > HBASE-17931.v01.patch, HBASE-17931.v02.patch, HBASE-17931.v03.patch > > > In branch-1 and master we have some improvement and new features on scanning > which is not compatible. > A client of old version to a server of new version is compatible (must be a > bug if not, maybe need some test?). > A client of new version may not be able to read from a server of old version > correctly (because of scan limit, moreResults flag, etc), which is ok for > major/minor upgrade and we can tell users to upgrade server before upgrading > client. But RS also use scan to read meta. If meta table is in RS of old > version, all RSs of new version may have trouble while scanning meta table. > So we should make sure meta table always in servers of new version. Force > meta table in Master and upgrade Master first, or assign meta table in region > servers with latest version? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17931) Assign system tables to servers with highest version
[ https://issues.apache.org/jira/browse/HBASE-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-17931: -- Attachment: (was: HBASE-17931.branch-1.v03.patch) > Assign system tables to servers with highest version > > > Key: HBASE-17931 > URL: https://issues.apache.org/jira/browse/HBASE-17931 > Project: HBase > Issue Type: Bug > Components: Region Assignment, scan >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17931.branch-1.v01.patch, > HBASE-17931.branch-1.v02.patch, HBASE-17931.v01.patch, HBASE-17931.v02.patch, > HBASE-17931.v03.patch > > > In branch-1 and master we have some improvement and new features on scanning > which is not compatible. > A client of old version to a server of new version is compatible (must be a > bug if not, maybe need some test?). > A client of new version may not be able to read from a server of old version > correctly (because of scan limit, moreResults flag, etc), which is ok for > major/minor upgrade and we can tell users to upgrade server before upgrading > client. But RS also use scan to read meta. If meta table is in RS of old > version, all RSs of new version may have trouble while scanning meta table. > So we should make sure meta table always in servers of new version. Force > meta table in Master and upgrade Master first, or assign meta table in region > servers with latest version? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17931) Assign system tables to servers with highest version
[ https://issues.apache.org/jira/browse/HBASE-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-17931: -- Attachment: HBASE-17931.branch-1.v03.patch > Assign system tables to servers with highest version > > > Key: HBASE-17931 > URL: https://issues.apache.org/jira/browse/HBASE-17931 > Project: HBase > Issue Type: Bug > Components: Region Assignment, scan >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17931.branch-1.v01.patch, > HBASE-17931.branch-1.v02.patch, HBASE-17931.branch-1.v03.patch, > HBASE-17931.v01.patch, HBASE-17931.v02.patch, HBASE-17931.v03.patch > > > In branch-1 and master we have some improvement and new features on scanning > which is not compatible. > A client of old version to a server of new version is compatible (must be a > bug if not, maybe need some test?). > A client of new version may not be able to read from a server of old version > correctly (because of scan limit, moreResults flag, etc), which is ok for > major/minor upgrade and we can tell users to upgrade server before upgrading > client. But RS also use scan to read meta. If meta table is in RS of old > version, all RSs of new version may have trouble while scanning meta table. > So we should make sure meta table always in servers of new version. Force > meta table in Master and upgrade Master first, or assign meta table in region > servers with latest version? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-17125) Inconsistent result when use filter to read data
[
https://issues.apache.org/jira/browse/HBASE-17125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-17125:
---
Attachment: HBASE-17125.master.014.patch
> Inconsistent result when use filter to read data
>
>
> Key: HBASE-17125
> URL: https://issues.apache.org/jira/browse/HBASE-17125
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: 17125-slack-13.txt, example.diff,
> HBASE-17125.master.001.patch, HBASE-17125.master.002.patch,
> HBASE-17125.master.002.patch, HBASE-17125.master.003.patch,
> HBASE-17125.master.004.patch, HBASE-17125.master.005.patch,
> HBASE-17125.master.006.patch, HBASE-17125.master.007.patch,
> HBASE-17125.master.008.patch, HBASE-17125.master.009.patch,
> HBASE-17125.master.009.patch, HBASE-17125.master.010.patch,
> HBASE-17125.master.011.patch, HBASE-17125.master.011.patch,
> HBASE-17125.master.012.patch, HBASE-17125.master.013.patch,
> HBASE-17125.master.014.patch, HBASE-17125.master.checkReturnedVersions.patch,
> HBASE-17125.master.no-specified-filter.patch
>
>
> Assume a cloumn's max versions is 3, then we write 4 versions of this column.
> The oldest version doesn't remove immediately. But from the user view, the
> oldest version has gone. When user use a filter to query, if the filter skip
> a new version, then the oldest version will be seen again. But after compact
> the region, then the oldest version will never been seen. So it is weird for
> user. The query will get inconsistent result before and after region
> compaction.
> The reason is matchColumn method of UserScanQueryMatcher. It first check the
> cell by filter, then check the number of versions needed. So if the filter
> skip the new version, then the oldest version will be seen again when it is
> not removed.
> Have a discussion offline with [~Apache9] and [~fenghh], now we have two
> solution for this problem. The first idea is check the number of versions
> first, then check the cell by filter. As the comment of setFilter, the filter
> is called after all tests for ttl, column match, deletes and max versions
> have been run.
> {code}
> /**
>* Apply the specified server-side filter when performing the Query.
>* Only {@link Filter#filterKeyValue(Cell)} is called AFTER all tests
>* for ttl, column match, deletes and max versions have been run.
>* @param filter filter to run on the server
>* @return this for invocation chaining
>*/
> public Query setFilter(Filter filter) {
> this.filter = filter;
> return this;
> }
> {code}
> But this idea has another problem, if a column's max version is 5 and the
> user query only need 3 versions. It first check the version's number, then
> check the cell by filter. So the cells number of the result may less than 3.
> But there are 2 versions which don't read anymore.
> So the second idea has three steps.
> 1. check by the max versions of this column
> 2. check the kv by filter
> 3. check the versions which user need.
> But this will lead the ScanQueryMatcher more complicated. And this will break
> the javadoc of Query.setFilter.
> Now we don't have a final solution for this problem. Suggestions are welcomed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18221) Switch from pread to stream should happen under HStore's reentrant lock
[
https://issues.apache.org/jira/browse/HBASE-18221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060429#comment-16060429
]
Hudson commented on HBASE-18221:
FAILURE: Integrated in Jenkins build HBase-2.0 #93 (See
[https://builds.apache.org/job/HBase-2.0/93/])
HBASE-18221 Switch from pread to stream should happen under HStore's
(ramkrishna: rev 49825aec6b6c6bf0d0c56fc35730efe80bafcb7b)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StripeStoreFileManager.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultStoreFileManager.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileManager.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java
> Switch from pread to stream should happen under HStore's reentrant lock
> ---
>
> Key: HBASE-18221
> URL: https://issues.apache.org/jira/browse/HBASE-18221
> Project: HBase
> Issue Type: Sub-task
> Components: Scanners
>Affects Versions: 2.0.0, 3.0.0, 2.0.0-alpha-1
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0, 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-18221_2_fortestcasefailure.patch,
> HBASE-18221_2.patch, HBASE-18221_2.patch
>
>
> Found this while debugging HBASE-18186. When we try to reopen the scanners on
> the storefiles while trying to switch over from pread to stream, we do not
> use the HStore's reentrant lock to get the current Storefiles from the
> StoreFileManager. All the scan APIs are guarded under that and we must do it
> here also other wise the CompactedHfileDischarger may cause race issues with
> the HStore's datastructures like here
> {code}
> 2017-06-14 18:16:17,223 WARN
> [RpcServer.default.FPBQ.Fifo.handler=23,queue=1,port=16020]
> regionserver.StoreScanner: failed to switch to stream read
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.getScannersForStoreFiles(StoreFileScanner.java:133)
> at
> org.apache.hadoop.hbase.regionserver.HStore.getScanners(HStore.java:1221)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.trySwitchToStreamRead(StoreScanner.java:997)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.shipped(StoreScanner.java:1134)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.shipped(KeyValueHeap.java:445)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.shipped(HRegion.java:6459)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices$RegionScannerShippedCallBack.run(RSRpcServices.java:339)
> at
> org.apache.hadoop.hbase.ipc.ServerCall.setResponse(ServerCall.java:252)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:166)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:278)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:258)
> {code}
> I have a working patch fixing this problem. Will do some more testing and try
> to upload the patch after I write a test case for this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-17125) Inconsistent result when use filter to read data
[
https://issues.apache.org/jira/browse/HBASE-17125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060430#comment-16060430
]
Anoop Sam John commented on HBASE-17125:
Reviewing HBASE-17125.master.014.patch now
> Inconsistent result when use filter to read data
>
>
> Key: HBASE-17125
> URL: https://issues.apache.org/jira/browse/HBASE-17125
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: 17125-slack-13.txt, example.diff,
> HBASE-17125.master.001.patch, HBASE-17125.master.002.patch,
> HBASE-17125.master.002.patch, HBASE-17125.master.003.patch,
> HBASE-17125.master.004.patch, HBASE-17125.master.005.patch,
> HBASE-17125.master.006.patch, HBASE-17125.master.007.patch,
> HBASE-17125.master.008.patch, HBASE-17125.master.009.patch,
> HBASE-17125.master.009.patch, HBASE-17125.master.010.patch,
> HBASE-17125.master.011.patch, HBASE-17125.master.011.patch,
> HBASE-17125.master.012.patch, HBASE-17125.master.013.patch,
> HBASE-17125.master.014.patch, HBASE-17125.master.checkReturnedVersions.patch,
> HBASE-17125.master.no-specified-filter.patch
>
>
> Assume a cloumn's max versions is 3, then we write 4 versions of this column.
> The oldest version doesn't remove immediately. But from the user view, the
> oldest version has gone. When user use a filter to query, if the filter skip
> a new version, then the oldest version will be seen again. But after compact
> the region, then the oldest version will never been seen. So it is weird for
> user. The query will get inconsistent result before and after region
> compaction.
> The reason is matchColumn method of UserScanQueryMatcher. It first check the
> cell by filter, then check the number of versions needed. So if the filter
> skip the new version, then the oldest version will be seen again when it is
> not removed.
> Have a discussion offline with [~Apache9] and [~fenghh], now we have two
> solution for this problem. The first idea is check the number of versions
> first, then check the cell by filter. As the comment of setFilter, the filter
> is called after all tests for ttl, column match, deletes and max versions
> have been run.
> {code}
> /**
>* Apply the specified server-side filter when performing the Query.
>* Only {@link Filter#filterKeyValue(Cell)} is called AFTER all tests
>* for ttl, column match, deletes and max versions have been run.
>* @param filter filter to run on the server
>* @return this for invocation chaining
>*/
> public Query setFilter(Filter filter) {
> this.filter = filter;
> return this;
> }
> {code}
> But this idea has another problem, if a column's max version is 5 and the
> user query only need 3 versions. It first check the version's number, then
> check the cell by filter. So the cells number of the result may less than 3.
> But there are 2 versions which don't read anymore.
> So the second idea has three steps.
> 1. check by the max versions of this column
> 2. check the kv by filter
> 3. check the versions which user need.
> But this will lead the ScanQueryMatcher more complicated. And this will break
> the javadoc of Query.setFilter.
> Now we don't have a final solution for this problem. Suggestions are welcomed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18218) List replication peers for the cluster
[ https://issues.apache.org/jira/browse/HBASE-18218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060435#comment-16060435 ] Ashish Singhi commented on HBASE-18218: --- Duplicate of HBASE-15809 ? /cc [~appy] > List replication peers for the cluster > -- > > Key: HBASE-18218 > URL: https://issues.apache.org/jira/browse/HBASE-18218 > Project: HBase > Issue Type: New Feature >Reporter: Ali >Assignee: Ali > Attachments: HBASE-18218.v1.branch-1.2.patch, > HBASE-18218.v2.branch-1.patch, HBASE-18218.v2.master.patch, > HBASE-18218.v3.master.patch, screenshot-1.png, screenshot-2.png, > screenshot-3.png > > > HBase Master page that listed all the replication peers for a cluster, with > their associated metadata -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HBASE-17931) Assign system tables to servers with highest version
[ https://issues.apache.org/jira/browse/HBASE-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060424#comment-16060424 ] Phil Yang edited comment on HBASE-17931 at 6/23/17 6:08 AM: Fix javadoc warning and failed test. was (Author: yangzhe1991): Fix javadoc warning and failed test. TestCompactionInDeadRegionServer failed because it runs a compaction after the region server is regarded as dead in master. But close log file may be delay so we should wait seconds. > Assign system tables to servers with highest version > > > Key: HBASE-17931 > URL: https://issues.apache.org/jira/browse/HBASE-17931 > Project: HBase > Issue Type: Bug > Components: Region Assignment, scan >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17931.branch-1.v01.patch, > HBASE-17931.branch-1.v02.patch, HBASE-17931.branch-1.v03.patch, > HBASE-17931.v01.patch, HBASE-17931.v02.patch, HBASE-17931.v03.patch > > > In branch-1 and master we have some improvement and new features on scanning > which is not compatible. > A client of old version to a server of new version is compatible (must be a > bug if not, maybe need some test?). > A client of new version may not be able to read from a server of old version > correctly (because of scan limit, moreResults flag, etc), which is ok for > major/minor upgrade and we can tell users to upgrade server before upgrading > client. But RS also use scan to read meta. If meta table is in RS of old > version, all RSs of new version may have trouble while scanning meta table. > So we should make sure meta table always in servers of new version. Force > meta table in Master and upgrade Master first, or assign meta table in region > servers with latest version? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17125) Inconsistent result when use filter to read data
[
https://issues.apache.org/jira/browse/HBASE-17125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060456#comment-16060456
]
Anoop Sam John commented on HBASE-17125:
So this patch is in line with some old proposal that u put right.. I like it..
Doing some more careful reviews. Pls put the latest patch in RB
> Inconsistent result when use filter to read data
>
>
> Key: HBASE-17125
> URL: https://issues.apache.org/jira/browse/HBASE-17125
> Project: HBase
> Issue Type: Bug
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: 17125-slack-13.txt, example.diff,
> HBASE-17125.master.001.patch, HBASE-17125.master.002.patch,
> HBASE-17125.master.002.patch, HBASE-17125.master.003.patch,
> HBASE-17125.master.004.patch, HBASE-17125.master.005.patch,
> HBASE-17125.master.006.patch, HBASE-17125.master.007.patch,
> HBASE-17125.master.008.patch, HBASE-17125.master.009.patch,
> HBASE-17125.master.009.patch, HBASE-17125.master.010.patch,
> HBASE-17125.master.011.patch, HBASE-17125.master.011.patch,
> HBASE-17125.master.012.patch, HBASE-17125.master.013.patch,
> HBASE-17125.master.014.patch, HBASE-17125.master.checkReturnedVersions.patch,
> HBASE-17125.master.no-specified-filter.patch
>
>
> Assume a cloumn's max versions is 3, then we write 4 versions of this column.
> The oldest version doesn't remove immediately. But from the user view, the
> oldest version has gone. When user use a filter to query, if the filter skip
> a new version, then the oldest version will be seen again. But after compact
> the region, then the oldest version will never been seen. So it is weird for
> user. The query will get inconsistent result before and after region
> compaction.
> The reason is matchColumn method of UserScanQueryMatcher. It first check the
> cell by filter, then check the number of versions needed. So if the filter
> skip the new version, then the oldest version will be seen again when it is
> not removed.
> Have a discussion offline with [~Apache9] and [~fenghh], now we have two
> solution for this problem. The first idea is check the number of versions
> first, then check the cell by filter. As the comment of setFilter, the filter
> is called after all tests for ttl, column match, deletes and max versions
> have been run.
> {code}
> /**
>* Apply the specified server-side filter when performing the Query.
>* Only {@link Filter#filterKeyValue(Cell)} is called AFTER all tests
>* for ttl, column match, deletes and max versions have been run.
>* @param filter filter to run on the server
>* @return this for invocation chaining
>*/
> public Query setFilter(Filter filter) {
> this.filter = filter;
> return this;
> }
> {code}
> But this idea has another problem, if a column's max version is 5 and the
> user query only need 3 versions. It first check the version's number, then
> check the cell by filter. So the cells number of the result may less than 3.
> But there are 2 versions which don't read anymore.
> So the second idea has three steps.
> 1. check by the max versions of this column
> 2. check the kv by filter
> 3. check the versions which user need.
> But this will lead the ScanQueryMatcher more complicated. And this will break
> the javadoc of Query.setFilter.
> Now we don't have a final solution for this problem. Suggestions are welcomed.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-18240) Add hbase-thirdparty, a project with hbase utility including an hbase-shaded-thirdparty module with guava, netty, etc.
[ https://issues.apache.org/jira/browse/HBASE-18240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-18240: -- Summary: Add hbase-thirdparty, a project with hbase utility including an hbase-shaded-thirdparty module with guava, netty, etc. (was: Add hbase-auxillary, a project with hbase utility including an hbase-shaded-thirdparty module with guava, netty, etc.) > Add hbase-thirdparty, a project with hbase utility including an > hbase-shaded-thirdparty module with guava, netty, etc. > -- > > Key: HBASE-18240 > URL: https://issues.apache.org/jira/browse/HBASE-18240 > Project: HBase > Issue Type: Sub-task > Components: dependencies >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: HBASE-18240.master.001.patch, hbase-auxillary.tgz > > > This issue is about adding a new related project to host hbase auxillary > utility. In this new project, the first thing we'd add is a module to host > shaded versions of third party libraries. > This task comes of discussion held here > http://apache-hbase.679495.n3.nabble.com/DISCUSS-More-Shading-td4083025.html > where one conclusion of the DISCUSSION was "... pushing this part forward > with some code is the next logical step. Seems to be consensus about taking > our known internal dependencies and performing this shade magic." -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18240) Add hbase-thirdparty, a project with hbase utility including an hbase-shaded-thirdparty module with guava, netty, etc.
[ https://issues.apache.org/jira/browse/HBASE-18240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060461#comment-16060461 ] stack commented on HBASE-18240: --- Pushed new patch on hbase-thirdparty. hbase-thirdparty has guava, gson, netty, and protobuf-util but not protobuf itself. Compiling the fat patched jar that included protobuf and guava and netty, etc., failed; dependencies of guava (errorprone) and netty src are not available so the compile fails. Will leave the production of the protobuf as is in hbase-protocol-shaded for now. Will come back and experiment with a multi-module build under hbase-thirdparty later; one module will do protobuf patching and then next module will do the shade of it all -- no compile. > Add hbase-thirdparty, a project with hbase utility including an > hbase-shaded-thirdparty module with guava, netty, etc. > -- > > Key: HBASE-18240 > URL: https://issues.apache.org/jira/browse/HBASE-18240 > Project: HBase > Issue Type: Sub-task > Components: dependencies >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: HBASE-18240.master.001.patch, hbase-auxillary.tgz > > > This issue is about adding a new related project to host hbase auxillary > utility. In this new project, the first thing we'd add is a module to host > shaded versions of third party libraries. > This task comes of discussion held here > http://apache-hbase.679495.n3.nabble.com/DISCUSS-More-Shading-td4083025.html > where one conclusion of the DISCUSSION was "... pushing this part forward > with some code is the next logical step. Seems to be consensus about taking > our known internal dependencies and performing this shade magic." -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18052) Add document for async admin
[
https://issues.apache.org/jira/browse/HBASE-18052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060464#comment-16060464
]
Duo Zhang commented on HBASE-18052:
---
'when the returned CompletableFuture complete, it means the admin operation
complete' -> 'when the returned CompletableFuture is done, it means the admin
operation has also been done.'
'it only means the compact request was send to HBase and may need time to
execute the compact operation' -> 'it only means the compact request was sent
to HBase and may need some time to finish the compact operation'
'it only support byte[] as parameter' -> 'we only accept byte[] as the
parameter type', and also the same for server name and table name. And you can
also use '`' to highlight the type name, just as what you have done to
{{AsyncConnection}}.
> Add document for async admin
>
>
> Key: HBASE-18052
> URL: https://issues.apache.org/jira/browse/HBASE-18052
> Project: HBase
> Issue Type: Sub-task
> Components: Client
>Reporter: Guanghao Zhang
>Assignee: Zheng Hu
> Fix For: 3.0.0, 2.0.0-alpha-2
>
> Attachments: HBASE-18052.master.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-17931) Assign system tables to servers with highest version
[ https://issues.apache.org/jira/browse/HBASE-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-17931: -- Attachment: HBASE-17931.branch-1.v04.patch Unblock mater init and some trivial improvements. > Assign system tables to servers with highest version > > > Key: HBASE-17931 > URL: https://issues.apache.org/jira/browse/HBASE-17931 > Project: HBase > Issue Type: Bug > Components: Region Assignment, scan >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17931.branch-1.v01.patch, > HBASE-17931.branch-1.v02.patch, HBASE-17931.branch-1.v03.patch, > HBASE-17931.branch-1.v04.patch, HBASE-17931.v01.patch, HBASE-17931.v02.patch, > HBASE-17931.v03.patch > > > In branch-1 and master we have some improvement and new features on scanning > which is not compatible. > A client of old version to a server of new version is compatible (must be a > bug if not, maybe need some test?). > A client of new version may not be able to read from a server of old version > correctly (because of scan limit, moreResults flag, etc), which is ok for > major/minor upgrade and we can tell users to upgrade server before upgrading > client. But RS also use scan to read meta. If meta table is in RS of old > version, all RSs of new version may have trouble while scanning meta table. > So we should make sure meta table always in servers of new version. Force > meta table in Master and upgrade Master first, or assign meta table in region > servers with latest version? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-17931) Assign system tables to servers with highest version
[ https://issues.apache.org/jira/browse/HBASE-17931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060471#comment-16060471 ] Phil Yang commented on HBASE-17931: --- RB for branch-1, https://reviews.apache.org/r/60386 After AMv2, the patches for branch-1 and master will not be similar, so let's do them one by one. > Assign system tables to servers with highest version > > > Key: HBASE-17931 > URL: https://issues.apache.org/jira/browse/HBASE-17931 > Project: HBase > Issue Type: Bug > Components: Region Assignment, scan >Reporter: Phil Yang >Assignee: Phil Yang >Priority: Blocker > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-17931.branch-1.v01.patch, > HBASE-17931.branch-1.v02.patch, HBASE-17931.branch-1.v03.patch, > HBASE-17931.branch-1.v04.patch, HBASE-17931.v01.patch, HBASE-17931.v02.patch, > HBASE-17931.v03.patch > > > In branch-1 and master we have some improvement and new features on scanning > which is not compatible. > A client of old version to a server of new version is compatible (must be a > bug if not, maybe need some test?). > A client of new version may not be able to read from a server of old version > correctly (because of scan limit, moreResults flag, etc), which is ok for > major/minor upgrade and we can tell users to upgrade server before upgrading > client. But RS also use scan to read meta. If meta table is in RS of old > version, all RSs of new version may have trouble while scanning meta table. > So we should make sure meta table always in servers of new version. Force > meta table in Master and upgrade Master first, or assign meta table in region > servers with latest version? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
