[jira] [Commented] (HBASE-20877) Hbase-1.2.0 OldWals age getting filled and not purged by Hmaster
[ https://issues.apache.org/jira/browse/HBASE-20877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541228#comment-16541228 ] Manjeet Singh commented on HBASE-20877: --- If I remove OldWal, it's not impacting on my actual data? and not even on Hbase solr indexer? Please note it's a production system. > Hbase-1.2.0 OldWals age getting filled and not purged by Hmaster > > > Key: HBASE-20877 > URL: https://issues.apache.org/jira/browse/HBASE-20877 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.2.0 >Reporter: Manjeet Singh >Priority: Major > > Hbase version 1.2.0 OldWals are getting filled and showing as below > 7.2 K 21.5 K /hbase/.hbase-snapshot > 0 0 /hbase/.tmp > 0 0 /hbase/MasterProcWALs > 18.3 G 60.2 G /hbase/WALs > 28.7 G 86.1 G /hbase/archive > 0 0 /hbase/corrupt > 1.7 T 5.2 T /hbase/data > 42 126 /hbase/hbase.id > 7 21 /hbase/hbase.version > 7.2 T 21.6 T /hbase/oldWALs > > It;s not getting purged by Hmaster as oldWals are supposed to be cleaned in > master background chore, HBASE-20352(for 1.x version) is created to speed up > cleaning oldWals, in our case it's not happening. > hbase.master.logcleaner.ttl is 1 minutes -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19572) RegionMover should use the configured default port number and not the one from HConstants
[
https://issues.apache.org/jira/browse/HBASE-19572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541225#comment-16541225
]
Toshihiro Suzuki commented on HBASE-19572:
--
Thank you [~reidchan]. I just attached a new patch.
> RegionMover should use the configured default port number and not the one
> from HConstants
> -
>
> Key: HBASE-19572
> URL: https://issues.apache.org/jira/browse/HBASE-19572
> Project: HBase
> Issue Type: Bug
>Reporter: Esteban Gutierrez
>Assignee: Toshihiro Suzuki
>Priority: Major
> Attachments: HBASE-19572.master.001.patch,
> HBASE-19572.master.001.patch, HBASE-19572.master.003.patch,
> HBASE-19572.master.004.patch, HBASE-19572.master.004.patch,
> HBASE-19572.master.005.patch
>
>
> The issue I ran into HBASE-19499 was due RegionMover not using the port used
> by {{hbase-site.xml}}. The tool should use the value used in the
> configuration before falling back to the hardcoded value
> {{HConstants.DEFAULT_REGIONSERVER_PORT}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HBASE-20877) Hbase-1.2.0 OldWals age getting filled and not purged by Hmaster
[ https://issues.apache.org/jira/browse/HBASE-20877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan resolved HBASE-20877. --- Resolution: Duplicate > Hbase-1.2.0 OldWals age getting filled and not purged by Hmaster > > > Key: HBASE-20877 > URL: https://issues.apache.org/jira/browse/HBASE-20877 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.2.0 >Reporter: Manjeet Singh >Priority: Major > > Hbase version 1.2.0 OldWals are getting filled and showing as below > 7.2 K 21.5 K /hbase/.hbase-snapshot > 0 0 /hbase/.tmp > 0 0 /hbase/MasterProcWALs > 18.3 G 60.2 G /hbase/WALs > 28.7 G 86.1 G /hbase/archive > 0 0 /hbase/corrupt > 1.7 T 5.2 T /hbase/data > 42 126 /hbase/hbase.id > 7 21 /hbase/hbase.version > 7.2 T 21.6 T /hbase/oldWALs > > It;s not getting purged by Hmaster as oldWals are supposed to be cleaned in > master background chore, HBASE-20352(for 1.x version) is created to speed up > cleaning oldWals, in our case it's not happening. > hbase.master.logcleaner.ttl is 1 minutes -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19572) RegionMover should use the configured default port number and not the one from HConstants
[
https://issues.apache.org/jira/browse/HBASE-19572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Toshihiro Suzuki updated HBASE-19572:
-
Attachment: HBASE-19572.master.005.patch
> RegionMover should use the configured default port number and not the one
> from HConstants
> -
>
> Key: HBASE-19572
> URL: https://issues.apache.org/jira/browse/HBASE-19572
> Project: HBase
> Issue Type: Bug
>Reporter: Esteban Gutierrez
>Assignee: Toshihiro Suzuki
>Priority: Major
> Attachments: HBASE-19572.master.001.patch,
> HBASE-19572.master.001.patch, HBASE-19572.master.003.patch,
> HBASE-19572.master.004.patch, HBASE-19572.master.004.patch,
> HBASE-19572.master.005.patch
>
>
> The issue I ran into HBASE-19499 was due RegionMover not using the port used
> by {{hbase-site.xml}}. The tool should use the value used in the
> configuration before falling back to the hardcoded value
> {{HConstants.DEFAULT_REGIONSERVER_PORT}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20877) Hbase-1.2.0 OldWals age getting filled and not purged by Hmaster
[ https://issues.apache.org/jira/browse/HBASE-20877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541224#comment-16541224 ] Reid Chan commented on HBASE-20877: --- Or you can clean oldWALs directly through hdfs -rm command. > Hbase-1.2.0 OldWals age getting filled and not purged by Hmaster > > > Key: HBASE-20877 > URL: https://issues.apache.org/jira/browse/HBASE-20877 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.2.0 >Reporter: Manjeet Singh >Priority: Major > > Hbase version 1.2.0 OldWals are getting filled and showing as below > 7.2 K 21.5 K /hbase/.hbase-snapshot > 0 0 /hbase/.tmp > 0 0 /hbase/MasterProcWALs > 18.3 G 60.2 G /hbase/WALs > 28.7 G 86.1 G /hbase/archive > 0 0 /hbase/corrupt > 1.7 T 5.2 T /hbase/data > 42 126 /hbase/hbase.id > 7 21 /hbase/hbase.version > 7.2 T 21.6 T /hbase/oldWALs > > It;s not getting purged by Hmaster as oldWals are supposed to be cleaned in > master background chore, HBASE-20352(for 1.x version) is created to speed up > cleaning oldWals, in our case it's not happening. > hbase.master.logcleaner.ttl is 1 minutes -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20877) Hbase-1.2.0 OldWals age getting filled and not purged by Hmaster
[ https://issues.apache.org/jira/browse/HBASE-20877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541221#comment-16541221 ] Reid Chan commented on HBASE-20877: --- hbase-1.2.0 doesn't have HBASE-20352 this patch. You may need to upgrade to higher version or apply that patch manually. > Hbase-1.2.0 OldWals age getting filled and not purged by Hmaster > > > Key: HBASE-20877 > URL: https://issues.apache.org/jira/browse/HBASE-20877 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 1.2.0 >Reporter: Manjeet Singh >Priority: Major > > Hbase version 1.2.0 OldWals are getting filled and showing as below > 7.2 K 21.5 K /hbase/.hbase-snapshot > 0 0 /hbase/.tmp > 0 0 /hbase/MasterProcWALs > 18.3 G 60.2 G /hbase/WALs > 28.7 G 86.1 G /hbase/archive > 0 0 /hbase/corrupt > 1.7 T 5.2 T /hbase/data > 42 126 /hbase/hbase.id > 7 21 /hbase/hbase.version > 7.2 T 21.6 T /hbase/oldWALs > > It;s not getting purged by Hmaster as oldWals are supposed to be cleaned in > master background chore, HBASE-20352(for 1.x version) is created to speed up > cleaning oldWals, in our case it's not happening. > hbase.master.logcleaner.ttl is 1 minutes -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19572) RegionMover should use the configured default port number and not the one from HConstants
[
https://issues.apache.org/jira/browse/HBASE-19572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Toshihiro Suzuki updated HBASE-19572:
-
Attachment: (was: HBASE-19572.patch)
> RegionMover should use the configured default port number and not the one
> from HConstants
> -
>
> Key: HBASE-19572
> URL: https://issues.apache.org/jira/browse/HBASE-19572
> Project: HBase
> Issue Type: Bug
>Reporter: Esteban Gutierrez
>Assignee: Toshihiro Suzuki
>Priority: Major
> Attachments: HBASE-19572.master.001.patch,
> HBASE-19572.master.001.patch, HBASE-19572.master.003.patch,
> HBASE-19572.master.004.patch, HBASE-19572.master.004.patch
>
>
> The issue I ran into HBASE-19499 was due RegionMover not using the port used
> by {{hbase-site.xml}}. The tool should use the value used in the
> configuration before falling back to the hardcoded value
> {{HConstants.DEFAULT_REGIONSERVER_PORT}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19572) RegionMover should use the configured default port number and not the one from HConstants
[
https://issues.apache.org/jira/browse/HBASE-19572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Toshihiro Suzuki updated HBASE-19572:
-
Attachment: (was: HBASE-19572.patch)
> RegionMover should use the configured default port number and not the one
> from HConstants
> -
>
> Key: HBASE-19572
> URL: https://issues.apache.org/jira/browse/HBASE-19572
> Project: HBase
> Issue Type: Bug
>Reporter: Esteban Gutierrez
>Assignee: Toshihiro Suzuki
>Priority: Major
> Attachments: HBASE-19572.master.001.patch,
> HBASE-19572.master.001.patch, HBASE-19572.master.003.patch,
> HBASE-19572.master.004.patch, HBASE-19572.master.004.patch
>
>
> The issue I ran into HBASE-19499 was due RegionMover not using the port used
> by {{hbase-site.xml}}. The tool should use the value used in the
> configuration before falling back to the hardcoded value
> {{HConstants.DEFAULT_REGIONSERVER_PORT}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-20877) Hbase-1.2.0 OldWals age getting filled and not purged by Hmaster
Manjeet Singh created HBASE-20877: - Summary: Hbase-1.2.0 OldWals age getting filled and not purged by Hmaster Key: HBASE-20877 URL: https://issues.apache.org/jira/browse/HBASE-20877 Project: HBase Issue Type: Bug Components: Replication Affects Versions: 1.2.0 Reporter: Manjeet Singh Hbase version 1.2.0 OldWals are getting filled and showing as below 7.2 K 21.5 K /hbase/.hbase-snapshot 0 0 /hbase/.tmp 0 0 /hbase/MasterProcWALs 18.3 G 60.2 G /hbase/WALs 28.7 G 86.1 G /hbase/archive 0 0 /hbase/corrupt 1.7 T 5.2 T /hbase/data 42 126 /hbase/hbase.id 7 21 /hbase/hbase.version 7.2 T 21.6 T /hbase/oldWALs It;s not getting purged by Hmaster as oldWals are supposed to be cleaned in master background chore, HBASE-20352(for 1.x version) is created to speed up cleaning oldWals, in our case it's not happening. hbase.master.logcleaner.ttl is 1 minutes -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19572) RegionMover should use the configured default port number and not the one from HConstants
[
https://issues.apache.org/jira/browse/HBASE-19572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541214#comment-16541214
]
Reid Chan commented on HBASE-19572:
---
Got your mind, ok.
> RegionMover should use the configured default port number and not the one
> from HConstants
> -
>
> Key: HBASE-19572
> URL: https://issues.apache.org/jira/browse/HBASE-19572
> Project: HBase
> Issue Type: Bug
>Reporter: Esteban Gutierrez
>Assignee: Toshihiro Suzuki
>Priority: Major
> Attachments: HBASE-19572.master.001.patch,
> HBASE-19572.master.001.patch, HBASE-19572.master.003.patch,
> HBASE-19572.master.004.patch, HBASE-19572.master.004.patch,
> HBASE-19572.patch, HBASE-19572.patch
>
>
> The issue I ran into HBASE-19499 was due RegionMover not using the port used
> by {{hbase-site.xml}}. The tool should use the value used in the
> configuration before falling back to the hardcoded value
> {{HConstants.DEFAULT_REGIONSERVER_PORT}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20697) Can't cache All region locations of the specify table by calling table.getRegionLocator().getAllRegionLocations()
[
https://issues.apache.org/jira/browse/HBASE-20697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-20697:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Can't cache All region locations of the specify table by calling
> table.getRegionLocator().getAllRegionLocations()
> -
>
> Key: HBASE-20697
> URL: https://issues.apache.org/jira/browse/HBASE-20697
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.1, 1.2.6, 2.0.1
>Reporter: zhaoyuan
>Assignee: zhaoyuan
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.4.6, 2.0.2, 2.2.0, 2.1.1
>
> Attachments: HBASE-20697.branch-1.2.001.patch,
> HBASE-20697.branch-1.2.002.patch, HBASE-20697.branch-1.2.003.patch,
> HBASE-20697.branch-1.2.004.patch, HBASE-20697.branch-1.addendum.patch,
> HBASE-20697.master.001.patch, HBASE-20697.master.002.patch,
> HBASE-20697.master.002.patch, HBASE-20697.master.003.patch
>
>
> When we upgrade and restart a new version application which will read and
> write to HBase, we will get some operation timeout. The time out is expected
> because when the application restarts，It will not hold any region locations
> cache and do communication with zk and meta regionserver to get region
> locations.
> We want to avoid these timeouts so we do warmup work and as far as I am
> concerned,the method table.getRegionLocator().getAllRegionLocations() will
> fetch all region locations and cache them. However, it didn't work good.
> There are still a lot of time outs,so it confused me.
> I dig into the source code and find something below
> {code:java}
> // code placeholder
> public List getAllRegionLocations() throws IOException {
> TableName tableName = getName();
> NavigableMap locations =
> MetaScanner.allTableRegions(this.connection, tableName);
> ArrayList regions = new ArrayList<>(locations.size());
> for (Entry entry : locations.entrySet()) {
> regions.add(new HRegionLocation(entry.getKey(), entry.getValue()));
> }
> if (regions.size() > 0) {
> connection.cacheLocation(tableName, new RegionLocations(regions));
> }
> return regions;
> }
> In MetaCache
> public void cacheLocation(final TableName tableName, final RegionLocations
> locations) {
> byte [] startKey =
> locations.getRegionLocation().getRegionInfo().getStartKey();
> ConcurrentMap tableLocations =
> getTableLocations(tableName);
> RegionLocations oldLocation = tableLocations.putIfAbsent(startKey,
> locations);
> boolean isNewCacheEntry = (oldLocation == null);
> if (isNewCacheEntry) {
> if (LOG.isTraceEnabled()) {
> LOG.trace("Cached location: " + locations);
> }
> addToCachedServers(locations);
> return;
> }
> {code}
> It will collect all regions into one RegionLocations object and only cache
> the first not null region location and then when we put or get to hbase, we
> do getCacheLocation()
> {code:java}
> // code placeholder
> public RegionLocations getCachedLocation(final TableName tableName, final
> byte [] row) {
> ConcurrentNavigableMap tableLocations =
> getTableLocations(tableName);
> Entry e = tableLocations.floorEntry(row);
> if (e == null) {
> if (metrics!= null) metrics.incrMetaCacheMiss();
> return null;
> }
> RegionLocations possibleRegion = e.getValue();
> // make sure that the end key is greater than the row we're looking
> // for, otherwise the row actually belongs in the next region, not
> // this one. the exception case is when the endkey is
> // HConstants.EMPTY_END_ROW, signifying that the region we're
> // checking is actually the last region in the table.
> byte[] endKey =
> possibleRegion.getRegionLocation().getRegionInfo().getEndKey();
> if (Bytes.equals(endKey, HConstants.EMPTY_END_ROW) ||
> getRowComparator(tableName).compareRows(
> endKey, 0, endKey.length, row, 0, row.length) > 0) {
> if (metrics != null) metrics.incrMetaCacheHit();
> return possibleRegion;
> }
> // Passed all the way through, so we got nothing - complete cache miss
> if (metrics != null) metrics.incrMetaCacheMiss();
> return null;
> }
> {code}
> It will choose the first location to be possibleRegion and possibly it will
> miss match.
> So did I forget something or may be wrong somewhere? If this is indeed a bug
> I think it can be fixed not very hard.
> Hope commiters and PMC review this !
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20697) Can't cache All region locations of the specify table by calling table.getRegionLocator().getAllRegionLocations()
[
https://issues.apache.org/jira/browse/HBASE-20697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541208#comment-16541208
]
Guanghao Zhang commented on HBASE-20697:
Pushed the addenum patch to branch-1 and branch-1.4. Thanks [~apurtell] for
reviewing.
> Can't cache All region locations of the specify table by calling
> table.getRegionLocator().getAllRegionLocations()
> -
>
> Key: HBASE-20697
> URL: https://issues.apache.org/jira/browse/HBASE-20697
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.1, 1.2.6, 2.0.1
>Reporter: zhaoyuan
>Assignee: zhaoyuan
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.4.6, 2.0.2, 2.2.0, 2.1.1
>
> Attachments: HBASE-20697.branch-1.2.001.patch,
> HBASE-20697.branch-1.2.002.patch, HBASE-20697.branch-1.2.003.patch,
> HBASE-20697.branch-1.2.004.patch, HBASE-20697.branch-1.addendum.patch,
> HBASE-20697.master.001.patch, HBASE-20697.master.002.patch,
> HBASE-20697.master.002.patch, HBASE-20697.master.003.patch
>
>
> When we upgrade and restart a new version application which will read and
> write to HBase, we will get some operation timeout. The time out is expected
> because when the application restarts，It will not hold any region locations
> cache and do communication with zk and meta regionserver to get region
> locations.
> We want to avoid these timeouts so we do warmup work and as far as I am
> concerned,the method table.getRegionLocator().getAllRegionLocations() will
> fetch all region locations and cache them. However, it didn't work good.
> There are still a lot of time outs,so it confused me.
> I dig into the source code and find something below
> {code:java}
> // code placeholder
> public List getAllRegionLocations() throws IOException {
> TableName tableName = getName();
> NavigableMap locations =
> MetaScanner.allTableRegions(this.connection, tableName);
> ArrayList regions = new ArrayList<>(locations.size());
> for (Entry entry : locations.entrySet()) {
> regions.add(new HRegionLocation(entry.getKey(), entry.getValue()));
> }
> if (regions.size() > 0) {
> connection.cacheLocation(tableName, new RegionLocations(regions));
> }
> return regions;
> }
> In MetaCache
> public void cacheLocation(final TableName tableName, final RegionLocations
> locations) {
> byte [] startKey =
> locations.getRegionLocation().getRegionInfo().getStartKey();
> ConcurrentMap tableLocations =
> getTableLocations(tableName);
> RegionLocations oldLocation = tableLocations.putIfAbsent(startKey,
> locations);
> boolean isNewCacheEntry = (oldLocation == null);
> if (isNewCacheEntry) {
> if (LOG.isTraceEnabled()) {
> LOG.trace("Cached location: " + locations);
> }
> addToCachedServers(locations);
> return;
> }
> {code}
> It will collect all regions into one RegionLocations object and only cache
> the first not null region location and then when we put or get to hbase, we
> do getCacheLocation()
> {code:java}
> // code placeholder
> public RegionLocations getCachedLocation(final TableName tableName, final
> byte [] row) {
> ConcurrentNavigableMap tableLocations =
> getTableLocations(tableName);
> Entry e = tableLocations.floorEntry(row);
> if (e == null) {
> if (metrics!= null) metrics.incrMetaCacheMiss();
> return null;
> }
> RegionLocations possibleRegion = e.getValue();
> // make sure that the end key is greater than the row we're looking
> // for, otherwise the row actually belongs in the next region, not
> // this one. the exception case is when the endkey is
> // HConstants.EMPTY_END_ROW, signifying that the region we're
> // checking is actually the last region in the table.
> byte[] endKey =
> possibleRegion.getRegionLocation().getRegionInfo().getEndKey();
> if (Bytes.equals(endKey, HConstants.EMPTY_END_ROW) ||
> getRowComparator(tableName).compareRows(
> endKey, 0, endKey.length, row, 0, row.length) > 0) {
> if (metrics != null) metrics.incrMetaCacheHit();
> return possibleRegion;
> }
> // Passed all the way through, so we got nothing - complete cache miss
> if (metrics != null) metrics.incrMetaCacheMiss();
> return null;
> }
> {code}
> It will choose the first location to be possibleRegion and possibly it will
> miss match.
> So did I forget something or may be wrong somewhere? If this is indeed a bug
> I think it can be fixed not very hard.
> Hope commiters and PMC review this !
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19572) RegionMover should use the configured default port number and not the one from HConstants
[
https://issues.apache.org/jira/browse/HBASE-19572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541207#comment-16541207
]
Toshihiro Suzuki commented on HBASE-19572:
--
[~reidchan] I think we need the "conf" as a member of RegionMoverBuilder,
because its instance is created in RegionMoverBuilder and then it's passed to
RegionMover:
{code}
private RegionMover(RegionMoverBuilder builder) {
this.hostname = builder.hostname;
this.filename = builder.filename;
this.excludeFile = builder.excludeFile;
this.maxthreads = builder.maxthreads;
this.ack = builder.ack;
this.port = builder.port;
this.timeout = builder.timeout;
setConf(builder.conf); // here
}
{code}
If you agree, I'll upload a new patch. Thanks.
> RegionMover should use the configured default port number and not the one
> from HConstants
> -
>
> Key: HBASE-19572
> URL: https://issues.apache.org/jira/browse/HBASE-19572
> Project: HBase
> Issue Type: Bug
>Reporter: Esteban Gutierrez
>Assignee: Toshihiro Suzuki
>Priority: Major
> Attachments: HBASE-19572.master.001.patch,
> HBASE-19572.master.001.patch, HBASE-19572.master.003.patch,
> HBASE-19572.master.004.patch, HBASE-19572.master.004.patch,
> HBASE-19572.patch, HBASE-19572.patch
>
>
> The issue I ran into HBASE-19499 was due RegionMover not using the port used
> by {{hbase-site.xml}}. The tool should use the value used in the
> configuration before falling back to the hardcoded value
> {{HConstants.DEFAULT_REGIONSERVER_PORT}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-20876) Improve docs style in HConstants
Reid Chan created HBASE-20876:
-
Summary: Improve docs style in HConstants
Key: HBASE-20876
URL: https://issues.apache.org/jira/browse/HBASE-20876
Project: HBase
Issue Type: Improvement
Reporter: Reid Chan
In {{HConstants}}, there's a docs snippet:
{code}
/** Don't use it! This'll get you the wrong path in a secure cluster.
* Use FileSystem.getHomeDirectory() or
* "/user/" + UserGroupInformation.getCurrentUser().getShortUserName() */
{code}
It's ugly style.
Let's improve this docs with following
{code}
/**
* Description
*/
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-17885) Backport HBASE-15871 to branch-1
[ https://issues.apache.org/jira/browse/HBASE-17885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541198#comment-16541198 ] jamjae kim commented on HBASE-17885: [~minwoo.kang] Thanks, very helpful. Maybe Same situation.. Dump Situation was many delete operations and long runnging Reverse scan. I'm waiting user for "backport" to branch-1. My HBase version is 1.1.13(branch-1.1), 1.2.7(branch-1.2) > Backport HBASE-15871 to branch-1 > > > Key: HBASE-17885 > URL: https://issues.apache.org/jira/browse/HBASE-17885 > Project: HBase > Issue Type: Bug > Components: Scanners >Affects Versions: 1.3.1, 1.2.5, 1.1.8 >Reporter: ramkrishna.s.vasudevan >Priority: Major > Fix For: 1.5.0, 1.3.3, 1.2.8, 1.4.6 > > > Will try to rebase the branch-1 patch at the earliest. Hope the fix versions > are correct. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20838) Include hbase-server in precommit test if CommonFSUtils is changed
[
https://issues.apache.org/jira/browse/HBASE-20838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-20838:
--
Fix Version/s: 2.1.1
Thanks for the quick response sir [~Apache9], just pushed into branch-2.1
> Include hbase-server in precommit test if CommonFSUtils is changed
> --
>
> Key: HBASE-20838
> URL: https://issues.apache.org/jira/browse/HBASE-20838
> Project: HBase
> Issue Type: Test
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1
>
> Attachments: HBASE-20838.patch, HBASE-20838.patch,
> HBASE-20838.v2.patch
>
>
> -As per
> [discussed|https://issues.apache.org/jira/browse/HBASE-20691?focusedCommentId=16517662&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16517662]
> in HBASE-20691, since the setStoragePolicy code is in CommonFSUtils, the
> test should be in TestCommonFSUtils-
> We don't want to introduce dependency on hadoop-hdfs to hbase-common so
> decided to leave the setStoragePolicy related tests in TestCommonFSUtils.
> Instead, we will change the personality script to include hbase-server in
> unit test if any change made against {{CommonFSUtils}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20565) ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect result since 1.4
[ https://issues.apache.org/jira/browse/HBASE-20565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541187#comment-16541187 ] Jerry He commented on HBASE-20565: -- [~openinx] Is there any good news on this one? I think we need to fix it. Thanks. > ColumnRangeFilter combined with ColumnPaginationFilter can produce incorrect > result since 1.4 > - > > Key: HBASE-20565 > URL: https://issues.apache.org/jira/browse/HBASE-20565 > Project: HBase > Issue Type: Bug > Components: Filters >Affects Versions: 1.4.4 >Reporter: Jerry He >Assignee: Zheng Hu >Priority: Major > Attachments: debug.diff, debug.log, test-branch-1.4.patch > > > When ColumnPaginationFilter is combined with ColumnRangeFilter, we may see > incorrect result. > Here is a simple example. > One row with 10 columns c0, c1, c2, .., c9. I have a ColumnRangeFilter for > range c2 to c9. Then I have a ColumnPaginationFilter with limit 5 and offset > 0. FileterList is FilterList(Operator.MUST_PASS_ALL, ColumnRangeFilter, > ColumnPaginationFilter). > We expect 5 columns being returned. But in HBase 1.4 and after, 4 columns > are returned. > In 1.2.x, the correct 5 columns are returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20870) Wrong HBase root dir in ITBLL's Search Tool
[
https://issues.apache.org/jira/browse/HBASE-20870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541186#comment-16541186
]
Allan Yang commented on HBASE-20870:
[~mdrob] Just checked HBASE-19947, yes, you tried to resolve this problem in
that issue, but seems like there is still some problem.
This line was not committed
{code}
+this.conf.set("original.hbase.dir", this.conf.get(HConstants.HBASE_DIR));
{code}
And HConstants.HBASE_DIR is not restored in IntegrationTestingUtility
> Wrong HBase root dir in ITBLL's Search Tool
> ---
>
> Key: HBASE-20870
> URL: https://issues.apache.org/jira/browse/HBASE-20870
> Project: HBase
> Issue Type: Bug
> Components: integration tests
>Affects Versions: 3.0.0, 2.1.0, 2.0.1
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Minor
> Attachments: HBASE-20870.branch-2.0.001.patch
>
>
> When using IntegrationTestBigLinkedList's Search tools, it always fails since
> it tries to read WALs in a wrong HBase root dir. Turned out that when
> initializing IntegrationTestingUtility in IntegrationTestBigLinkedList, its
> super class HBaseTestingUtility will change hbase.rootdir to a local random
> dir. It is not wrong since HBaseTestingUtility is mostly used by Minicluster.
> But for IntegrationTest runs on distributed clusters, we should change it
> back.
> Here is the error info.
> {code:java}
> 2018-07-11 16:35:49,679 DEBUG [main] hbase.HBaseCommonTestingUtility: Setting
> hbase.rootdir to
> /home/hadoop/target/test-data/deb67611-2737-4696-abe9-32a7783df7bb
> 2018-07-11 16:35:50,736 ERROR [main] util.AbstractHBaseTool: Error running
> command-line tool java.io.FileNotFoundException: File
> file:/home/hadoop/target/test-data/deb67611-2737-4696-abe9-32a7783df7bb/WALs
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:431)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1517)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20866) HBase 1.x scan performance degradation compared to 0.98 version
[
https://issues.apache.org/jira/browse/HBASE-20866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikas Vishwakarma updated HBASE-20866:
--
Attachment: HBASE-20866.branch-1.3.004.patch
> HBase 1.x scan performance degradation compared to 0.98 version
> ---
>
> Key: HBASE-20866
> URL: https://issues.apache.org/jira/browse/HBASE-20866
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.2
>Reporter: Vikas Vishwakarma
>Assignee: Vikas Vishwakarma
>Priority: Critical
> Fix For: 1.5.0, 1.2.7, 1.3.3, 1.4.6
>
> Attachments: HBASE-20866.branch-1.3.001.patch,
> HBASE-20866.branch-1.3.002.patch, HBASE-20866.branch-1.3.003.patch
>
>
> Internally while testing 1.3 as part of migration from 0.98 to 1.3 we
> observed perf degradation in scan performance for phoenix queries varying
> from few 10's to upto 200% depending on the query being executed. We tried
> simple native HBase scan and there also we saw upto 40% degradation in
> performance when the number of column qualifiers are high (40-50+)
> To identify the root cause of performance diff between 0.98 and 1.3 we
> carried out lot of experiments with profiling and git bisect iterations,
> however we were not able to identify any particular source of scan
> performance degradation and it looked like this is an accumulated degradation
> of 5-10% over various enhancements and refactoring.
> We identified few major enhancements like partialResult handling,
> ScannerContext with heartbeat processing, time/size limiting, RPC
> refactoring, etc that could have contributed to small degradation in
> performance which put together could be leading to large overall degradation.
> One of the changes is
> [HBASE-11544|https://jira.apache.org/jira/browse/HBASE-11544] which
> implements partialResult handling. In ClientScanner.java the results received
> from server are cached on the client side by converting the result array into
> an ArrayList. This function gets called in a loop depending on the number of
> rows in the scan result. Example for ten’s of millions of rows scanned, this
> can be called in the order of millions of times.
> In almost all the cases 99% of the time (except for handling partial results,
> etc). We are just taking the resultsFromServer converting it into a ArrayList
> resultsToAddToCache in addResultsToList(..) and then iterating over the list
> again and adding it to cache in loadCache(..) as given in the code path below
> In ClientScanner → loadCache(..) → getResultsToAddToCache(..) →
> addResultsToList(..) →
> {code:java}
> loadCache() {
> ...
> List resultsToAddToCache =
> getResultsToAddToCache(values, callable.isHeartbeatMessage());
> ...
> …
> for (Result rs : resultsToAddToCache) {
> rs = filterLoadedCell(rs);
> cache.add(rs);
> ...
> }
> }
> getResultsToAddToCache(..) {
> ..
> final boolean isBatchSet = scan != null && scan.getBatch() > 0;
> final boolean allowPartials = scan != null &&
> scan.getAllowPartialResults();
> ..
> if (allowPartials || isBatchSet) {
> addResultsToList(resultsToAddToCache, resultsFromServer, 0,
> (null == resultsFromServer ? 0 : resultsFromServer.length));
> return resultsToAddToCache;
> }
> ...
> }
> private void addResultsToList(List outputList, Result[] inputArray,
> int start, int end) {
> if (inputArray == null || start < 0 || end > inputArray.length) return;
> for (int i = start; i < end; i++) {
> outputList.add(inputArray[i]);
> }
> }{code}
>
> It looks like we can avoid the result array to arraylist conversion
> (resultsFromServer --> resultsToAddToCache ) for the first case which is also
> the most frequent case and instead directly take the values arraay returned
> by callable and add it to the cache without converting it into ArrayList.
> I have taken both these flags allowPartials and isBatchSet out in loadcahe()
> and I am directly adding values to scanner cache if the above condition is
> pass instead of coverting it into arrayList by calling
> getResultsToAddToCache(). For example:
> {code:java}
> protected void loadCache() throws IOException {
> Result[] values = null;
> ..
> final boolean isBatchSet = scan != null && scan.getBatch() > 0;
> final boolean allowPartials = scan != null && scan.getAllowPartialResults();
> ..
> for (;;) {
> try {
> values = call(callable, caller, scannerTimeout);
> ..
> } catch (DoNotRetryIOException | NeedUnmanagedConnectionException e) {
> ..
> }
> if (allowPartials || isBatchSet) { // DIRECTLY COPY values TO CACHE
> if (values != null) {
> for (int v=0; v Result rs = values[v];
>
> cache.add(rs);
> ...
> } else { // DO ALL THE REGULAR PARTIAL RESULT HANDLING ..
> List resultsToAddToCache =
> getResul
[jira] [Updated] (HBASE-20866) HBase 1.x scan performance degradation compared to 0.98 version
[
https://issues.apache.org/jira/browse/HBASE-20866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikas Vishwakarma updated HBASE-20866:
--
Attachment: (was: HBASE-20866.branch-1.3.004.patch)
> HBase 1.x scan performance degradation compared to 0.98 version
> ---
>
> Key: HBASE-20866
> URL: https://issues.apache.org/jira/browse/HBASE-20866
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.2
>Reporter: Vikas Vishwakarma
>Assignee: Vikas Vishwakarma
>Priority: Critical
> Fix For: 1.5.0, 1.2.7, 1.3.3, 1.4.6
>
> Attachments: HBASE-20866.branch-1.3.001.patch,
> HBASE-20866.branch-1.3.002.patch, HBASE-20866.branch-1.3.003.patch
>
>
> Internally while testing 1.3 as part of migration from 0.98 to 1.3 we
> observed perf degradation in scan performance for phoenix queries varying
> from few 10's to upto 200% depending on the query being executed. We tried
> simple native HBase scan and there also we saw upto 40% degradation in
> performance when the number of column qualifiers are high (40-50+)
> To identify the root cause of performance diff between 0.98 and 1.3 we
> carried out lot of experiments with profiling and git bisect iterations,
> however we were not able to identify any particular source of scan
> performance degradation and it looked like this is an accumulated degradation
> of 5-10% over various enhancements and refactoring.
> We identified few major enhancements like partialResult handling,
> ScannerContext with heartbeat processing, time/size limiting, RPC
> refactoring, etc that could have contributed to small degradation in
> performance which put together could be leading to large overall degradation.
> One of the changes is
> [HBASE-11544|https://jira.apache.org/jira/browse/HBASE-11544] which
> implements partialResult handling. In ClientScanner.java the results received
> from server are cached on the client side by converting the result array into
> an ArrayList. This function gets called in a loop depending on the number of
> rows in the scan result. Example for ten’s of millions of rows scanned, this
> can be called in the order of millions of times.
> In almost all the cases 99% of the time (except for handling partial results,
> etc). We are just taking the resultsFromServer converting it into a ArrayList
> resultsToAddToCache in addResultsToList(..) and then iterating over the list
> again and adding it to cache in loadCache(..) as given in the code path below
> In ClientScanner → loadCache(..) → getResultsToAddToCache(..) →
> addResultsToList(..) →
> {code:java}
> loadCache() {
> ...
> List resultsToAddToCache =
> getResultsToAddToCache(values, callable.isHeartbeatMessage());
> ...
> …
> for (Result rs : resultsToAddToCache) {
> rs = filterLoadedCell(rs);
> cache.add(rs);
> ...
> }
> }
> getResultsToAddToCache(..) {
> ..
> final boolean isBatchSet = scan != null && scan.getBatch() > 0;
> final boolean allowPartials = scan != null &&
> scan.getAllowPartialResults();
> ..
> if (allowPartials || isBatchSet) {
> addResultsToList(resultsToAddToCache, resultsFromServer, 0,
> (null == resultsFromServer ? 0 : resultsFromServer.length));
> return resultsToAddToCache;
> }
> ...
> }
> private void addResultsToList(List outputList, Result[] inputArray,
> int start, int end) {
> if (inputArray == null || start < 0 || end > inputArray.length) return;
> for (int i = start; i < end; i++) {
> outputList.add(inputArray[i]);
> }
> }{code}
>
> It looks like we can avoid the result array to arraylist conversion
> (resultsFromServer --> resultsToAddToCache ) for the first case which is also
> the most frequent case and instead directly take the values arraay returned
> by callable and add it to the cache without converting it into ArrayList.
> I have taken both these flags allowPartials and isBatchSet out in loadcahe()
> and I am directly adding values to scanner cache if the above condition is
> pass instead of coverting it into arrayList by calling
> getResultsToAddToCache(). For example:
> {code:java}
> protected void loadCache() throws IOException {
> Result[] values = null;
> ..
> final boolean isBatchSet = scan != null && scan.getBatch() > 0;
> final boolean allowPartials = scan != null && scan.getAllowPartialResults();
> ..
> for (;;) {
> try {
> values = call(callable, caller, scannerTimeout);
> ..
> } catch (DoNotRetryIOException | NeedUnmanagedConnectionException e) {
> ..
> }
> if (allowPartials || isBatchSet) { // DIRECTLY COPY values TO CACHE
> if (values != null) {
> for (int v=0; v Result rs = values[v];
>
> cache.add(rs);
> ...
> } else { // DO ALL THE REGULAR PARTIAL RESULT HANDLING ..
> List resultsToAddToCache =
[jira] [Commented] (HBASE-20866) HBase 1.x scan performance degradation compared to 0.98 version
[
https://issues.apache.org/jira/browse/HBASE-20866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541158#comment-16541158
]
Vikas Vishwakarma commented on HBASE-20866:
---
thanks for the review and suggestions.
I have just committed a change with a trivial change in HRegion to get a full
QA run as suggested by [~yuzhih...@gmail.com]
I will also put up the patch on RB and take a look at the master branch
> HBase 1.x scan performance degradation compared to 0.98 version
> ---
>
> Key: HBASE-20866
> URL: https://issues.apache.org/jira/browse/HBASE-20866
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.2
>Reporter: Vikas Vishwakarma
>Assignee: Vikas Vishwakarma
>Priority: Critical
> Fix For: 1.5.0, 1.2.7, 1.3.3, 1.4.6
>
> Attachments: HBASE-20866.branch-1.3.001.patch,
> HBASE-20866.branch-1.3.002.patch, HBASE-20866.branch-1.3.003.patch
>
>
> Internally while testing 1.3 as part of migration from 0.98 to 1.3 we
> observed perf degradation in scan performance for phoenix queries varying
> from few 10's to upto 200% depending on the query being executed. We tried
> simple native HBase scan and there also we saw upto 40% degradation in
> performance when the number of column qualifiers are high (40-50+)
> To identify the root cause of performance diff between 0.98 and 1.3 we
> carried out lot of experiments with profiling and git bisect iterations,
> however we were not able to identify any particular source of scan
> performance degradation and it looked like this is an accumulated degradation
> of 5-10% over various enhancements and refactoring.
> We identified few major enhancements like partialResult handling,
> ScannerContext with heartbeat processing, time/size limiting, RPC
> refactoring, etc that could have contributed to small degradation in
> performance which put together could be leading to large overall degradation.
> One of the changes is
> [HBASE-11544|https://jira.apache.org/jira/browse/HBASE-11544] which
> implements partialResult handling. In ClientScanner.java the results received
> from server are cached on the client side by converting the result array into
> an ArrayList. This function gets called in a loop depending on the number of
> rows in the scan result. Example for ten’s of millions of rows scanned, this
> can be called in the order of millions of times.
> In almost all the cases 99% of the time (except for handling partial results,
> etc). We are just taking the resultsFromServer converting it into a ArrayList
> resultsToAddToCache in addResultsToList(..) and then iterating over the list
> again and adding it to cache in loadCache(..) as given in the code path below
> In ClientScanner → loadCache(..) → getResultsToAddToCache(..) →
> addResultsToList(..) →
> {code:java}
> loadCache() {
> ...
> List resultsToAddToCache =
> getResultsToAddToCache(values, callable.isHeartbeatMessage());
> ...
> …
> for (Result rs : resultsToAddToCache) {
> rs = filterLoadedCell(rs);
> cache.add(rs);
> ...
> }
> }
> getResultsToAddToCache(..) {
> ..
> final boolean isBatchSet = scan != null && scan.getBatch() > 0;
> final boolean allowPartials = scan != null &&
> scan.getAllowPartialResults();
> ..
> if (allowPartials || isBatchSet) {
> addResultsToList(resultsToAddToCache, resultsFromServer, 0,
> (null == resultsFromServer ? 0 : resultsFromServer.length));
> return resultsToAddToCache;
> }
> ...
> }
> private void addResultsToList(List outputList, Result[] inputArray,
> int start, int end) {
> if (inputArray == null || start < 0 || end > inputArray.length) return;
> for (int i = start; i < end; i++) {
> outputList.add(inputArray[i]);
> }
> }{code}
>
> It looks like we can avoid the result array to arraylist conversion
> (resultsFromServer --> resultsToAddToCache ) for the first case which is also
> the most frequent case and instead directly take the values arraay returned
> by callable and add it to the cache without converting it into ArrayList.
> I have taken both these flags allowPartials and isBatchSet out in loadcahe()
> and I am directly adding values to scanner cache if the above condition is
> pass instead of coverting it into arrayList by calling
> getResultsToAddToCache(). For example:
> {code:java}
> protected void loadCache() throws IOException {
> Result[] values = null;
> ..
> final boolean isBatchSet = scan != null && scan.getBatch() > 0;
> final boolean allowPartials = scan != null && scan.getAllowPartialResults();
> ..
> for (;;) {
> try {
> values = call(callable, caller, scannerTimeout);
> ..
> } catch (DoNotRetryIOException | NeedUnmanagedConnectionException e) {
> ..
> }
> if (allowPartial
[jira] [Updated] (HBASE-20866) HBase 1.x scan performance degradation compared to 0.98 version
[
https://issues.apache.org/jira/browse/HBASE-20866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vikas Vishwakarma updated HBASE-20866:
--
Attachment: HBASE-20866.branch-1.3.003.patch
> HBase 1.x scan performance degradation compared to 0.98 version
> ---
>
> Key: HBASE-20866
> URL: https://issues.apache.org/jira/browse/HBASE-20866
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.2
>Reporter: Vikas Vishwakarma
>Assignee: Vikas Vishwakarma
>Priority: Critical
> Fix For: 1.5.0, 1.2.7, 1.3.3, 1.4.6
>
> Attachments: HBASE-20866.branch-1.3.001.patch,
> HBASE-20866.branch-1.3.002.patch, HBASE-20866.branch-1.3.003.patch
>
>
> Internally while testing 1.3 as part of migration from 0.98 to 1.3 we
> observed perf degradation in scan performance for phoenix queries varying
> from few 10's to upto 200% depending on the query being executed. We tried
> simple native HBase scan and there also we saw upto 40% degradation in
> performance when the number of column qualifiers are high (40-50+)
> To identify the root cause of performance diff between 0.98 and 1.3 we
> carried out lot of experiments with profiling and git bisect iterations,
> however we were not able to identify any particular source of scan
> performance degradation and it looked like this is an accumulated degradation
> of 5-10% over various enhancements and refactoring.
> We identified few major enhancements like partialResult handling,
> ScannerContext with heartbeat processing, time/size limiting, RPC
> refactoring, etc that could have contributed to small degradation in
> performance which put together could be leading to large overall degradation.
> One of the changes is
> [HBASE-11544|https://jira.apache.org/jira/browse/HBASE-11544] which
> implements partialResult handling. In ClientScanner.java the results received
> from server are cached on the client side by converting the result array into
> an ArrayList. This function gets called in a loop depending on the number of
> rows in the scan result. Example for ten’s of millions of rows scanned, this
> can be called in the order of millions of times.
> In almost all the cases 99% of the time (except for handling partial results,
> etc). We are just taking the resultsFromServer converting it into a ArrayList
> resultsToAddToCache in addResultsToList(..) and then iterating over the list
> again and adding it to cache in loadCache(..) as given in the code path below
> In ClientScanner → loadCache(..) → getResultsToAddToCache(..) →
> addResultsToList(..) →
> {code:java}
> loadCache() {
> ...
> List resultsToAddToCache =
> getResultsToAddToCache(values, callable.isHeartbeatMessage());
> ...
> …
> for (Result rs : resultsToAddToCache) {
> rs = filterLoadedCell(rs);
> cache.add(rs);
> ...
> }
> }
> getResultsToAddToCache(..) {
> ..
> final boolean isBatchSet = scan != null && scan.getBatch() > 0;
> final boolean allowPartials = scan != null &&
> scan.getAllowPartialResults();
> ..
> if (allowPartials || isBatchSet) {
> addResultsToList(resultsToAddToCache, resultsFromServer, 0,
> (null == resultsFromServer ? 0 : resultsFromServer.length));
> return resultsToAddToCache;
> }
> ...
> }
> private void addResultsToList(List outputList, Result[] inputArray,
> int start, int end) {
> if (inputArray == null || start < 0 || end > inputArray.length) return;
> for (int i = start; i < end; i++) {
> outputList.add(inputArray[i]);
> }
> }{code}
>
> It looks like we can avoid the result array to arraylist conversion
> (resultsFromServer --> resultsToAddToCache ) for the first case which is also
> the most frequent case and instead directly take the values arraay returned
> by callable and add it to the cache without converting it into ArrayList.
> I have taken both these flags allowPartials and isBatchSet out in loadcahe()
> and I am directly adding values to scanner cache if the above condition is
> pass instead of coverting it into arrayList by calling
> getResultsToAddToCache(). For example:
> {code:java}
> protected void loadCache() throws IOException {
> Result[] values = null;
> ..
> final boolean isBatchSet = scan != null && scan.getBatch() > 0;
> final boolean allowPartials = scan != null && scan.getAllowPartialResults();
> ..
> for (;;) {
> try {
> values = call(callable, caller, scannerTimeout);
> ..
> } catch (DoNotRetryIOException | NeedUnmanagedConnectionException e) {
> ..
> }
> if (allowPartials || isBatchSet) { // DIRECTLY COPY values TO CACHE
> if (values != null) {
> for (int v=0; v Result rs = values[v];
>
> cache.add(rs);
> ...
> } else { // DO ALL THE REGULAR PARTIAL RESULT HANDLING ..
> List resultsToAddToCache =
> getResul
[jira] [Commented] (HBASE-20847) The parent procedure of RegionTransitionProcedure may not have the table lock
[
https://issues.apache.org/jira/browse/HBASE-20847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541154#comment-16541154
]
Hudson commented on HBASE-20847:
Results for branch branch-2
[build #972 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/972/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/972//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/972//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/972//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> The parent procedure of RegionTransitionProcedure may not have the table lock
> -
>
> Key: HBASE-20847
> URL: https://issues.apache.org/jira/browse/HBASE-20847
> Project: HBase
> Issue Type: Sub-task
> Components: proc-v2, Region Assignment
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1
>
> Attachments: HBASE-20847-addendum.patch,
> HBASE-20847-branch-2.0-v1.patch, HBASE-20847-branch-2.0.patch,
> HBASE-20847-v1.patch, HBASE-20847-v2.patch, HBASE-20847-v3.patch,
> HBASE-20847.patch
>
>
> For example, SCP can also schedule AssignProcedure and obviously it will not
> hold the table lock.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19572) RegionMover should use the configured default port number and not the one from HConstants
[
https://issues.apache.org/jira/browse/HBASE-19572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541150#comment-16541150
]
Reid Chan commented on HBASE-19572:
---
Oh ye, it's {{RegionMoverBuilder}}.
{code}
+private static Configuration createConf() {
+ Configuration conf = HBaseConfiguration.create();
+ conf.setInt("hbase.client.prefetch.limit", 1);
+ conf.setInt("hbase.client.pause", 500);
+ conf.setInt("hbase.client.retries.number", 100);
+ return conf;
+}
{code}
I think there's a conf in super class {{AbstractHBaseTool.conf}}, should we
create a new one unless is null? Following should be fine, (origin one also has
problem).
{code}
+private static Configuration createConf() {
+ if (conf == null) {
+conf = HBaseConfiguration.create();
+ }
+ conf.setInt("hbase.client.prefetch.limit", 1);
+ conf.setInt("hbase.client.pause", 500);
+ conf.setInt("hbase.client.retries.number", 100);
+ return conf;
+}
{code}
bq. What did you mean?
{code}
+ } else {
+this.port = conf.getInt(HConstants.REGIONSERVER_PORT,
HConstants.DEFAULT_REGIONSERVER_PORT);
}
setDefaultfilename(this.hostname);
+ this.conf = conf;
{code}
I only see it used for initializing port and does nothing afterwards, so i
doubt it to be a member of {{RegionMoverBuilder}}.
> RegionMover should use the configured default port number and not the one
> from HConstants
> -
>
> Key: HBASE-19572
> URL: https://issues.apache.org/jira/browse/HBASE-19572
> Project: HBase
> Issue Type: Bug
>Reporter: Esteban Gutierrez
>Assignee: Toshihiro Suzuki
>Priority: Major
> Attachments: HBASE-19572.master.001.patch,
> HBASE-19572.master.001.patch, HBASE-19572.master.003.patch,
> HBASE-19572.master.004.patch, HBASE-19572.master.004.patch,
> HBASE-19572.patch, HBASE-19572.patch
>
>
> The issue I ran into HBASE-19499 was due RegionMover not using the port used
> by {{hbase-site.xml}}. The tool should use the value used in the
> configuration before falling back to the hardcoded value
> {{HConstants.DEFAULT_REGIONSERVER_PORT}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[
https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541148#comment-16541148
]
Hadoop QA commented on HBASE-20704:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
5s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
6s{color} | {color:red} hbase-server: The patch generated 7 new + 43 unchanged
- 0 fixed = 50 total (was 43) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
16s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 23s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}171m
51s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}210m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20704 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12931253/HBASE-20704.003.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 33abdcb37624 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14
08:53:28 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 4a08135dd8 |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/13601/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13601/testReport/ |
| Max. process+thread count | 5204 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://b
[jira] [Commented] (HBASE-20697) Can't cache All region locations of the specify table by calling table.getRegionLocator().getAllRegionLocations()
[
https://issues.apache.org/jira/browse/HBASE-20697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541147#comment-16541147
]
Andrew Purtell commented on HBASE-20697:
+1 on addendum, thanks
> Can't cache All region locations of the specify table by calling
> table.getRegionLocator().getAllRegionLocations()
> -
>
> Key: HBASE-20697
> URL: https://issues.apache.org/jira/browse/HBASE-20697
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.1, 1.2.6, 2.0.1
>Reporter: zhaoyuan
>Assignee: zhaoyuan
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.4.6, 2.0.2, 2.2.0, 2.1.1
>
> Attachments: HBASE-20697.branch-1.2.001.patch,
> HBASE-20697.branch-1.2.002.patch, HBASE-20697.branch-1.2.003.patch,
> HBASE-20697.branch-1.2.004.patch, HBASE-20697.branch-1.addendum.patch,
> HBASE-20697.master.001.patch, HBASE-20697.master.002.patch,
> HBASE-20697.master.002.patch, HBASE-20697.master.003.patch
>
>
> When we upgrade and restart a new version application which will read and
> write to HBase, we will get some operation timeout. The time out is expected
> because when the application restarts，It will not hold any region locations
> cache and do communication with zk and meta regionserver to get region
> locations.
> We want to avoid these timeouts so we do warmup work and as far as I am
> concerned,the method table.getRegionLocator().getAllRegionLocations() will
> fetch all region locations and cache them. However, it didn't work good.
> There are still a lot of time outs,so it confused me.
> I dig into the source code and find something below
> {code:java}
> // code placeholder
> public List getAllRegionLocations() throws IOException {
> TableName tableName = getName();
> NavigableMap locations =
> MetaScanner.allTableRegions(this.connection, tableName);
> ArrayList regions = new ArrayList<>(locations.size());
> for (Entry entry : locations.entrySet()) {
> regions.add(new HRegionLocation(entry.getKey(), entry.getValue()));
> }
> if (regions.size() > 0) {
> connection.cacheLocation(tableName, new RegionLocations(regions));
> }
> return regions;
> }
> In MetaCache
> public void cacheLocation(final TableName tableName, final RegionLocations
> locations) {
> byte [] startKey =
> locations.getRegionLocation().getRegionInfo().getStartKey();
> ConcurrentMap tableLocations =
> getTableLocations(tableName);
> RegionLocations oldLocation = tableLocations.putIfAbsent(startKey,
> locations);
> boolean isNewCacheEntry = (oldLocation == null);
> if (isNewCacheEntry) {
> if (LOG.isTraceEnabled()) {
> LOG.trace("Cached location: " + locations);
> }
> addToCachedServers(locations);
> return;
> }
> {code}
> It will collect all regions into one RegionLocations object and only cache
> the first not null region location and then when we put or get to hbase, we
> do getCacheLocation()
> {code:java}
> // code placeholder
> public RegionLocations getCachedLocation(final TableName tableName, final
> byte [] row) {
> ConcurrentNavigableMap tableLocations =
> getTableLocations(tableName);
> Entry e = tableLocations.floorEntry(row);
> if (e == null) {
> if (metrics!= null) metrics.incrMetaCacheMiss();
> return null;
> }
> RegionLocations possibleRegion = e.getValue();
> // make sure that the end key is greater than the row we're looking
> // for, otherwise the row actually belongs in the next region, not
> // this one. the exception case is when the endkey is
> // HConstants.EMPTY_END_ROW, signifying that the region we're
> // checking is actually the last region in the table.
> byte[] endKey =
> possibleRegion.getRegionLocation().getRegionInfo().getEndKey();
> if (Bytes.equals(endKey, HConstants.EMPTY_END_ROW) ||
> getRowComparator(tableName).compareRows(
> endKey, 0, endKey.length, row, 0, row.length) > 0) {
> if (metrics != null) metrics.incrMetaCacheHit();
> return possibleRegion;
> }
> // Passed all the way through, so we got nothing - complete cache miss
> if (metrics != null) metrics.incrMetaCacheMiss();
> return null;
> }
> {code}
> It will choose the first location to be possibleRegion and possibly it will
> miss match.
> So did I forget something or may be wrong somewhere? If this is indeed a bug
> I think it can be fixed not very hard.
> Hope commiters and PMC review this !
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-17877) Improve HBase's byte[] comparator
[ https://issues.apache.org/jira/browse/HBASE-17877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vikas Vishwakarma updated HBASE-17877: -- Attachment: HBASE-17877.branch-1.3.004.patch > Improve HBase's byte[] comparator > - > > Key: HBASE-17877 > URL: https://issues.apache.org/jira/browse/HBASE-17877 > Project: HBase > Issue Type: Improvement > Components: util >Reporter: Lars Hofhansl >Assignee: Vikas Vishwakarma >Priority: Major > Fix For: 1.4.0, 1.3.2, 2.0.0 > > Attachments: 17877-1.2.patch, 17877-v2-1.3.patch, 17877-v3-1.3.patch, > 17877-v4-1.3.patch, ByteComparatorJiraHBASE-17877.pdf, > HBASE-17877.branch-1.3.001.patch, HBASE-17877.branch-1.3.002.patch, > HBASE-17877.branch-1.3.003.patch, HBASE-17877.branch-1.3.004.patch, > HBASE-17877.master.001.patch, HBASE-17877.master.002.patch, > HBASE-17877.master.003.patch > > > [~vik.karma] did some extensive tests and found that Hadoop's version is > faster - dramatically faster in some cases. > Patch forthcoming. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20847) The parent procedure of RegionTransitionProcedure may not have the table lock
[
https://issues.apache.org/jira/browse/HBASE-20847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541134#comment-16541134
]
Hudson commented on HBASE-20847:
Results for branch branch-2.1
[build #50 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/50/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/50//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/50//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/50//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> The parent procedure of RegionTransitionProcedure may not have the table lock
> -
>
> Key: HBASE-20847
> URL: https://issues.apache.org/jira/browse/HBASE-20847
> Project: HBase
> Issue Type: Sub-task
> Components: proc-v2, Region Assignment
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.2.0, 2.1.1
>
> Attachments: HBASE-20847-addendum.patch,
> HBASE-20847-branch-2.0-v1.patch, HBASE-20847-branch-2.0.patch,
> HBASE-20847-v1.patch, HBASE-20847-v2.patch, HBASE-20847-v3.patch,
> HBASE-20847.patch
>
>
> For example, SCP can also schedule AssignProcedure and obviously it will not
> hold the table lock.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19572) RegionMover should use the configured default port number and not the one from HConstants
[
https://issues.apache.org/jira/browse/HBASE-19572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541128#comment-16541128
]
Toshihiro Suzuki commented on HBASE-19572:
--
Thank you for reviewing [~reidchan].
Regarding 1, If I remember correctly, I thought we needed to make "final int
port" package private for test, so I made all members package private for
consistency. But I don't think it's strong reason, I'll revert it.
Regarding 2, I think "final Configuration conf" is not a member of RegionMover,
but RegionMoverBuilder. What did you mean?
Thanks.
> RegionMover should use the configured default port number and not the one
> from HConstants
> -
>
> Key: HBASE-19572
> URL: https://issues.apache.org/jira/browse/HBASE-19572
> Project: HBase
> Issue Type: Bug
>Reporter: Esteban Gutierrez
>Assignee: Toshihiro Suzuki
>Priority: Major
> Attachments: HBASE-19572.master.001.patch,
> HBASE-19572.master.001.patch, HBASE-19572.master.003.patch,
> HBASE-19572.master.004.patch, HBASE-19572.master.004.patch,
> HBASE-19572.patch, HBASE-19572.patch
>
>
> The issue I ran into HBASE-19499 was due RegionMover not using the port used
> by {{hbase-site.xml}}. The tool should use the value used in the
> configuration before falling back to the hardcoded value
> {{HConstants.DEFAULT_REGIONSERVER_PORT}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20697) Can't cache All region locations of the specify table by calling table.getRegionLocator().getAllRegionLocations()
[
https://issues.apache.org/jira/browse/HBASE-20697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541127#comment-16541127
]
Hadoop QA commented on HBASE-20697:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
40s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} branch-1 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} branch-1 passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
39s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} branch-1 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} branch-1 passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} hbase-client: The patch generated 0 new + 1
unchanged - 4 fixed = 1 total (was 5) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
36s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
1m 35s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
22s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:1f3957d |
| JIRA Issue | HBASE-20697 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12931267/HBASE-20697.branch-1.addendum.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopchec
[jira] [Updated] (HBASE-20697) Can't cache All region locations of the specify table by calling table.getRegionLocator().getAllRegionLocations()
[
https://issues.apache.org/jira/browse/HBASE-20697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-20697:
---
Status: Patch Available (was: Reopened)
> Can't cache All region locations of the specify table by calling
> table.getRegionLocator().getAllRegionLocations()
> -
>
> Key: HBASE-20697
> URL: https://issues.apache.org/jira/browse/HBASE-20697
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.1, 1.2.6, 1.3.1
>Reporter: zhaoyuan
>Assignee: zhaoyuan
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.4.6, 2.0.2, 2.2.0, 2.1.1
>
> Attachments: HBASE-20697.branch-1.2.001.patch,
> HBASE-20697.branch-1.2.002.patch, HBASE-20697.branch-1.2.003.patch,
> HBASE-20697.branch-1.2.004.patch, HBASE-20697.branch-1.addendum.patch,
> HBASE-20697.master.001.patch, HBASE-20697.master.002.patch,
> HBASE-20697.master.002.patch, HBASE-20697.master.003.patch
>
>
> When we upgrade and restart a new version application which will read and
> write to HBase, we will get some operation timeout. The time out is expected
> because when the application restarts，It will not hold any region locations
> cache and do communication with zk and meta regionserver to get region
> locations.
> We want to avoid these timeouts so we do warmup work and as far as I am
> concerned,the method table.getRegionLocator().getAllRegionLocations() will
> fetch all region locations and cache them. However, it didn't work good.
> There are still a lot of time outs,so it confused me.
> I dig into the source code and find something below
> {code:java}
> // code placeholder
> public List getAllRegionLocations() throws IOException {
> TableName tableName = getName();
> NavigableMap locations =
> MetaScanner.allTableRegions(this.connection, tableName);
> ArrayList regions = new ArrayList<>(locations.size());
> for (Entry entry : locations.entrySet()) {
> regions.add(new HRegionLocation(entry.getKey(), entry.getValue()));
> }
> if (regions.size() > 0) {
> connection.cacheLocation(tableName, new RegionLocations(regions));
> }
> return regions;
> }
> In MetaCache
> public void cacheLocation(final TableName tableName, final RegionLocations
> locations) {
> byte [] startKey =
> locations.getRegionLocation().getRegionInfo().getStartKey();
> ConcurrentMap tableLocations =
> getTableLocations(tableName);
> RegionLocations oldLocation = tableLocations.putIfAbsent(startKey,
> locations);
> boolean isNewCacheEntry = (oldLocation == null);
> if (isNewCacheEntry) {
> if (LOG.isTraceEnabled()) {
> LOG.trace("Cached location: " + locations);
> }
> addToCachedServers(locations);
> return;
> }
> {code}
> It will collect all regions into one RegionLocations object and only cache
> the first not null region location and then when we put or get to hbase, we
> do getCacheLocation()
> {code:java}
> // code placeholder
> public RegionLocations getCachedLocation(final TableName tableName, final
> byte [] row) {
> ConcurrentNavigableMap tableLocations =
> getTableLocations(tableName);
> Entry e = tableLocations.floorEntry(row);
> if (e == null) {
> if (metrics!= null) metrics.incrMetaCacheMiss();
> return null;
> }
> RegionLocations possibleRegion = e.getValue();
> // make sure that the end key is greater than the row we're looking
> // for, otherwise the row actually belongs in the next region, not
> // this one. the exception case is when the endkey is
> // HConstants.EMPTY_END_ROW, signifying that the region we're
> // checking is actually the last region in the table.
> byte[] endKey =
> possibleRegion.getRegionLocation().getRegionInfo().getEndKey();
> if (Bytes.equals(endKey, HConstants.EMPTY_END_ROW) ||
> getRowComparator(tableName).compareRows(
> endKey, 0, endKey.length, row, 0, row.length) > 0) {
> if (metrics != null) metrics.incrMetaCacheHit();
> return possibleRegion;
> }
> // Passed all the way through, so we got nothing - complete cache miss
> if (metrics != null) metrics.incrMetaCacheMiss();
> return null;
> }
> {code}
> It will choose the first location to be possibleRegion and possibly it will
> miss match.
> So did I forget something or may be wrong somewhere? If this is indeed a bug
> I think it can be fixed not very hard.
> Hope commiters and PMC review this !
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20697) Can't cache All region locations of the specify table by calling table.getRegionLocator().getAllRegionLocations()
[
https://issues.apache.org/jira/browse/HBASE-20697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-20697:
---
Attachment: HBASE-20697.branch-1.addendum.patch
> Can't cache All region locations of the specify table by calling
> table.getRegionLocator().getAllRegionLocations()
> -
>
> Key: HBASE-20697
> URL: https://issues.apache.org/jira/browse/HBASE-20697
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.1, 1.2.6, 2.0.1
>Reporter: zhaoyuan
>Assignee: zhaoyuan
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.4.6, 2.0.2, 2.2.0, 2.1.1
>
> Attachments: HBASE-20697.branch-1.2.001.patch,
> HBASE-20697.branch-1.2.002.patch, HBASE-20697.branch-1.2.003.patch,
> HBASE-20697.branch-1.2.004.patch, HBASE-20697.branch-1.addendum.patch,
> HBASE-20697.master.001.patch, HBASE-20697.master.002.patch,
> HBASE-20697.master.002.patch, HBASE-20697.master.003.patch
>
>
> When we upgrade and restart a new version application which will read and
> write to HBase, we will get some operation timeout. The time out is expected
> because when the application restarts，It will not hold any region locations
> cache and do communication with zk and meta regionserver to get region
> locations.
> We want to avoid these timeouts so we do warmup work and as far as I am
> concerned,the method table.getRegionLocator().getAllRegionLocations() will
> fetch all region locations and cache them. However, it didn't work good.
> There are still a lot of time outs,so it confused me.
> I dig into the source code and find something below
> {code:java}
> // code placeholder
> public List getAllRegionLocations() throws IOException {
> TableName tableName = getName();
> NavigableMap locations =
> MetaScanner.allTableRegions(this.connection, tableName);
> ArrayList regions = new ArrayList<>(locations.size());
> for (Entry entry : locations.entrySet()) {
> regions.add(new HRegionLocation(entry.getKey(), entry.getValue()));
> }
> if (regions.size() > 0) {
> connection.cacheLocation(tableName, new RegionLocations(regions));
> }
> return regions;
> }
> In MetaCache
> public void cacheLocation(final TableName tableName, final RegionLocations
> locations) {
> byte [] startKey =
> locations.getRegionLocation().getRegionInfo().getStartKey();
> ConcurrentMap tableLocations =
> getTableLocations(tableName);
> RegionLocations oldLocation = tableLocations.putIfAbsent(startKey,
> locations);
> boolean isNewCacheEntry = (oldLocation == null);
> if (isNewCacheEntry) {
> if (LOG.isTraceEnabled()) {
> LOG.trace("Cached location: " + locations);
> }
> addToCachedServers(locations);
> return;
> }
> {code}
> It will collect all regions into one RegionLocations object and only cache
> the first not null region location and then when we put or get to hbase, we
> do getCacheLocation()
> {code:java}
> // code placeholder
> public RegionLocations getCachedLocation(final TableName tableName, final
> byte [] row) {
> ConcurrentNavigableMap tableLocations =
> getTableLocations(tableName);
> Entry e = tableLocations.floorEntry(row);
> if (e == null) {
> if (metrics!= null) metrics.incrMetaCacheMiss();
> return null;
> }
> RegionLocations possibleRegion = e.getValue();
> // make sure that the end key is greater than the row we're looking
> // for, otherwise the row actually belongs in the next region, not
> // this one. the exception case is when the endkey is
> // HConstants.EMPTY_END_ROW, signifying that the region we're
> // checking is actually the last region in the table.
> byte[] endKey =
> possibleRegion.getRegionLocation().getRegionInfo().getEndKey();
> if (Bytes.equals(endKey, HConstants.EMPTY_END_ROW) ||
> getRowComparator(tableName).compareRows(
> endKey, 0, endKey.length, row, 0, row.length) > 0) {
> if (metrics != null) metrics.incrMetaCacheHit();
> return possibleRegion;
> }
> // Passed all the way through, so we got nothing - complete cache miss
> if (metrics != null) metrics.incrMetaCacheMiss();
> return null;
> }
> {code}
> It will choose the first location to be possibleRegion and possibly it will
> miss match.
> So did I forget something or may be wrong somewhere? If this is indeed a bug
> I think it can be fixed not very hard.
> Hope commiters and PMC review this !
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20679) Add the ability to compile JSP dynamically in Jetty
[
https://issues.apache.org/jira/browse/HBASE-20679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541098#comment-16541098
]
Allan Yang commented on HBASE-20679:
[~stack], are we still need this? Today, I use this trick fix the RIT regions
again in our test env. The Procedure wal is Corrupted with no reason.
{quote}
2018-07-12 10:11:01,082 ERROR [master/test-sy-vpc-219-002:16000]
procedure2.ProcedureExecutor(327): Corrupt pid=362, ppid=333,
state=RUNNABLE:REGION_TRANSITION_DISPATCH; UnassignProcedure table=h, regi
on=b3f59c4fb17d0b5a4836479e931d5640,
server=test-sy-vpc-219-004.hbase.9b78df04-b.rds.aliyuncs.com,16020,1529657866452
{quote}
> Add the ability to compile JSP dynamically in Jetty
> ---
>
> Key: HBASE-20679
> URL: https://issues.apache.org/jira/browse/HBASE-20679
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 2.0.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20679.002.patch, HBASE-20679.003.patch,
> HBASE-20679.004.patch, HBASE-20679.005.patch, HBASE-20679.patch
>
>
> As discussed in HBASE-20617, adding the ability to dynamically compile jsp
> enable us to do some hot fix.
> For example, several days ago, in our testing HBase-2.0 cluster,
> procedureWals were corrupted due to some unknown reasons. After restarting
> the cluster, since some procedures(AssignProcedure for example) were
> corrupted and couldn't be replayed. Some regions were stuck in RIT forever.
> We couldn't use HBCK since it haven't support AssignmentV2 yet. As a matter
> of fact, the namespace region was not online, so the master was not inited,
> we even couldn't use shell command like assign/move. But, we wrote a jsp and
> fix this issue easily. The jsp file is like this:
> {code:java}
> <%
> String action = request.getParameter("action");
> HMaster master = (HMaster)getServletContext().getAttribute(HMaster.MASTER);
> List offlineRegionsToAssign = new ArrayList<>();
> List regionRITs =
> master.getAssignmentManager()
> .getRegionStates().getRegionsInTransition();
> for (RegionStates.RegionStateNode regionStateNode : regionRITs) {
> // if regionStateNode don't have a procedure attached, but meta state
> shows
> // this region is in RIT, that means the previous procedure may be
> corrupted
> // we need to create a new assignProcedure to assign them
> if (!regionStateNode.isInTransition()) {
> offlineRegionsToAssign.add(regionStateNode.getRegionInfo());
> out.println("RIT region:" + regionStateNode);
> }
> }
> // Assign offline regions. Uses round-robin.
> if ("fix".equals(action) && offlineRegionsToAssign.size() > 0) {
>
> master.getMasterProcedureExecutor().submitProcedures(master.getAssignmentManager().
> createRoundRobinAssignProcedures(offlineRegionsToAssign));
> } else {
> out.println("use ?action=fix to fix RIT regions");
> }
> %>
> {code}
> Above it is only one example we can do if we have the ability to compile jsp
> dynamically. We think it is very useful.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (HBASE-20697) Can't cache All region locations of the specify table by calling table.getRegionLocator().getAllRegionLocations()
[
https://issues.apache.org/jira/browse/HBASE-20697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang reopened HBASE-20697:
Reopen for fix checkstyle for branch-1.
> Can't cache All region locations of the specify table by calling
> table.getRegionLocator().getAllRegionLocations()
> -
>
> Key: HBASE-20697
> URL: https://issues.apache.org/jira/browse/HBASE-20697
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.1, 1.2.6, 2.0.1
>Reporter: zhaoyuan
>Assignee: zhaoyuan
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.4.6, 2.0.2, 2.2.0, 2.1.1
>
> Attachments: HBASE-20697.branch-1.2.001.patch,
> HBASE-20697.branch-1.2.002.patch, HBASE-20697.branch-1.2.003.patch,
> HBASE-20697.branch-1.2.004.patch, HBASE-20697.master.001.patch,
> HBASE-20697.master.002.patch, HBASE-20697.master.002.patch,
> HBASE-20697.master.003.patch
>
>
> When we upgrade and restart a new version application which will read and
> write to HBase, we will get some operation timeout. The time out is expected
> because when the application restarts，It will not hold any region locations
> cache and do communication with zk and meta regionserver to get region
> locations.
> We want to avoid these timeouts so we do warmup work and as far as I am
> concerned,the method table.getRegionLocator().getAllRegionLocations() will
> fetch all region locations and cache them. However, it didn't work good.
> There are still a lot of time outs,so it confused me.
> I dig into the source code and find something below
> {code:java}
> // code placeholder
> public List getAllRegionLocations() throws IOException {
> TableName tableName = getName();
> NavigableMap locations =
> MetaScanner.allTableRegions(this.connection, tableName);
> ArrayList regions = new ArrayList<>(locations.size());
> for (Entry entry : locations.entrySet()) {
> regions.add(new HRegionLocation(entry.getKey(), entry.getValue()));
> }
> if (regions.size() > 0) {
> connection.cacheLocation(tableName, new RegionLocations(regions));
> }
> return regions;
> }
> In MetaCache
> public void cacheLocation(final TableName tableName, final RegionLocations
> locations) {
> byte [] startKey =
> locations.getRegionLocation().getRegionInfo().getStartKey();
> ConcurrentMap tableLocations =
> getTableLocations(tableName);
> RegionLocations oldLocation = tableLocations.putIfAbsent(startKey,
> locations);
> boolean isNewCacheEntry = (oldLocation == null);
> if (isNewCacheEntry) {
> if (LOG.isTraceEnabled()) {
> LOG.trace("Cached location: " + locations);
> }
> addToCachedServers(locations);
> return;
> }
> {code}
> It will collect all regions into one RegionLocations object and only cache
> the first not null region location and then when we put or get to hbase, we
> do getCacheLocation()
> {code:java}
> // code placeholder
> public RegionLocations getCachedLocation(final TableName tableName, final
> byte [] row) {
> ConcurrentNavigableMap tableLocations =
> getTableLocations(tableName);
> Entry e = tableLocations.floorEntry(row);
> if (e == null) {
> if (metrics!= null) metrics.incrMetaCacheMiss();
> return null;
> }
> RegionLocations possibleRegion = e.getValue();
> // make sure that the end key is greater than the row we're looking
> // for, otherwise the row actually belongs in the next region, not
> // this one. the exception case is when the endkey is
> // HConstants.EMPTY_END_ROW, signifying that the region we're
> // checking is actually the last region in the table.
> byte[] endKey =
> possibleRegion.getRegionLocation().getRegionInfo().getEndKey();
> if (Bytes.equals(endKey, HConstants.EMPTY_END_ROW) ||
> getRowComparator(tableName).compareRows(
> endKey, 0, endKey.length, row, 0, row.length) > 0) {
> if (metrics != null) metrics.incrMetaCacheHit();
> return possibleRegion;
> }
> // Passed all the way through, so we got nothing - complete cache miss
> if (metrics != null) metrics.incrMetaCacheMiss();
> return null;
> }
> {code}
> It will choose the first location to be possibleRegion and possibly it will
> miss match.
> So did I forget something or may be wrong somewhere? If this is indeed a bug
> I think it can be fixed not very hard.
> Hope commiters and PMC review this !
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19572) RegionMover should use the configured default port number and not the one from HConstants
[
https://issues.apache.org/jira/browse/HBASE-19572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541081#comment-16541081
]
Reid Chan commented on HBASE-19572:
---
LGTM overall.
Two questions,
1. Any particular reason why changes the visibility to package visible
{code}
boolean ack = true;
{code}
2. {{final Configuration conf}} do we have to make it a member of
{{RegionMover}}. Remove it otherwise
> RegionMover should use the configured default port number and not the one
> from HConstants
> -
>
> Key: HBASE-19572
> URL: https://issues.apache.org/jira/browse/HBASE-19572
> Project: HBase
> Issue Type: Bug
>Reporter: Esteban Gutierrez
>Assignee: Toshihiro Suzuki
>Priority: Major
> Attachments: HBASE-19572.master.001.patch,
> HBASE-19572.master.001.patch, HBASE-19572.master.003.patch,
> HBASE-19572.master.004.patch, HBASE-19572.master.004.patch,
> HBASE-19572.patch, HBASE-19572.patch
>
>
> The issue I ran into HBASE-19499 was due RegionMover not using the port used
> by {{hbase-site.xml}}. The tool should use the value used in the
> configuration before falling back to the hardcoded value
> {{HConstants.DEFAULT_REGIONSERVER_PORT}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20697) Can't cache All region locations of the specify table by calling table.getRegionLocator().getAllRegionLocations()
[
https://issues.apache.org/jira/browse/HBASE-20697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541082#comment-16541082
]
Guanghao Zhang commented on HBASE-20697:
Pushed to branch-2.0, too.
> Can't cache All region locations of the specify table by calling
> table.getRegionLocator().getAllRegionLocations()
> -
>
> Key: HBASE-20697
> URL: https://issues.apache.org/jira/browse/HBASE-20697
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.1, 1.2.6, 2.0.1
>Reporter: zhaoyuan
>Assignee: zhaoyuan
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.4.6, 2.0.2, 2.2.0, 2.1.1
>
> Attachments: HBASE-20697.branch-1.2.001.patch,
> HBASE-20697.branch-1.2.002.patch, HBASE-20697.branch-1.2.003.patch,
> HBASE-20697.branch-1.2.004.patch, HBASE-20697.master.001.patch,
> HBASE-20697.master.002.patch, HBASE-20697.master.002.patch,
> HBASE-20697.master.003.patch
>
>
> When we upgrade and restart a new version application which will read and
> write to HBase, we will get some operation timeout. The time out is expected
> because when the application restarts，It will not hold any region locations
> cache and do communication with zk and meta regionserver to get region
> locations.
> We want to avoid these timeouts so we do warmup work and as far as I am
> concerned,the method table.getRegionLocator().getAllRegionLocations() will
> fetch all region locations and cache them. However, it didn't work good.
> There are still a lot of time outs,so it confused me.
> I dig into the source code and find something below
> {code:java}
> // code placeholder
> public List getAllRegionLocations() throws IOException {
> TableName tableName = getName();
> NavigableMap locations =
> MetaScanner.allTableRegions(this.connection, tableName);
> ArrayList regions = new ArrayList<>(locations.size());
> for (Entry entry : locations.entrySet()) {
> regions.add(new HRegionLocation(entry.getKey(), entry.getValue()));
> }
> if (regions.size() > 0) {
> connection.cacheLocation(tableName, new RegionLocations(regions));
> }
> return regions;
> }
> In MetaCache
> public void cacheLocation(final TableName tableName, final RegionLocations
> locations) {
> byte [] startKey =
> locations.getRegionLocation().getRegionInfo().getStartKey();
> ConcurrentMap tableLocations =
> getTableLocations(tableName);
> RegionLocations oldLocation = tableLocations.putIfAbsent(startKey,
> locations);
> boolean isNewCacheEntry = (oldLocation == null);
> if (isNewCacheEntry) {
> if (LOG.isTraceEnabled()) {
> LOG.trace("Cached location: " + locations);
> }
> addToCachedServers(locations);
> return;
> }
> {code}
> It will collect all regions into one RegionLocations object and only cache
> the first not null region location and then when we put or get to hbase, we
> do getCacheLocation()
> {code:java}
> // code placeholder
> public RegionLocations getCachedLocation(final TableName tableName, final
> byte [] row) {
> ConcurrentNavigableMap tableLocations =
> getTableLocations(tableName);
> Entry e = tableLocations.floorEntry(row);
> if (e == null) {
> if (metrics!= null) metrics.incrMetaCacheMiss();
> return null;
> }
> RegionLocations possibleRegion = e.getValue();
> // make sure that the end key is greater than the row we're looking
> // for, otherwise the row actually belongs in the next region, not
> // this one. the exception case is when the endkey is
> // HConstants.EMPTY_END_ROW, signifying that the region we're
> // checking is actually the last region in the table.
> byte[] endKey =
> possibleRegion.getRegionLocation().getRegionInfo().getEndKey();
> if (Bytes.equals(endKey, HConstants.EMPTY_END_ROW) ||
> getRowComparator(tableName).compareRows(
> endKey, 0, endKey.length, row, 0, row.length) > 0) {
> if (metrics != null) metrics.incrMetaCacheHit();
> return possibleRegion;
> }
> // Passed all the way through, so we got nothing - complete cache miss
> if (metrics != null) metrics.incrMetaCacheMiss();
> return null;
> }
> {code}
> It will choose the first location to be possibleRegion and possibly it will
> miss match.
> So did I forget something or may be wrong somewhere? If this is indeed a bug
> I think it can be fixed not very hard.
> Hope commiters and PMC review this !
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20875) MemStoreLABImp::copyIntoCell uses 7% CPU when writing
[
https://issues.apache.org/jira/browse/HBASE-20875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541074#comment-16541074
]
Hadoop QA commented on HBASE-20875:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 3s{color}
| {color:red} HBASE-20875 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.7.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20875 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12931264/0001-HBASE-20875-MemStoreLABImp-copyIntoCell-uses-7-CPU-w.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/13602/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> MemStoreLABImp::copyIntoCell uses 7% CPU when writing
> -
>
> Key: HBASE-20875
> URL: https://issues.apache.org/jira/browse/HBASE-20875
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
>Affects Versions: 2.0.1
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.0.2
>
> Attachments:
> 0001-HBASE-20875-MemStoreLABImp-copyIntoCell-uses-7-CPU-w.patch,
> 2.0707.baseline.91935.cpu.svg, 2.0711.patched.145414.cpu.svg, Screen Shot
> 2018-07-11 at 9.52.46 PM.png
>
>
> Looks like this with a PE random write loading:
> {code}
> ./hbase/bin/hbase --config ~/conf_hbase
> org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --presplit=40
> --size=30 --columns=10 --valueSize=100 randomWrite 200
> {code}
> ... against a single server.
> {code}
> 12.47% perf-91935.map
> [.] Lorg/apache/hadoop/hbase/BBKVComparator;::compare
> 10.42% libjvm.so
> [.]
> ParNewGeneration::copy_to_survivor_space_avoiding_promotion_undo(ParScanThreadState*,
> oopDesc*, unsigned long, markOopDesc*)
> 6.78% perf-91935.map
> [.]
> Lorg/apache/hadoop/hbase/regionserver/MemStoreLABImpl;::copyCellInto
>
> {code}
> These are top CPU consumers using perf-map-agent ./bin/perf-java-top...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20875) MemStoreLABImp::copyIntoCell uses 7% CPU when writing
[
https://issues.apache.org/jira/browse/HBASE-20875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541072#comment-16541072
]
stack commented on HBASE-20875:
---
Makes for a small improvement in throughput. Nothing dramatic so no graph.
#compare still the long pole.
> MemStoreLABImp::copyIntoCell uses 7% CPU when writing
> -
>
> Key: HBASE-20875
> URL: https://issues.apache.org/jira/browse/HBASE-20875
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
>Affects Versions: 2.0.1
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.0.2
>
> Attachments:
> 0001-HBASE-20875-MemStoreLABImp-copyIntoCell-uses-7-CPU-w.patch,
> 2.0707.baseline.91935.cpu.svg, 2.0711.patched.145414.cpu.svg, Screen Shot
> 2018-07-11 at 9.52.46 PM.png
>
>
> Looks like this with a PE random write loading:
> {code}
> ./hbase/bin/hbase --config ~/conf_hbase
> org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --presplit=40
> --size=30 --columns=10 --valueSize=100 randomWrite 200
> {code}
> ... against a single server.
> {code}
> 12.47% perf-91935.map
> [.] Lorg/apache/hadoop/hbase/BBKVComparator;::compare
> 10.42% libjvm.so
> [.]
> ParNewGeneration::copy_to_survivor_space_avoiding_promotion_undo(ParScanThreadState*,
> oopDesc*, unsigned long, markOopDesc*)
> 6.78% perf-91935.map
> [.]
> Lorg/apache/hadoop/hbase/regionserver/MemStoreLABImpl;::copyCellInto
>
> {code}
> These are top CPU consumers using perf-map-agent ./bin/perf-java-top...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20875) MemStoreLABImp::copyIntoCell uses 7% CPU when writing
[
https://issues.apache.org/jira/browse/HBASE-20875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-20875:
--
Assignee: stack
Fix Version/s: 2.0.2
Affects Version/s: 2.0.1
Status: Patch Available (was: Open)
Small patch. Adds a smaller copyCellInto called copyBBECellInto picked up by
the hotpath.
> MemStoreLABImp::copyIntoCell uses 7% CPU when writing
> -
>
> Key: HBASE-20875
> URL: https://issues.apache.org/jira/browse/HBASE-20875
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
>Affects Versions: 2.0.1
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.0.2
>
> Attachments:
> 0001-HBASE-20875-MemStoreLABImp-copyIntoCell-uses-7-CPU-w.patch,
> 2.0707.baseline.91935.cpu.svg, 2.0711.patched.145414.cpu.svg, Screen Shot
> 2018-07-11 at 9.52.46 PM.png
>
>
> Looks like this with a PE random write loading:
> {code}
> ./hbase/bin/hbase --config ~/conf_hbase
> org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --presplit=40
> --size=30 --columns=10 --valueSize=100 randomWrite 200
> {code}
> ... against a single server.
> {code}
> 12.47% perf-91935.map
> [.] Lorg/apache/hadoop/hbase/BBKVComparator;::compare
> 10.42% libjvm.so
> [.]
> ParNewGeneration::copy_to_survivor_space_avoiding_promotion_undo(ParScanThreadState*,
> oopDesc*, unsigned long, markOopDesc*)
> 6.78% perf-91935.map
> [.]
> Lorg/apache/hadoop/hbase/regionserver/MemStoreLABImpl;::copyCellInto
>
> {code}
> These are top CPU consumers using perf-map-agent ./bin/perf-java-top...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20875) MemStoreLABImp::copyIntoCell uses 7% CPU when writing
[
https://issues.apache.org/jira/browse/HBASE-20875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-20875:
--
Component/s: Performance
> MemStoreLABImp::copyIntoCell uses 7% CPU when writing
> -
>
> Key: HBASE-20875
> URL: https://issues.apache.org/jira/browse/HBASE-20875
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
>Affects Versions: 2.0.1
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.0.2
>
> Attachments:
> 0001-HBASE-20875-MemStoreLABImp-copyIntoCell-uses-7-CPU-w.patch,
> 2.0707.baseline.91935.cpu.svg, 2.0711.patched.145414.cpu.svg, Screen Shot
> 2018-07-11 at 9.52.46 PM.png
>
>
> Looks like this with a PE random write loading:
> {code}
> ./hbase/bin/hbase --config ~/conf_hbase
> org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --presplit=40
> --size=30 --columns=10 --valueSize=100 randomWrite 200
> {code}
> ... against a single server.
> {code}
> 12.47% perf-91935.map
> [.] Lorg/apache/hadoop/hbase/BBKVComparator;::compare
> 10.42% libjvm.so
> [.]
> ParNewGeneration::copy_to_survivor_space_avoiding_promotion_undo(ParScanThreadState*,
> oopDesc*, unsigned long, markOopDesc*)
> 6.78% perf-91935.map
> [.]
> Lorg/apache/hadoop/hbase/regionserver/MemStoreLABImpl;::copyCellInto
>
> {code}
> These are top CPU consumers using perf-map-agent ./bin/perf-java-top...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20875) MemStoreLABImp::copyIntoCell uses 7% CPU when writing
[
https://issues.apache.org/jira/browse/HBASE-20875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-20875:
--
Attachment: 0001-HBASE-20875-MemStoreLABImp-copyIntoCell-uses-7-CPU-w.patch
> MemStoreLABImp::copyIntoCell uses 7% CPU when writing
> -
>
> Key: HBASE-20875
> URL: https://issues.apache.org/jira/browse/HBASE-20875
> Project: HBase
> Issue Type: Sub-task
>Reporter: stack
>Priority: Major
> Attachments:
> 0001-HBASE-20875-MemStoreLABImp-copyIntoCell-uses-7-CPU-w.patch,
> 2.0707.baseline.91935.cpu.svg, 2.0711.patched.145414.cpu.svg, Screen Shot
> 2018-07-11 at 9.52.46 PM.png
>
>
> Looks like this with a PE random write loading:
> {code}
> ./hbase/bin/hbase --config ~/conf_hbase
> org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --presplit=40
> --size=30 --columns=10 --valueSize=100 randomWrite 200
> {code}
> ... against a single server.
> {code}
> 12.47% perf-91935.map
> [.] Lorg/apache/hadoop/hbase/BBKVComparator;::compare
> 10.42% libjvm.so
> [.]
> ParNewGeneration::copy_to_survivor_space_avoiding_promotion_undo(ParScanThreadState*,
> oopDesc*, unsigned long, markOopDesc*)
> 6.78% perf-91935.map
> [.]
> Lorg/apache/hadoop/hbase/regionserver/MemStoreLABImpl;::copyCellInto
>
> {code}
> These are top CPU consumers using perf-map-agent ./bin/perf-java-top...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20875) MemStoreLABImp::copyIntoCell uses 7% CPU when writing
[
https://issues.apache.org/jira/browse/HBASE-20875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541070#comment-16541070
]
stack commented on HBASE-20875:
---
!Screen Shot 2018-07-11 at 9.52.46 PM.png! is from jitwatch. See how
copyCellInto is just over inlin'ing threashold of 3k. This is before patch.
Look at [^2.0707.baseline.91935.cpu.svg] to the right of the biggest hump
where we are doing maybeCloneWithAllocator. See how it is > 10%. In patched
view, it is < 5%... copyBBECellInto is the diff.
> MemStoreLABImp::copyIntoCell uses 7% CPU when writing
> -
>
> Key: HBASE-20875
> URL: https://issues.apache.org/jira/browse/HBASE-20875
> Project: HBase
> Issue Type: Sub-task
>Reporter: stack
>Priority: Major
> Attachments: 2.0707.baseline.91935.cpu.svg,
> 2.0711.patched.145414.cpu.svg, Screen Shot 2018-07-11 at 9.52.46 PM.png
>
>
> Looks like this with a PE random write loading:
> {code}
> ./hbase/bin/hbase --config ~/conf_hbase
> org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --presplit=40
> --size=30 --columns=10 --valueSize=100 randomWrite 200
> {code}
> ... against a single server.
> {code}
> 12.47% perf-91935.map
> [.] Lorg/apache/hadoop/hbase/BBKVComparator;::compare
> 10.42% libjvm.so
> [.]
> ParNewGeneration::copy_to_survivor_space_avoiding_promotion_undo(ParScanThreadState*,
> oopDesc*, unsigned long, markOopDesc*)
> 6.78% perf-91935.map
> [.]
> Lorg/apache/hadoop/hbase/regionserver/MemStoreLABImpl;::copyCellInto
>
> {code}
> These are top CPU consumers using perf-map-agent ./bin/perf-java-top...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20866) HBase 1.x scan performance degradation compared to 0.98 version
[
https://issues.apache.org/jira/browse/HBASE-20866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541069#comment-16541069
]
Reid Chan commented on HBASE-20866:
---
The core is to avoid conversion (resultsFromServer --> resultsToAddToCache)
LGTM.
Same concern as Mike, and i skimmed through branch-1 and master branch. The
conversion becomes [] to sub[], then add to cache. and there're much more work.
> HBase 1.x scan performance degradation compared to 0.98 version
> ---
>
> Key: HBASE-20866
> URL: https://issues.apache.org/jira/browse/HBASE-20866
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.2
>Reporter: Vikas Vishwakarma
>Assignee: Vikas Vishwakarma
>Priority: Critical
> Fix For: 1.5.0, 1.2.7, 1.3.3, 1.4.6
>
> Attachments: HBASE-20866.branch-1.3.001.patch,
> HBASE-20866.branch-1.3.002.patch
>
>
> Internally while testing 1.3 as part of migration from 0.98 to 1.3 we
> observed perf degradation in scan performance for phoenix queries varying
> from few 10's to upto 200% depending on the query being executed. We tried
> simple native HBase scan and there also we saw upto 40% degradation in
> performance when the number of column qualifiers are high (40-50+)
> To identify the root cause of performance diff between 0.98 and 1.3 we
> carried out lot of experiments with profiling and git bisect iterations,
> however we were not able to identify any particular source of scan
> performance degradation and it looked like this is an accumulated degradation
> of 5-10% over various enhancements and refactoring.
> We identified few major enhancements like partialResult handling,
> ScannerContext with heartbeat processing, time/size limiting, RPC
> refactoring, etc that could have contributed to small degradation in
> performance which put together could be leading to large overall degradation.
> One of the changes is
> [HBASE-11544|https://jira.apache.org/jira/browse/HBASE-11544] which
> implements partialResult handling. In ClientScanner.java the results received
> from server are cached on the client side by converting the result array into
> an ArrayList. This function gets called in a loop depending on the number of
> rows in the scan result. Example for ten’s of millions of rows scanned, this
> can be called in the order of millions of times.
> In almost all the cases 99% of the time (except for handling partial results,
> etc). We are just taking the resultsFromServer converting it into a ArrayList
> resultsToAddToCache in addResultsToList(..) and then iterating over the list
> again and adding it to cache in loadCache(..) as given in the code path below
> In ClientScanner → loadCache(..) → getResultsToAddToCache(..) →
> addResultsToList(..) →
> {code:java}
> loadCache() {
> ...
> List resultsToAddToCache =
> getResultsToAddToCache(values, callable.isHeartbeatMessage());
> ...
> …
> for (Result rs : resultsToAddToCache) {
> rs = filterLoadedCell(rs);
> cache.add(rs);
> ...
> }
> }
> getResultsToAddToCache(..) {
> ..
> final boolean isBatchSet = scan != null && scan.getBatch() > 0;
> final boolean allowPartials = scan != null &&
> scan.getAllowPartialResults();
> ..
> if (allowPartials || isBatchSet) {
> addResultsToList(resultsToAddToCache, resultsFromServer, 0,
> (null == resultsFromServer ? 0 : resultsFromServer.length));
> return resultsToAddToCache;
> }
> ...
> }
> private void addResultsToList(List outputList, Result[] inputArray,
> int start, int end) {
> if (inputArray == null || start < 0 || end > inputArray.length) return;
> for (int i = start; i < end; i++) {
> outputList.add(inputArray[i]);
> }
> }{code}
>
> It looks like we can avoid the result array to arraylist conversion
> (resultsFromServer --> resultsToAddToCache ) for the first case which is also
> the most frequent case and instead directly take the values arraay returned
> by callable and add it to the cache without converting it into ArrayList.
> I have taken both these flags allowPartials and isBatchSet out in loadcahe()
> and I am directly adding values to scanner cache if the above condition is
> pass instead of coverting it into arrayList by calling
> getResultsToAddToCache(). For example:
> {code:java}
> protected void loadCache() throws IOException {
> Result[] values = null;
> ..
> final boolean isBatchSet = scan != null && scan.getBatch() > 0;
> final boolean allowPartials = scan != null && scan.getAllowPartialResults();
> ..
> for (;;) {
> try {
> values = call(callable, caller, scannerTimeout);
> ..
> } catch (DoNotRetryIOException | NeedUnmanagedConnectionException e) {
> ..
> }
> if (allowPartials || isBatchSet) { // DIRECTLY COPY values TO
[jira] [Updated] (HBASE-20875) MemStoreLABImp::copyIntoCell uses 7% CPU when writing
[
https://issues.apache.org/jira/browse/HBASE-20875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-20875:
--
Attachment: 2.0711.patched.145414.cpu.svg
> MemStoreLABImp::copyIntoCell uses 7% CPU when writing
> -
>
> Key: HBASE-20875
> URL: https://issues.apache.org/jira/browse/HBASE-20875
> Project: HBase
> Issue Type: Sub-task
>Reporter: stack
>Priority: Major
> Attachments: 2.0707.baseline.91935.cpu.svg,
> 2.0711.patched.145414.cpu.svg, Screen Shot 2018-07-11 at 9.52.46 PM.png
>
>
> Looks like this with a PE random write loading:
> {code}
> ./hbase/bin/hbase --config ~/conf_hbase
> org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --presplit=40
> --size=30 --columns=10 --valueSize=100 randomWrite 200
> {code}
> ... against a single server.
> {code}
> 12.47% perf-91935.map
> [.] Lorg/apache/hadoop/hbase/BBKVComparator;::compare
> 10.42% libjvm.so
> [.]
> ParNewGeneration::copy_to_survivor_space_avoiding_promotion_undo(ParScanThreadState*,
> oopDesc*, unsigned long, markOopDesc*)
> 6.78% perf-91935.map
> [.]
> Lorg/apache/hadoop/hbase/regionserver/MemStoreLABImpl;::copyCellInto
>
> {code}
> These are top CPU consumers using perf-map-agent ./bin/perf-java-top...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20875) MemStoreLABImp::copyIntoCell uses 7% CPU when writing
[
https://issues.apache.org/jira/browse/HBASE-20875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-20875:
--
Attachment: 2.0707.baseline.91935.cpu.svg
> MemStoreLABImp::copyIntoCell uses 7% CPU when writing
> -
>
> Key: HBASE-20875
> URL: https://issues.apache.org/jira/browse/HBASE-20875
> Project: HBase
> Issue Type: Sub-task
>Reporter: stack
>Priority: Major
> Attachments: 2.0707.baseline.91935.cpu.svg,
> 2.0711.patched.145414.cpu.svg, Screen Shot 2018-07-11 at 9.52.46 PM.png
>
>
> Looks like this with a PE random write loading:
> {code}
> ./hbase/bin/hbase --config ~/conf_hbase
> org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --presplit=40
> --size=30 --columns=10 --valueSize=100 randomWrite 200
> {code}
> ... against a single server.
> {code}
> 12.47% perf-91935.map
> [.] Lorg/apache/hadoop/hbase/BBKVComparator;::compare
> 10.42% libjvm.so
> [.]
> ParNewGeneration::copy_to_survivor_space_avoiding_promotion_undo(ParScanThreadState*,
> oopDesc*, unsigned long, markOopDesc*)
> 6.78% perf-91935.map
> [.]
> Lorg/apache/hadoop/hbase/regionserver/MemStoreLABImpl;::copyCellInto
>
> {code}
> These are top CPU consumers using perf-map-agent ./bin/perf-java-top...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20865) CreateTableProcedure is stuck in retry loop in CREATE_TABLE_WRITE_FS_LAYOUT state
[ https://issues.apache.org/jira/browse/HBASE-20865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541067#comment-16541067 ] Ted Yu commented on HBASE-20865: lgtm [~Apache9]: Can you take a look as well ? > CreateTableProcedure is stuck in retry loop in CREATE_TABLE_WRITE_FS_LAYOUT > state > - > > Key: HBASE-20865 > URL: https://issues.apache.org/jira/browse/HBASE-20865 > Project: HBase > Issue Type: Bug > Components: amv2 >Reporter: Toshihiro Suzuki >Assignee: Toshihiro Suzuki >Priority: Major > Attachments: HBASE-20865.master.001.patch > > > Similar to HBASE-20616, CreateTableProcedure gets stuck in retry loop in > CREATE_TABLE_WRITE_FS_LAYOUT state when writing HDFS fails. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20875) MemStoreLABImp::copyIntoCell uses 7% CPU when writing
[
https://issues.apache.org/jira/browse/HBASE-20875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541066#comment-16541066
]
stack commented on HBASE-20875:
---
Let me upload a patch. Has the method now using 1.6% of CPU. Here is the top
methods:
{code}
16.31% libjvm.so
[.]
ParNewGeneration::copy_to_survivor_space_avoiding_promotion_undo(ParScanThreadState*,
oopDesc*, unsigned long, markOopDesc*)
11.97% perf-145414.map
[.] Lorg/apache/hadoop/hbase/BBKVComparator;::compare
6.16% perf-145414.map
[.] Lorg/apache/hadoop/hbase/BBKVComparator;::compare
4.41% perf-145414.map
[.] Ljava/util/concurrent/ConcurrentSkipListMap;::doPut
3.93% perf-145414.map
[.]
Lorg/apache/hadoop/hbase/regionserver/AbstractMemStore;::add
2.86% perf-145414.map
[.] Lorg/apache/hadoop/hbase/client/Put;::add
2.39% perf-145414.map
[.]
Lorg/apache/hadoop/hbase/regionserver/HRegion;::updateCellTimestamps
1.96% perf-145414.map
[.]
Lorg/apache/hadoop/hbase/regionserver/RSRpcServices;::checkCellSizeLimit
1.71% perf-145414.map
[.]
Lorg/apache/hadoop/hbase/regionserver/wal/AsyncProtobufLogWriter;::append
1.64% perf-145414.map
[.]
Lorg/apache/hadoop/hbase/regionserver/wal/AbstractFSWAL;::stampSequenceIdAndPublishToRingBuffer
1.61% perf-145414.map
[.]
Lorg/apache/hadoop/hbase/regionserver/MemStoreLABImpl;::copyBBECellInto
{code}
The new, purposed method name is copyBBECellInto, the equiv of the old
copyCellInto.
> MemStoreLABImp::copyIntoCell uses 7% CPU when writing
> -
>
> Key: HBASE-20875
> URL: https://issues.apache.org/jira/browse/HBASE-20875
> Project: HBase
> Issue Type: Sub-task
>Reporter: stack
>Priority: Major
> Attachments: Screen Shot 2018-07-11 at 9.52.46 PM.png
>
>
> Looks like this with a PE random write loading:
> {code}
> ./hbase/bin/hbase --config ~/conf_hbase
> org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --presplit=40
> --size=30 --columns=10 --valueSize=100 randomWrite 200
> {code}
> ... against a single server.
> {code}
> 12.47% perf-91935.map
> [.] Lorg/apache/hadoop/hbase/BBKVComparator;::compare
> 10.42% libjvm.so
> [.]
> ParNewGeneration::copy_to_survivor_space_avoiding_promotion_undo(ParScanThreadState*,
> oopDesc*, unsigned long, markOopDesc*)
> 6.78% perf-91935.map
> [.]
> Lorg/apache/hadoop/hbase/regionserver/MemStoreLABImpl;::copyCellInto
>
> {code}
> These are top CPU consumers using perf-map-agent ./bin/perf-java-top...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-18840) Add functionality to refresh meta table at master startup
[
https://issues.apache.org/jira/browse/HBASE-18840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541064#comment-16541064
]
Hadoop QA commented on HBASE-18840:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HBASE-18477 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
18s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
21s{color} | {color:green} HBASE-18477 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
13s{color} | {color:green} HBASE-18477 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
1s{color} | {color:green} HBASE-18477 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
30s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
47s{color} | {color:green} HBASE-18477 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} HBASE-18477 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} hbase-client: The patch generated 0 new + 124
unchanged - 1 fixed = 124 total (was 125) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
22s{color} | {color:red} hbase-server: The patch generated 2 new + 242
unchanged - 0 fixed = 244 total (was 242) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
31s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
13m 29s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
44s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}206m 40s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}273m 39s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestMetaWithReplicas |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-18840 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12931227/HBASE-18840.HBASE-18477.007.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux a5afd9e4e399 4.4.0-130-g
[jira] [Updated] (HBASE-20875) MemStoreLABImp::copyIntoCell uses 7% CPU when writing
[
https://issues.apache.org/jira/browse/HBASE-20875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-20875:
--
Attachment: Screen Shot 2018-07-11 at 9.52.46 PM.png
> MemStoreLABImp::copyIntoCell uses 7% CPU when writing
> -
>
> Key: HBASE-20875
> URL: https://issues.apache.org/jira/browse/HBASE-20875
> Project: HBase
> Issue Type: Sub-task
>Reporter: stack
>Priority: Major
> Attachments: Screen Shot 2018-07-11 at 9.52.46 PM.png
>
>
> Looks like this with a PE random write loading:
> {code}
> ./hbase/bin/hbase --config ~/conf_hbase
> org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --presplit=40
> --size=30 --columns=10 --valueSize=100 randomWrite 200
> {code}
> ... against a single server.
> {code}
> 12.47% perf-91935.map
> [.] Lorg/apache/hadoop/hbase/BBKVComparator;::compare
> 10.42% libjvm.so
> [.]
> ParNewGeneration::copy_to_survivor_space_avoiding_promotion_undo(ParScanThreadState*,
> oopDesc*, unsigned long, markOopDesc*)
> 6.78% perf-91935.map
> [.]
> Lorg/apache/hadoop/hbase/regionserver/MemStoreLABImpl;::copyCellInto
>
> {code}
> These are top CPU consumers using perf-map-agent ./bin/perf-java-top...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20865) CreateTableProcedure is stuck in retry loop in CREATE_TABLE_WRITE_FS_LAYOUT state
[ https://issues.apache.org/jira/browse/HBASE-20865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541054#comment-16541054 ] Toshihiro Suzuki commented on HBASE-20865: -- Could you please review the patch? [~yuzhih...@gmail.com] Thanks. > CreateTableProcedure is stuck in retry loop in CREATE_TABLE_WRITE_FS_LAYOUT > state > - > > Key: HBASE-20865 > URL: https://issues.apache.org/jira/browse/HBASE-20865 > Project: HBase > Issue Type: Bug > Components: amv2 >Reporter: Toshihiro Suzuki >Assignee: Toshihiro Suzuki >Priority: Major > Attachments: HBASE-20865.master.001.patch > > > Similar to HBASE-20616, CreateTableProcedure gets stuck in retry loop in > CREATE_TABLE_WRITE_FS_LAYOUT state when writing HDFS fails. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20875) MemStoreLABImp::copyIntoCell uses 7% CPU when writing
stack created HBASE-20875:
-
Summary: MemStoreLABImp::copyIntoCell uses 7% CPU when writing
Key: HBASE-20875
URL: https://issues.apache.org/jira/browse/HBASE-20875
Project: HBase
Issue Type: Sub-task
Reporter: stack
Looks like this with a PE random write loading:
{code}
./hbase/bin/hbase --config ~/conf_hbase
org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --presplit=40
--size=30 --columns=10 --valueSize=100 randomWrite 200
{code}
... against a single server.
{code}
12.47% perf-91935.map
[.] Lorg/apache/hadoop/hbase/BBKVComparator;::compare
10.42% libjvm.so
[.]
ParNewGeneration::copy_to_survivor_space_avoiding_promotion_undo(ParScanThreadState*,
oopDesc*, unsigned long, markOopDesc*)
6.78% perf-91935.map
[.]
Lorg/apache/hadoop/hbase/regionserver/MemStoreLABImpl;::copyCellInto
{code}
These are top CPU consumers using perf-map-agent ./bin/perf-java-top...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20846) Table's shared lock is not held by sub-procedures after master restart
[
https://issues.apache.org/jira/browse/HBASE-20846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541048#comment-16541048
]
Duo Zhang commented on HBASE-20846:
---
Anyway let me provide a patch first. The optimization can be done later.
> Table's shared lock is not held by sub-procedures after master restart
> --
>
> Key: HBASE-20846
> URL: https://issues.apache.org/jira/browse/HBASE-20846
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.1.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Major
> Fix For: 3.0.0, 2.0.2, 2.1.1
>
> Attachments: HBASE-20846.branch-2.0.002.patch,
> HBASE-20846.branch-2.0.patch
>
>
> Found this one when investigating ModifyTableProcedure got stuck while there
> was a MoveRegionProcedure going on after master restart.
> Though this issue can be solved by HBASE-20752. But I discovered something
> else.
> Before a MoveRegionProcedure can execute, it will hold the table's shared
> lock. so,, when a UnassignProcedure was spwaned, it will not check the
> table's shared lock since it is sure that its parent(MoveRegionProcedure) has
> aquired the table's lock.
> {code:java}
> // If there is parent procedure, it would have already taken xlock, so no
> need to take
> // shared lock here. Otherwise, take shared lock.
> if (!procedure.hasParent()
> && waitTableQueueSharedLock(procedure, table) == null) {
> return true;
> }
> {code}
> But, it is not the case when Master was restarted. The child
> procedure(UnassignProcedure) will be executed first after restart. Though it
> has a parent(MoveRegionProcedure), but apprently the parent didn't hold the
> table's lock.
> So, since it began to execute without hold the table's shared lock. A
> ModifyTableProcedure can aquire the table's exclusive lock and execute at the
> same time. Which is not possible if the master was not restarted.
> This will cause a stuck before HBASE-20752. But since HBASE-20752 has fixed,
> I wrote a simple UT to repo this case.
> I think we don't have to check the parent for table's shared lock. It is a
> shared lock, right? I think we can acquire it every time we need it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20838) Include hbase-server in precommit test if CommonFSUtils is changed
[
https://issues.apache.org/jira/browse/HBASE-20838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541046#comment-16541046
]
Duo Zhang commented on HBASE-20838:
---
+1 on committing to branch-2.1.
> Include hbase-server in precommit test if CommonFSUtils is changed
> --
>
> Key: HBASE-20838
> URL: https://issues.apache.org/jira/browse/HBASE-20838
> Project: HBase
> Issue Type: Test
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-20838.patch, HBASE-20838.patch,
> HBASE-20838.v2.patch
>
>
> -As per
> [discussed|https://issues.apache.org/jira/browse/HBASE-20691?focusedCommentId=16517662&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16517662]
> in HBASE-20691, since the setStoragePolicy code is in CommonFSUtils, the
> test should be in TestCommonFSUtils-
> We don't want to introduce dependency on hadoop-hdfs to hbase-common so
> decided to leave the setStoragePolicy related tests in TestCommonFSUtils.
> Instead, we will change the personality script to include hbase-server in
> unit test if any change made against {{CommonFSUtils}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20838) Include hbase-server in precommit test if CommonFSUtils is changed
[
https://issues.apache.org/jira/browse/HBASE-20838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-20838:
--
Resolution: Fixed
Fix Version/s: 2.2.0
3.0.0
Status: Resolved (was: Patch Available)
Pushed into master and branch-2. [~stack] [~Apache9] please let me know if
you'd like this one in branch-2.0/2.1, thanks.
Closing issue for now.
> Include hbase-server in precommit test if CommonFSUtils is changed
> --
>
> Key: HBASE-20838
> URL: https://issues.apache.org/jira/browse/HBASE-20838
> Project: HBase
> Issue Type: Test
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-20838.patch, HBASE-20838.patch,
> HBASE-20838.v2.patch
>
>
> -As per
> [discussed|https://issues.apache.org/jira/browse/HBASE-20691?focusedCommentId=16517662&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16517662]
> in HBASE-20691, since the setStoragePolicy code is in CommonFSUtils, the
> test should be in TestCommonFSUtils-
> We don't want to introduce dependency on hadoop-hdfs to hbase-common so
> decided to leave the setStoragePolicy related tests in TestCommonFSUtils.
> Instead, we will change the personality script to include hbase-server in
> unit test if any change made against {{CommonFSUtils}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20838) Include hbase-server in precommit test if CommonFSUtils is changed
[
https://issues.apache.org/jira/browse/HBASE-20838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated HBASE-20838:
--
Hadoop Flags: Reviewed
Description:
-As per
[discussed|https://issues.apache.org/jira/browse/HBASE-20691?focusedCommentId=16517662&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16517662]
in HBASE-20691, since the setStoragePolicy code is in CommonFSUtils, the test
should be in TestCommonFSUtils-
We don't want to introduce dependency on hadoop-hdfs to hbase-common so decided
to leave the setStoragePolicy related tests in TestCommonFSUtils. Instead, we
will change the personality script to include hbase-server in unit test if any
change made against {{CommonFSUtils}}
was:As per
[discussed|https://issues.apache.org/jira/browse/HBASE-20691?focusedCommentId=16517662&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16517662]
in HBASE-20691, since the setStoragePolicy code is in CommonFSUtils, the test
should be in TestCommonFSUtils
Updated title and description.
Thanks [~busbey] for review, will commit this soon.
> Include hbase-server in precommit test if CommonFSUtils is changed
> --
>
> Key: HBASE-20838
> URL: https://issues.apache.org/jira/browse/HBASE-20838
> Project: HBase
> Issue Type: Test
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Major
> Attachments: HBASE-20838.patch, HBASE-20838.patch,
> HBASE-20838.v2.patch
>
>
> -As per
> [discussed|https://issues.apache.org/jira/browse/HBASE-20691?focusedCommentId=16517662&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16517662]
> in HBASE-20691, since the setStoragePolicy code is in CommonFSUtils, the
> test should be in TestCommonFSUtils-
> We don't want to introduce dependency on hadoop-hdfs to hbase-common so
> decided to leave the setStoragePolicy related tests in TestCommonFSUtils.
> Instead, we will change the personality script to include hbase-server in
> unit test if any change made against {{CommonFSUtils}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20838) Include hbase-server in precommit test if CommonFSUtils is changed
[ https://issues.apache.org/jira/browse/HBASE-20838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated HBASE-20838: -- Summary: Include hbase-server in precommit test if CommonFSUtils is changed (was: Move all setStorage related UT cases from TestFSUtils to TestCommonFSUtils) > Include hbase-server in precommit test if CommonFSUtils is changed > -- > > Key: HBASE-20838 > URL: https://issues.apache.org/jira/browse/HBASE-20838 > Project: HBase > Issue Type: Test >Reporter: Yu Li >Assignee: Yu Li >Priority: Major > Attachments: HBASE-20838.patch, HBASE-20838.patch, > HBASE-20838.v2.patch > > > As per > [discussed|https://issues.apache.org/jira/browse/HBASE-20691?focusedCommentId=16517662&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16517662] > in HBASE-20691, since the setStoragePolicy code is in CommonFSUtils, the > test should be in TestCommonFSUtils -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20870) Wrong HBase root dir in ITBLL's Search Tool
[
https://issues.apache.org/jira/browse/HBASE-20870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541020#comment-16541020
]
Mike Drob commented on HBASE-20870:
---
I'm a little confused on what we're setting and unsetting. IIRC I had tested
these on distributed clusters already.
{code}
-conf.set("fs.defaultFS", conf.get("original.defaultFS"));
+Configuration localConf = HBaseConfiguration.create();
+conf.set("fs.defaultFS", localConf.get("fs.defaultFS"));
+conf.set(HConstants.HBASE_DIR, localConf.get(HConstants.HBASE_DIR));
{code}
Is the defaultFS in localConf different from the old value that we saved? I
wouldn't expect it to be.
Can we give HBASE_DIR the same treatment? I think I was saving it in
HBaseTestingUtility in some version of my patch but then it appeared that
wasn't necessary. See HBASE-19947
> Wrong HBase root dir in ITBLL's Search Tool
> ---
>
> Key: HBASE-20870
> URL: https://issues.apache.org/jira/browse/HBASE-20870
> Project: HBase
> Issue Type: Bug
> Components: integration tests
>Affects Versions: 3.0.0, 2.1.0, 2.0.1
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Minor
> Attachments: HBASE-20870.branch-2.0.001.patch
>
>
> When using IntegrationTestBigLinkedList's Search tools, it always fails since
> it tries to read WALs in a wrong HBase root dir. Turned out that when
> initializing IntegrationTestingUtility in IntegrationTestBigLinkedList, its
> super class HBaseTestingUtility will change hbase.rootdir to a local random
> dir. It is not wrong since HBaseTestingUtility is mostly used by Minicluster.
> But for IntegrationTest runs on distributed clusters, we should change it
> back.
> Here is the error info.
> {code:java}
> 2018-07-11 16:35:49,679 DEBUG [main] hbase.HBaseCommonTestingUtility: Setting
> hbase.rootdir to
> /home/hadoop/target/test-data/deb67611-2737-4696-abe9-32a7783df7bb
> 2018-07-11 16:35:50,736 ERROR [main] util.AbstractHBaseTool: Error running
> command-line tool java.io.FileNotFoundException: File
> file:/home/hadoop/target/test-data/deb67611-2737-4696-abe9-32a7783df7bb/WALs
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:431)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1517)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20859) Backup and incremental load could fail in secure clusters
[
https://issues.apache.org/jira/browse/HBASE-20859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541018#comment-16541018
]
Reid Chan commented on HBASE-20859:
---
{code}
/** Don't use it! This'll get you the wrong path in a secure cluster.
* Use FileSystem.getHomeDirectory() or
* "/user/" + UserGroupInformation.getCurrentUser().getShortUserName() */
{code}
This docs style is not good??
Or at least we can file a jira for {{beginner}}.
> Backup and incremental load could fail in secure clusters
> -
>
> Key: HBASE-20859
> URL: https://issues.apache.org/jira/browse/HBASE-20859
> Project: HBase
> Issue Type: Bug
> Components: backup&restore
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20859.master.001.patch,
> HBASE-20859.master.002.patch, HBASE-20859.master.003.patch
>
>
> HBase Backup and incremental load uses
> HConstants.DEFAULT_TEMPORARY_HDFS_DIRECTORY for temporary path.
> HConstants.DEFAULT_TEMPORARY_HDFS_DIRECTORY uses the Java runtime user name
> to generate a temporary path on HDFS. This can be a wrong assumption in a
> secure cluster where Kerberos principal name can be different from the system
> user name.
> {code:java}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This constant variable is used in BackupUtils.java and HFileOutputFormat2.java
> In such cases, you will not be able to write files to the temporary location
> on HDFS due to permission error, and therefore operations such as backup will
> fail.
> This bug is similar in nature to HDFS-12485.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20846) Table's shared lock is not held by sub-procedures after master restart
[
https://issues.apache.org/jira/browse/HBASE-20846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540992#comment-16540992
]
stack commented on HBASE-20846:
---
bq. We have already done one step, and we expect that no one can interrupt us,
so obviously this behavior is broken.
Good objection.
So, persist wasExecuted before we do first step?
bq. For now, this is done by a very tricky way - we always put the already
being executed table procedure to the front of the queue, even in the yield
method.
Yes, these should take exclusive lock for life of the procedure.
> Table's shared lock is not held by sub-procedures after master restart
> --
>
> Key: HBASE-20846
> URL: https://issues.apache.org/jira/browse/HBASE-20846
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.1.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Major
> Fix For: 3.0.0, 2.0.2, 2.1.1
>
> Attachments: HBASE-20846.branch-2.0.002.patch,
> HBASE-20846.branch-2.0.patch
>
>
> Found this one when investigating ModifyTableProcedure got stuck while there
> was a MoveRegionProcedure going on after master restart.
> Though this issue can be solved by HBASE-20752. But I discovered something
> else.
> Before a MoveRegionProcedure can execute, it will hold the table's shared
> lock. so,, when a UnassignProcedure was spwaned, it will not check the
> table's shared lock since it is sure that its parent(MoveRegionProcedure) has
> aquired the table's lock.
> {code:java}
> // If there is parent procedure, it would have already taken xlock, so no
> need to take
> // shared lock here. Otherwise, take shared lock.
> if (!procedure.hasParent()
> && waitTableQueueSharedLock(procedure, table) == null) {
> return true;
> }
> {code}
> But, it is not the case when Master was restarted. The child
> procedure(UnassignProcedure) will be executed first after restart. Though it
> has a parent(MoveRegionProcedure), but apprently the parent didn't hold the
> table's lock.
> So, since it began to execute without hold the table's shared lock. A
> ModifyTableProcedure can aquire the table's exclusive lock and execute at the
> same time. Which is not possible if the master was not restarted.
> This will cause a stuck before HBASE-20752. But since HBASE-20752 has fixed,
> I wrote a simple UT to repo this case.
> I think we don't have to check the parent for table's shared lock. It is a
> shared lock, right? I think we can acquire it every time we need it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20870) Wrong HBase root dir in ITBLL's Search Tool
[
https://issues.apache.org/jira/browse/HBASE-20870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540975#comment-16540975
]
Allan Yang commented on HBASE-20870:
{quote}
lgtm, assuming you have tested on cluster.
{quote}
Yes, I have it tested on a real cluster, thanks for reviewing,
[~yuzhih...@gmail.com].
> Wrong HBase root dir in ITBLL's Search Tool
> ---
>
> Key: HBASE-20870
> URL: https://issues.apache.org/jira/browse/HBASE-20870
> Project: HBase
> Issue Type: Bug
> Components: integration tests
>Affects Versions: 3.0.0, 2.1.0, 2.0.1
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Minor
> Attachments: HBASE-20870.branch-2.0.001.patch
>
>
> When using IntegrationTestBigLinkedList's Search tools, it always fails since
> it tries to read WALs in a wrong HBase root dir. Turned out that when
> initializing IntegrationTestingUtility in IntegrationTestBigLinkedList, its
> super class HBaseTestingUtility will change hbase.rootdir to a local random
> dir. It is not wrong since HBaseTestingUtility is mostly used by Minicluster.
> But for IntegrationTest runs on distributed clusters, we should change it
> back.
> Here is the error info.
> {code:java}
> 2018-07-11 16:35:49,679 DEBUG [main] hbase.HBaseCommonTestingUtility: Setting
> hbase.rootdir to
> /home/hadoop/target/test-data/deb67611-2737-4696-abe9-32a7783df7bb
> 2018-07-11 16:35:50,736 ERROR [main] util.AbstractHBaseTool: Error running
> command-line tool java.io.FileNotFoundException: File
> file:/home/hadoop/target/test-data/deb67611-2737-4696-abe9-32a7783df7bb/WALs
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:431)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1517)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[ https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-20704: Attachment: HBASE-20704.003.patch > Sometimes some compacted storefiles are not archived on region close > > > Key: HBASE-20704 > URL: https://issues.apache.org/jira/browse/HBASE-20704 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch, > HBASE-20704.003.patch > > > During region close compacted files which have not yet been archived by the > discharger are archived as part of the region closing process. It is > important that these files are wholly archived to insure data consistency. ie > a storefile containing delete tombstones can be archived while older > storefiles containing cells that were supposed to be deleted are left > unarchived thereby undeleting those cells. > On region close a compacted storefile is skipped from archiving if it has > read references (ie open scanners). This behavior is correct for when the > discharger chore runs but on region close consistency is of course more > important so we should add a special case to ignore any references on the > storefile and go ahead and archive it. > Attached patch contains a unit test that reproduces the problem and the > proposed fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[ https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540972#comment-16540972 ] Francis Liu commented on HBASE-20704: - OK I did some testing NPE only actually occurs in the 1.x. For the master branch it's one of two scenarios depending on wether a pread was used or not as stream scanners open their own inputstream, reader, etc. # pread - you'll get a "stream closed" IOException. In 1.x it's an NPE because the stream references are set to null after the streams are closed. # not pread - Since the HStore has no knowledge of the created stream it does not close them. What happens is either the current running scan request is processed or it will get an IOException (replica not found), since the region close operation may have archived and the cleaner chore deleted the file. It sounds to me that #1 is good and #2 is acceptable? Let me know. If #2 is not acceptable I'd have to add a map to keep track of the new readers created when a stream scanner is created and close those on region close so exceptions will be like #1. I've attached another patch which add another test method to surface the exceptions. > Sometimes some compacted storefiles are not archived on region close > > > Key: HBASE-20704 > URL: https://issues.apache.org/jira/browse/HBASE-20704 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch > > > During region close compacted files which have not yet been archived by the > discharger are archived as part of the region closing process. It is > important that these files are wholly archived to insure data consistency. ie > a storefile containing delete tombstones can be archived while older > storefiles containing cells that were supposed to be deleted are left > unarchived thereby undeleting those cells. > On region close a compacted storefile is skipped from archiving if it has > read references (ie open scanners). This behavior is correct for when the > discharger chore runs but on region close consistency is of course more > important so we should add a special case to ignore any references on the > storefile and go ahead and archive it. > Attached patch contains a unit test that reproduces the problem and the > proposed fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20846) Table's shared lock is not held by sub-procedures after master restart
[
https://issues.apache.org/jira/browse/HBASE-20846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540955#comment-16540955
]
Duo Zhang commented on HBASE-20846:
---
{quote}
Do we need a new boolean? What cases do we need to handle on load post-crash?
To my mind, the procedure that holds a lock for the life of the procedure is
the only case that needs re-institution?
{quote}
Yes, this is the truth for now, holdLock is true, and wasExecuted is true, then
we give it the lock when loading procedures. But there is still a problem, as
we will only persist the wasExecuted state into the procedure store after we
finish at least one step of the procedure. So it is possible that, a procedure
has acquired the lock, and done some modifications, and then we crashed. On
restart, we will not give the lock to it, and another procedure may jump it. We
have already done one step, and we expect that no one can interrupt us, so
obviously this behavior is broken.
And when reviewing the table related procedures, I think we should make
holdLock depend on the state of the procedure. For table related procedure,
ideally we should acquire the exclusive lock all the time, until we schedule
the reopen regions procedure. If not there will be problem, think of two
ModifyTableProcedures run at the same time and both want to modify the table
descriptor and layout... For now, this is done by a very tricky way - we always
put the already being executed table procedure to the front of the queue, even
in the yield method. This is not a good design as it will be easily broken in
the future development.
> Table's shared lock is not held by sub-procedures after master restart
> --
>
> Key: HBASE-20846
> URL: https://issues.apache.org/jira/browse/HBASE-20846
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.1.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Major
> Fix For: 3.0.0, 2.0.2, 2.1.1
>
> Attachments: HBASE-20846.branch-2.0.002.patch,
> HBASE-20846.branch-2.0.patch
>
>
> Found this one when investigating ModifyTableProcedure got stuck while there
> was a MoveRegionProcedure going on after master restart.
> Though this issue can be solved by HBASE-20752. But I discovered something
> else.
> Before a MoveRegionProcedure can execute, it will hold the table's shared
> lock. so,, when a UnassignProcedure was spwaned, it will not check the
> table's shared lock since it is sure that its parent(MoveRegionProcedure) has
> aquired the table's lock.
> {code:java}
> // If there is parent procedure, it would have already taken xlock, so no
> need to take
> // shared lock here. Otherwise, take shared lock.
> if (!procedure.hasParent()
> && waitTableQueueSharedLock(procedure, table) == null) {
> return true;
> }
> {code}
> But, it is not the case when Master was restarted. The child
> procedure(UnassignProcedure) will be executed first after restart. Though it
> has a parent(MoveRegionProcedure), but apprently the parent didn't hold the
> table's lock.
> So, since it began to execute without hold the table's shared lock. A
> ModifyTableProcedure can aquire the table's exclusive lock and execute at the
> same time. Which is not possible if the master was not restarted.
> This will cause a stuck before HBASE-20752. But since HBASE-20752 has fixed,
> I wrote a simple UT to repo this case.
> I think we don't have to check the parent for table's shared lock. It is a
> shared lock, right? I think we can acquire it every time we need it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20847) The parent procedure of RegionTransitionProcedure may not have the table lock
[ https://issues.apache.org/jira/browse/HBASE-20847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20847: -- Resolution: Fixed Fix Version/s: 2.1.1 2.2.0 3.0.0 Status: Resolved (was: Patch Available) Pushed the addendum to branch-2,1+. Thanks all for reviewing. > The parent procedure of RegionTransitionProcedure may not have the table lock > - > > Key: HBASE-20847 > URL: https://issues.apache.org/jira/browse/HBASE-20847 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, Region Assignment >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0, 2.2.0, 2.1.1 > > Attachments: HBASE-20847-addendum.patch, > HBASE-20847-branch-2.0-v1.patch, HBASE-20847-branch-2.0.patch, > HBASE-20847-v1.patch, HBASE-20847-v2.patch, HBASE-20847-v3.patch, > HBASE-20847.patch > > > For example, SCP can also schedule AssignProcedure and obviously it will not > hold the table lock. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20869) Endpoint-based Export use incorrect user to write to destination
[
https://issues.apache.org/jira/browse/HBASE-20869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540928#comment-16540928
]
Hadoop QA commented on HBASE-20869:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
34s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} hbase-endpoint: The patch generated 2 new + 0
unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
35s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 14s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
49s{color} | {color:green} hbase-endpoint in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 35m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20869 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12931247/HBASE-20869.master.001.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 03a4deae3b14 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 0d33caa39a |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/13600/artifact/patchprocess/diff-checkstyle-hbase-endpoint.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13600/testReport/ |
| Max. process+thread count | 3086 (vs. ulimit of 1) |
| modules | C: hbase-endpoint U: hbase-endpoint |
| Console outp
[jira] [Commented] (HBASE-20800) Master-orchestrated compactions
[ https://issues.apache.org/jira/browse/HBASE-20800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540929#comment-16540929 ] Duo Zhang commented on HBASE-20800: --- Can do this on a feature branch? > Master-orchestrated compactions > --- > > Key: HBASE-20800 > URL: https://issues.apache.org/jira/browse/HBASE-20800 > Project: HBase > Issue Type: Umbrella > Components: Compaction >Reporter: stack >Assignee: Mohit Goel >Priority: Major > > An umbrella issue for having compactions go via the Master so we can have a > centralized arbitrator of cluster i/o. If we put Master in the way, we can do > stuff like: > * Ask the Master for current cluster compaction state; what is running, what > is blocked > * Master can manage cluster-wide compaction policy and/or throttling/or > blocking of compaction i/os. > * Master can schedule when and where compactions run so we can guard against > the pathological where all RegionServers decide now is the time to major > compact bringing on a compaction storm. > Other side-benefits might include being able to farm out the compaction work > to another process -- e.g. the splice machine model of having spark run the > compactions -- or just to a separate compactor that we might i/o nice. > * We'll need to figure how to externalize the CompactionRequest so it can be > passed over RPC. > * We'll need to have something like a CompactionManager in the Master > process that keeps up current cluster state. > MOB needs a compaction fabric it can use. Its compactions are currently > Master-based only and so don't scale. It could make use of this mechanism to > ask the Master to farm out its compaction requests. > This is an umbrella issue. I thought I'd filed one already on this topic but > can't find it. Will shut it down if I trip over it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-20874) Sending compaction descriptions from all regionservers to master.
[ https://issues.apache.org/jira/browse/HBASE-20874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Goel reassigned HBASE-20874: -- Assignee: Mohit Goel > Sending compaction descriptions from all regionservers to master. > - > > Key: HBASE-20874 > URL: https://issues.apache.org/jira/browse/HBASE-20874 > Project: HBase > Issue Type: Sub-task >Reporter: Mohit Goel >Assignee: Mohit Goel >Priority: Minor > > Need to send the compaction description from region servers to Master , to > let master know of the entire compaction state of the cluster. Further need > to change the implementation of client Side API than like getCompactionState, > which will consult master for the result instead of sending individual > request to regionservers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20874) Sending compaction descriptions from all regionservers to master.
Mohit Goel created HBASE-20874: -- Summary: Sending compaction descriptions from all regionservers to master. Key: HBASE-20874 URL: https://issues.apache.org/jira/browse/HBASE-20874 Project: HBase Issue Type: Sub-task Reporter: Mohit Goel Need to send the compaction description from region servers to Master , to let master know of the entire compaction state of the cluster. Further need to change the implementation of client Side API than like getCompactionState, which will consult master for the result instead of sending individual request to regionservers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20859) Backup and incremental load could fail in secure clusters
[
https://issues.apache.org/jira/browse/HBASE-20859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540910#comment-16540910
]
Wei-Chiu Chuang commented on HBASE-20859:
-
Thanks [~yuzhih...@gmail.com]!
> Backup and incremental load could fail in secure clusters
> -
>
> Key: HBASE-20859
> URL: https://issues.apache.org/jira/browse/HBASE-20859
> Project: HBase
> Issue Type: Bug
> Components: backup&restore
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20859.master.001.patch,
> HBASE-20859.master.002.patch, HBASE-20859.master.003.patch
>
>
> HBase Backup and incremental load uses
> HConstants.DEFAULT_TEMPORARY_HDFS_DIRECTORY for temporary path.
> HConstants.DEFAULT_TEMPORARY_HDFS_DIRECTORY uses the Java runtime user name
> to generate a temporary path on HDFS. This can be a wrong assumption in a
> secure cluster where Kerberos principal name can be different from the system
> user name.
> {code:java}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This constant variable is used in BackupUtils.java and HFileOutputFormat2.java
> In such cases, you will not be able to write files to the temporary location
> on HDFS due to permission error, and therefore operations such as backup will
> fail.
> This bug is similar in nature to HDFS-12485.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20859) Backup and incremental load could fail in secure clusters
[
https://issues.apache.org/jira/browse/HBASE-20859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-20859:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks for the patch, Wei-Chiu
Thanks for the reviews.
> Backup and incremental load could fail in secure clusters
> -
>
> Key: HBASE-20859
> URL: https://issues.apache.org/jira/browse/HBASE-20859
> Project: HBase
> Issue Type: Bug
> Components: backup&restore
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HBASE-20859.master.001.patch,
> HBASE-20859.master.002.patch, HBASE-20859.master.003.patch
>
>
> HBase Backup and incremental load uses
> HConstants.DEFAULT_TEMPORARY_HDFS_DIRECTORY for temporary path.
> HConstants.DEFAULT_TEMPORARY_HDFS_DIRECTORY uses the Java runtime user name
> to generate a temporary path on HDFS. This can be a wrong assumption in a
> secure cluster where Kerberos principal name can be different from the system
> user name.
> {code:java}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This constant variable is used in BackupUtils.java and HFileOutputFormat2.java
> In such cases, you will not be able to write files to the temporary location
> on HDFS due to permission error, and therefore operations such as backup will
> fail.
> This bug is similar in nature to HDFS-12485.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20870) Wrong HBase root dir in ITBLL's Search Tool
[
https://issues.apache.org/jira/browse/HBASE-20870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540905#comment-16540905
]
Ted Yu commented on HBASE-20870:
lgtm, assuming you have tested on cluster.
> Wrong HBase root dir in ITBLL's Search Tool
> ---
>
> Key: HBASE-20870
> URL: https://issues.apache.org/jira/browse/HBASE-20870
> Project: HBase
> Issue Type: Bug
> Components: integration tests
>Affects Versions: 3.0.0, 2.1.0, 2.0.1
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Minor
> Attachments: HBASE-20870.branch-2.0.001.patch
>
>
> When using IntegrationTestBigLinkedList's Search tools, it always fails since
> it tries to read WALs in a wrong HBase root dir. Turned out that when
> initializing IntegrationTestingUtility in IntegrationTestBigLinkedList, its
> super class HBaseTestingUtility will change hbase.rootdir to a local random
> dir. It is not wrong since HBaseTestingUtility is mostly used by Minicluster.
> But for IntegrationTest runs on distributed clusters, we should change it
> back.
> Here is the error info.
> {code:java}
> 2018-07-11 16:35:49,679 DEBUG [main] hbase.HBaseCommonTestingUtility: Setting
> hbase.rootdir to
> /home/hadoop/target/test-data/deb67611-2737-4696-abe9-32a7783df7bb
> 2018-07-11 16:35:50,736 ERROR [main] util.AbstractHBaseTool: Error running
> command-line tool java.io.FileNotFoundException: File
> file:/home/hadoop/target/test-data/deb67611-2737-4696-abe9-32a7783df7bb/WALs
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:431)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1517)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-20873) Update doc for Endpoint-based Export
Wei-Chiu Chuang created HBASE-20873: --- Summary: Update doc for Endpoint-based Export Key: HBASE-20873 URL: https://issues.apache.org/jira/browse/HBASE-20873 Project: HBase Issue Type: Improvement Components: documentation Affects Versions: 2.0.0 Reporter: Wei-Chiu Chuang Assignee: Wei-Chiu Chuang The current documentation on the usage is a little vague. I'd like to take a stab at expanding it, based on my experience. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20869) Endpoint-based Export use incorrect user to write to destination
[
https://issues.apache.org/jira/browse/HBASE-20869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HBASE-20869:
Status: Patch Available (was: Open)
> Endpoint-based Export use incorrect user to write to destination
>
>
> Key: HBASE-20869
> URL: https://issues.apache.org/jira/browse/HBASE-20869
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors
>Affects Versions: 2.0.0
> Environment: Hadoop 3.0.0 + HBase 2.0.0, Kerberos.
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HBASE-20869.master.001.patch
>
>
> HBASE-15806 implemented an endpoint based export. It gets caller's HDFS
> delegation token, and RegionServer is supposed to write out exported files as
> the caller.
> Everything works fine if you use run export as hbase user. However, once you
> use a different user to export, it fails.
> To reproduce,
> Add to configuration key hbase.coprocessor.region.classes the coprocessor
> class org.apache.hadoop.hbase.coprocessor.Export.
> create a table t1, assign permission to a user foo:
>
> {noformat}
> hbase(main):004:0> user_permission 't1'
> User Namespace,Table,Family,Qualifier:Permission
> hbase default,t1,,: [Permission: actions=READ,WRITE,EXEC,CREATE,ADMIN]
> foo default,t1,,: [Permission: actions=READ,WRITE,EXEC,CREATE,ADMIN]{noformat}
>
> As user foo, execute the following command:
>
> {noformat}
> $ hdfs dfs -mkdir /tmp/export_hbase2
> $ hbase org.apache.hadoop.hbase.coprocessor.Export t1 /tmp/export_hbase2/t2/
>
> 18/07/10 14:03:59 INFO client.RpcRetryingCallerImpl: Call exception, tries=6,
> retries=6, started=4457 ms ago, cancelled=false,
> msg=org.apache.hadoop.security.AccessControlException: Permission denied:
> user=hbase, access=WRITE,
> inode="/tmp/export_hbase2/t2":foo:supergroup:drwxr-xr-x
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:400)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:256)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:194)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1846)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1830)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1789)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.resolvePathForStartFile(FSDirWriteFileOp.java:316)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2411)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2343)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:764)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:451)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
> at sun.reflect.GeneratedConstructorAccessor25.newInstance(Unknown Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:278)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1195)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1174)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1112)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:462)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:459)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
[jira] [Updated] (HBASE-20869) Endpoint-based Export use incorrect user to write to destination
[
https://issues.apache.org/jira/browse/HBASE-20869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HBASE-20869:
Attachment: HBASE-20869.master.001.patch
> Endpoint-based Export use incorrect user to write to destination
>
>
> Key: HBASE-20869
> URL: https://issues.apache.org/jira/browse/HBASE-20869
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors
>Affects Versions: 2.0.0
> Environment: Hadoop 3.0.0 + HBase 2.0.0, Kerberos.
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HBASE-20869.master.001.patch
>
>
> HBASE-15806 implemented an endpoint based export. It gets caller's HDFS
> delegation token, and RegionServer is supposed to write out exported files as
> the caller.
> Everything works fine if you use run export as hbase user. However, once you
> use a different user to export, it fails.
> To reproduce,
> Add to configuration key hbase.coprocessor.region.classes the coprocessor
> class org.apache.hadoop.hbase.coprocessor.Export.
> create a table t1, assign permission to a user foo:
>
> {noformat}
> hbase(main):004:0> user_permission 't1'
> User Namespace,Table,Family,Qualifier:Permission
> hbase default,t1,,: [Permission: actions=READ,WRITE,EXEC,CREATE,ADMIN]
> foo default,t1,,: [Permission: actions=READ,WRITE,EXEC,CREATE,ADMIN]{noformat}
>
> As user foo, execute the following command:
>
> {noformat}
> $ hdfs dfs -mkdir /tmp/export_hbase2
> $ hbase org.apache.hadoop.hbase.coprocessor.Export t1 /tmp/export_hbase2/t2/
>
> 18/07/10 14:03:59 INFO client.RpcRetryingCallerImpl: Call exception, tries=6,
> retries=6, started=4457 ms ago, cancelled=false,
> msg=org.apache.hadoop.security.AccessControlException: Permission denied:
> user=hbase, access=WRITE,
> inode="/tmp/export_hbase2/t2":foo:supergroup:drwxr-xr-x
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:400)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:256)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:194)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1846)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1830)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1789)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.resolvePathForStartFile(FSDirWriteFileOp.java:316)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2411)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2343)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:764)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:451)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
> at sun.reflect.GeneratedConstructorAccessor25.newInstance(Unknown Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:278)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1195)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1174)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1112)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:462)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$8.doCall(DistributedFileSystem.java:459)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>
[jira] [Commented] (HBASE-20175) hbase-spark needs scala dependency convergance
[
https://issues.apache.org/jira/browse/HBASE-20175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540878#comment-16540878
]
Hadoop QA commented on HBASE-20175:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
34s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
6s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
11m 54s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}202m
37s{color} | {color:green} root in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
52s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}259m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20175 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12931207/HBASE-20175.v04.patch
|
| Optional Tests | asflicense javac javadoc unit shadedjars hadoopcheck
xml compile |
| uname | Linux 806715fa4ced 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 0d33caa39a |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_171 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13597/testReport/ |
| Max. process+thread count | 5009 (vs. ulimit of 1) |
| modules | C: hbase-spark . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/13597/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> hbase-spark needs scala dependency convergance
> --
>
> Key: HBASE-20175
> URL: https://issues.apache.org/jira/browse/HBASE-20175
>
[jira] [Updated] (HBASE-20578) Support region server group in target cluster
[ https://issues.apache.org/jira/browse/HBASE-20578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-20578: --- Component/s: rsgroup > Support region server group in target cluster > - > > Key: HBASE-20578 > URL: https://issues.apache.org/jira/browse/HBASE-20578 > Project: HBase > Issue Type: Improvement > Components: Replication, rsgroup >Reporter: Ted Yu >Assignee: Albert Lee >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20578-001.patch > > > When source tables belong to non-default region server group(s) and there are > region server group counterpart in the target cluster, we should support > replicating to target cluster using the region server group mapping. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20734) Colocate recovered edits directory with hbase.wal.dir
[ https://issues.apache.org/jira/browse/HBASE-20734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540796#comment-16540796 ] Zach York commented on HBASE-20734: --- Yep, I'll work on getting a patch for master branch. It was just easier for me to test on cluster with branch-1. > Colocate recovered edits directory with hbase.wal.dir > - > > Key: HBASE-20734 > URL: https://issues.apache.org/jira/browse/HBASE-20734 > Project: HBase > Issue Type: Improvement > Components: MTTR, Recovery, wal >Reporter: Ted Yu >Assignee: Zach York >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20734.branch-1.001.patch > > > During investigation of HBASE-20723, I realized that we wouldn't get the best > performance when hbase.wal.dir is configured to be on different (fast) media > than hbase rootdir w.r.t. recovered edits since recovered edits directory is > currently under rootdir. > Such setup may not result in fast recovery when there is region server > failover. > This issue is to find proper (hopefully backward compatible) way in > colocating recovered edits directory with hbase.wal.dir . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20838) Move all setStorage related UT cases from TestFSUtils to TestCommonFSUtils
[ https://issues.apache.org/jira/browse/HBASE-20838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540764#comment-16540764 ] Sean Busbey commented on HBASE-20838: - +1 > Move all setStorage related UT cases from TestFSUtils to TestCommonFSUtils > -- > > Key: HBASE-20838 > URL: https://issues.apache.org/jira/browse/HBASE-20838 > Project: HBase > Issue Type: Test >Reporter: Yu Li >Assignee: Yu Li >Priority: Major > Attachments: HBASE-20838.patch, HBASE-20838.patch, > HBASE-20838.v2.patch > > > As per > [discussed|https://issues.apache.org/jira/browse/HBASE-20691?focusedCommentId=16517662&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16517662] > in HBASE-20691, since the setStoragePolicy code is in CommonFSUtils, the > test should be in TestCommonFSUtils -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20847) The parent procedure of RegionTransitionProcedure may not have the table lock
[
https://issues.apache.org/jira/browse/HBASE-20847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540766#comment-16540766
]
Hudson commented on HBASE-20847:
Results for branch branch-2.1
[build #49 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/49/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/49//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/49//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/49//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> The parent procedure of RegionTransitionProcedure may not have the table lock
> -
>
> Key: HBASE-20847
> URL: https://issues.apache.org/jira/browse/HBASE-20847
> Project: HBase
> Issue Type: Sub-task
> Components: proc-v2, Region Assignment
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Attachments: HBASE-20847-addendum.patch,
> HBASE-20847-branch-2.0-v1.patch, HBASE-20847-branch-2.0.patch,
> HBASE-20847-v1.patch, HBASE-20847-v2.patch, HBASE-20847-v3.patch,
> HBASE-20847.patch
>
>
> For example, SCP can also schedule AssignProcedure and obviously it will not
> hold the table lock.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20847) The parent procedure of RegionTransitionProcedure may not have the table lock
[
https://issues.apache.org/jira/browse/HBASE-20847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540756#comment-16540756
]
Hudson commented on HBASE-20847:
Results for branch branch-2
[build #971 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/971/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/971//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/971//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/971//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> The parent procedure of RegionTransitionProcedure may not have the table lock
> -
>
> Key: HBASE-20847
> URL: https://issues.apache.org/jira/browse/HBASE-20847
> Project: HBase
> Issue Type: Sub-task
> Components: proc-v2, Region Assignment
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Attachments: HBASE-20847-addendum.patch,
> HBASE-20847-branch-2.0-v1.patch, HBASE-20847-branch-2.0.patch,
> HBASE-20847-v1.patch, HBASE-20847-v2.patch, HBASE-20847-v3.patch,
> HBASE-20847.patch
>
>
> For example, SCP can also schedule AssignProcedure and obviously it will not
> hold the table lock.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20617) Upgrade/remove jetty-jsp
[
https://issues.apache.org/jira/browse/HBASE-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540757#comment-16540757
]
Hudson commented on HBASE-20617:
Results for branch branch-2
[build #971 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/971/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/971//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/971//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/971//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Upgrade/remove jetty-jsp
>
>
> Key: HBASE-20617
> URL: https://issues.apache.org/jira/browse/HBASE-20617
> Project: HBase
> Issue Type: Improvement
>Reporter: Sakthi
>Assignee: Sakthi
>Priority: Minor
> Fix For: 3.0.0, 2.2.0
>
> Attachments: hbase-20617.master.001.patch
>
>
> jetty-jsp removed after jetty-9.2.x version. We use the 9.2 version. Research
> so far brings out that apache-jsp might be of interest to us in jetty-9.4.x
> version(as JettyJspServlet.class is in apache-jsp). Yet to figure out about
> jetty-9.3.x.
> Filing to track this along.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20542) Better heap utilization for IMC with MSLABs
[
https://issues.apache.org/jira/browse/HBASE-20542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540755#comment-16540755
]
Hudson commented on HBASE-20542:
Results for branch branch-2
[build #971 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/971/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/971//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/971//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/971//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Better heap utilization for IMC with MSLABs
> ---
>
> Key: HBASE-20542
> URL: https://issues.apache.org/jira/browse/HBASE-20542
> Project: HBase
> Issue Type: Sub-task
> Components: in-memory-compaction
>Reporter: Eshcar Hillel
>Assignee: Eshcar Hillel
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-20542-addendum.master.005.patch,
> HBASE-20542.branch-2.001.patch, HBASE-20542.branch-2.003.patch,
> HBASE-20542.branch-2.004.patch, HBASE-20542.branch-2.005.patch,
> HBASE-20542.master.003.patch, HBASE-20542.master.005-addendum.patch, run.sh,
> workloada, workloadc, workloadx, workloady
>
>
> Following HBASE-20188 we realized in-memory compaction combined with MSLABs
> may suffer from heap under-utilization due to internal fragmentation. This
> jira presents a solution to circumvent this problem. The main idea is to have
> each update operation check if it will cause overflow in the active segment
> *before* it is writing the new value (instead of checking the size after the
> write is completed), and if it is then the active segment is atomically
> swapped with a new empty segment, and is pushed (full-yet-not-overflowed) to
> the compaction pipeline. Later on the IMC deamon will run its compaction
> operation (flatten index/merge indices/data compaction) in the background.
> Some subtle concurrency issues should be handled with care. We next elaborate
> on them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20734) Colocate recovered edits directory with hbase.wal.dir
[
https://issues.apache.org/jira/browse/HBASE-20734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540750#comment-16540750
]
Hadoop QA commented on HBASE-20734:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
55s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} branch-1 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} branch-1 passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
27s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
38s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} branch-1 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} branch-1 passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
25s{color} | {color:red} hbase-server: The patch generated 12 new + 659
unchanged - 12 fixed = 671 total (was 671) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
29s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
1m 32s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 28m 56s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 44s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.io.TestHeapSize |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:1f3957d |
| JIRA Issue | HBASE-20734 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12931222/HBASE-20734.branch-1.001.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 9fa7a9
[jira] [Commented] (HBASE-20866) HBase 1.x scan performance degradation compared to 0.98 version
[
https://issues.apache.org/jira/browse/HBASE-20866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540742#comment-16540742
]
Ted Yu commented on HBASE-20866:
Vikas:
Can you include trivial change to hbase-server module so that the tests in that
module get run by QA ?
Thanks
> HBase 1.x scan performance degradation compared to 0.98 version
> ---
>
> Key: HBASE-20866
> URL: https://issues.apache.org/jira/browse/HBASE-20866
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.2
>Reporter: Vikas Vishwakarma
>Assignee: Vikas Vishwakarma
>Priority: Critical
> Fix For: 1.5.0, 1.2.7, 1.3.3, 1.4.6
>
> Attachments: HBASE-20866.branch-1.3.001.patch,
> HBASE-20866.branch-1.3.002.patch
>
>
> Internally while testing 1.3 as part of migration from 0.98 to 1.3 we
> observed perf degradation in scan performance for phoenix queries varying
> from few 10's to upto 200% depending on the query being executed. We tried
> simple native HBase scan and there also we saw upto 40% degradation in
> performance when the number of column qualifiers are high (40-50+)
> To identify the root cause of performance diff between 0.98 and 1.3 we
> carried out lot of experiments with profiling and git bisect iterations,
> however we were not able to identify any particular source of scan
> performance degradation and it looked like this is an accumulated degradation
> of 5-10% over various enhancements and refactoring.
> We identified few major enhancements like partialResult handling,
> ScannerContext with heartbeat processing, time/size limiting, RPC
> refactoring, etc that could have contributed to small degradation in
> performance which put together could be leading to large overall degradation.
> One of the changes is
> [HBASE-11544|https://jira.apache.org/jira/browse/HBASE-11544] which
> implements partialResult handling. In ClientScanner.java the results received
> from server are cached on the client side by converting the result array into
> an ArrayList. This function gets called in a loop depending on the number of
> rows in the scan result. Example for ten’s of millions of rows scanned, this
> can be called in the order of millions of times.
> In almost all the cases 99% of the time (except for handling partial results,
> etc). We are just taking the resultsFromServer converting it into a ArrayList
> resultsToAddToCache in addResultsToList(..) and then iterating over the list
> again and adding it to cache in loadCache(..) as given in the code path below
> In ClientScanner → loadCache(..) → getResultsToAddToCache(..) →
> addResultsToList(..) →
> {code:java}
> loadCache() {
> ...
> List resultsToAddToCache =
> getResultsToAddToCache(values, callable.isHeartbeatMessage());
> ...
> …
> for (Result rs : resultsToAddToCache) {
> rs = filterLoadedCell(rs);
> cache.add(rs);
> ...
> }
> }
> getResultsToAddToCache(..) {
> ..
> final boolean isBatchSet = scan != null && scan.getBatch() > 0;
> final boolean allowPartials = scan != null &&
> scan.getAllowPartialResults();
> ..
> if (allowPartials || isBatchSet) {
> addResultsToList(resultsToAddToCache, resultsFromServer, 0,
> (null == resultsFromServer ? 0 : resultsFromServer.length));
> return resultsToAddToCache;
> }
> ...
> }
> private void addResultsToList(List outputList, Result[] inputArray,
> int start, int end) {
> if (inputArray == null || start < 0 || end > inputArray.length) return;
> for (int i = start; i < end; i++) {
> outputList.add(inputArray[i]);
> }
> }{code}
>
> It looks like we can avoid the result array to arraylist conversion
> (resultsFromServer --> resultsToAddToCache ) for the first case which is also
> the most frequent case and instead directly take the values arraay returned
> by callable and add it to the cache without converting it into ArrayList.
> I have taken both these flags allowPartials and isBatchSet out in loadcahe()
> and I am directly adding values to scanner cache if the above condition is
> pass instead of coverting it into arrayList by calling
> getResultsToAddToCache(). For example:
> {code:java}
> protected void loadCache() throws IOException {
> Result[] values = null;
> ..
> final boolean isBatchSet = scan != null && scan.getBatch() > 0;
> final boolean allowPartials = scan != null && scan.getAllowPartialResults();
> ..
> for (;;) {
> try {
> values = call(callable, caller, scannerTimeout);
> ..
> } catch (DoNotRetryIOException | NeedUnmanagedConnectionException e) {
> ..
> }
> if (allowPartials || isBatchSet) { // DIRECTLY COPY values TO CACHE
> if (values != null) {
> for (int v=0; v Result rs = values[v];
>
> cache.add(rs);
> ...
> } else { // DO ALL THE RE
[jira] [Updated] (HBASE-18840) Add functionality to refresh meta table at master startup
[ https://issues.apache.org/jira/browse/HBASE-18840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach York updated HBASE-18840: -- Attachment: HBASE-18840.HBASE-18477.007.patch > Add functionality to refresh meta table at master startup > - > > Key: HBASE-18840 > URL: https://issues.apache.org/jira/browse/HBASE-18840 > Project: HBase > Issue Type: Sub-task >Affects Versions: HBASE-18477 >Reporter: Zach York >Assignee: Zach York >Priority: Major > Attachments: HBASE-18840.HBASE-18477.001.patch, > HBASE-18840.HBASE-18477.002.patch, HBASE-18840.HBASE-18477.003 (2) (1).patch, > HBASE-18840.HBASE-18477.003 (2).patch, HBASE-18840.HBASE-18477.003.patch, > HBASE-18840.HBASE-18477.004.patch, HBASE-18840.HBASE-18477.005.patch, > HBASE-18840.HBASE-18477.006.patch, HBASE-18840.HBASE-18477.007.patch > > > If a HBase cluster’s hbase:meta table is deleted or a cluster is started with > a new meta table, HBase needs the functionality to synchronize it’s metadata > from Storage. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20868) Fix TestCheckTestClasses on HBASE-18477
[ https://issues.apache.org/jira/browse/HBASE-20868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach York updated HBASE-20868: -- Resolution: Fixed Status: Resolved (was: Patch Available) pushed to HBASE-18477 > Fix TestCheckTestClasses on HBASE-18477 > --- > > Key: HBASE-20868 > URL: https://issues.apache.org/jira/browse/HBASE-20868 > Project: HBase > Issue Type: Sub-task >Affects Versions: HBASE-18477 >Reporter: Zach York >Assignee: Zach York >Priority: Minor > Fix For: HBASE-18477 > > Attachments: HBASE-20868.HBASE-18477.001.patch, > HBASE-20868.HBASE-18477.002.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20734) Colocate recovered edits directory with hbase.wal.dir
[ https://issues.apache.org/jira/browse/HBASE-20734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540733#comment-16540733 ] Ted Yu commented on HBASE-20734: If it is not too much trouble, can you base patch on master branch ? Please also publish it on reviewboard. > Colocate recovered edits directory with hbase.wal.dir > - > > Key: HBASE-20734 > URL: https://issues.apache.org/jira/browse/HBASE-20734 > Project: HBase > Issue Type: Improvement > Components: MTTR, Recovery, wal >Reporter: Ted Yu >Assignee: Zach York >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20734.branch-1.001.patch > > > During investigation of HBASE-20723, I realized that we wouldn't get the best > performance when hbase.wal.dir is configured to be on different (fast) media > than hbase rootdir w.r.t. recovered edits since recovered edits directory is > currently under rootdir. > Such setup may not result in fast recovery when there is region server > failover. > This issue is to find proper (hopefully backward compatible) way in > colocating recovered edits directory with hbase.wal.dir . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20651) Master, prevents hbck or shell command to reassign the split parent region
[ https://issues.apache.org/jira/browse/HBASE-20651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-20651: --- Fix Version/s: 1.4.6 1.3.3 1.2.7 > Master, prevents hbck or shell command to reassign the split parent region > -- > > Key: HBASE-20651 > URL: https://issues.apache.org/jira/browse/HBASE-20651 > Project: HBase > Issue Type: Improvement > Components: master >Affects Versions: 1.2.6 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Fix For: 1.5.0, 1.2.7, 1.3.3, 1.4.6 > > Attachments: HBASE-20651-branch-1-v001.patch, > HBASE-20651-branch-1-v002.patch, HBASE-20651-branch-1-v003.patch > > > We are seeing that hbck brings back split parent region and this causes > region inconsistency. More details will be filled as reproduce is still > ongoing. Might need to do something at hbck or master to prevent this from > happening. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20734) Colocate recovered edits directory with hbase.wal.dir
[ https://issues.apache.org/jira/browse/HBASE-20734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach York updated HBASE-20734: -- Status: Patch Available (was: Open) > Colocate recovered edits directory with hbase.wal.dir > - > > Key: HBASE-20734 > URL: https://issues.apache.org/jira/browse/HBASE-20734 > Project: HBase > Issue Type: Improvement > Components: MTTR, Recovery, wal >Reporter: Ted Yu >Assignee: Zach York >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20734.branch-1.001.patch > > > During investigation of HBASE-20723, I realized that we wouldn't get the best > performance when hbase.wal.dir is configured to be on different (fast) media > than hbase rootdir w.r.t. recovered edits since recovered edits directory is > currently under rootdir. > Such setup may not result in fast recovery when there is region server > failover. > This issue is to find proper (hopefully backward compatible) way in > colocating recovered edits directory with hbase.wal.dir . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20734) Colocate recovered edits directory with hbase.wal.dir
[ https://issues.apache.org/jira/browse/HBASE-20734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach York updated HBASE-20734: -- Attachment: HBASE-20734.branch-1.001.patch > Colocate recovered edits directory with hbase.wal.dir > - > > Key: HBASE-20734 > URL: https://issues.apache.org/jira/browse/HBASE-20734 > Project: HBase > Issue Type: Improvement > Components: MTTR, Recovery, wal >Reporter: Ted Yu >Assignee: Zach York >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-20734.branch-1.001.patch > > > During investigation of HBASE-20723, I realized that we wouldn't get the best > performance when hbase.wal.dir is configured to be on different (fast) media > than hbase rootdir w.r.t. recovered edits since recovered edits directory is > currently under rootdir. > Such setup may not result in fast recovery when there is region server > failover. > This issue is to find proper (hopefully backward compatible) way in > colocating recovered edits directory with hbase.wal.dir . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-18477) Umbrella JIRA for HBase Read Replica clusters
[ https://issues.apache.org/jira/browse/HBASE-18477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540615#comment-16540615 ] Sean Busbey commented on HBASE-18477: - FYI, haven't forgotten about this. just a bit underwater. should be back with bandwidth next week. > Umbrella JIRA for HBase Read Replica clusters > - > > Key: HBASE-18477 > URL: https://issues.apache.org/jira/browse/HBASE-18477 > Project: HBase > Issue Type: New Feature >Reporter: Zach York >Assignee: Zach York >Priority: Major > Attachments: HBase Read-Replica Clusters Scope doc.docx, HBase > Read-Replica Clusters Scope doc.pdf, HBase Read-Replica Clusters Scope > doc_v2.docx, HBase Read-Replica Clusters Scope doc_v2.pdf > > > Recently, changes (such as HBASE-17437) have unblocked HBase to run with a > root directory external to the cluster (such as in Amazon S3). This means > that the data is stored outside of the cluster and can be accessible after > the cluster has been terminated. One use case that is often asked about is > pointing multiple clusters to one root directory (sharing the data) to have > read resiliency in the case of a cluster failure. > > This JIRA is an umbrella JIRA to contain all the tasks necessary to create a > read-replica HBase cluster that is pointed at the same root directory. > > This requires making the Read-Replica cluster Read-Only (no metadata > operation or data operations). > Separating the hbase:meta table for each cluster (Otherwise HBase gets > confused with multiple clusters trying to update the meta table with their ip > addresses) > Adding refresh functionality for the meta table to ensure new metadata is > picked up on the read replica cluster. > Adding refresh functionality for HFiles for a given table to ensure new data > is picked up on the read replica cluster. > > This can be used with any existing cluster that is backed by an external > filesystem. > > Please note that this feature is still quite manual (with the potential for > automation later). > > More information on this particular feature can be found here: > https://aws.amazon.com/blogs/big-data/setting-up-read-replica-clusters-with-hbase-on-amazon-s3/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20697) Can't cache All region locations of the specify table by calling table.getRegionLocator().getAllRegionLocations()
[
https://issues.apache.org/jira/browse/HBASE-20697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540608#comment-16540608
]
Hudson commented on HBASE-20697:
Results for branch branch-1.4
[build #382 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/382/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/382//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/382//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/382//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> Can't cache All region locations of the specify table by calling
> table.getRegionLocator().getAllRegionLocations()
> -
>
> Key: HBASE-20697
> URL: https://issues.apache.org/jira/browse/HBASE-20697
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.1, 1.2.6, 2.0.1
>Reporter: zhaoyuan
>Assignee: zhaoyuan
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 1.4.6, 2.0.2, 2.2.0, 2.1.1
>
> Attachments: HBASE-20697.branch-1.2.001.patch,
> HBASE-20697.branch-1.2.002.patch, HBASE-20697.branch-1.2.003.patch,
> HBASE-20697.branch-1.2.004.patch, HBASE-20697.master.001.patch,
> HBASE-20697.master.002.patch, HBASE-20697.master.002.patch,
> HBASE-20697.master.003.patch
>
>
> When we upgrade and restart a new version application which will read and
> write to HBase, we will get some operation timeout. The time out is expected
> because when the application restarts，It will not hold any region locations
> cache and do communication with zk and meta regionserver to get region
> locations.
> We want to avoid these timeouts so we do warmup work and as far as I am
> concerned,the method table.getRegionLocator().getAllRegionLocations() will
> fetch all region locations and cache them. However, it didn't work good.
> There are still a lot of time outs,so it confused me.
> I dig into the source code and find something below
> {code:java}
> // code placeholder
> public List getAllRegionLocations() throws IOException {
> TableName tableName = getName();
> NavigableMap locations =
> MetaScanner.allTableRegions(this.connection, tableName);
> ArrayList regions = new ArrayList<>(locations.size());
> for (Entry entry : locations.entrySet()) {
> regions.add(new HRegionLocation(entry.getKey(), entry.getValue()));
> }
> if (regions.size() > 0) {
> connection.cacheLocation(tableName, new RegionLocations(regions));
> }
> return regions;
> }
> In MetaCache
> public void cacheLocation(final TableName tableName, final RegionLocations
> locations) {
> byte [] startKey =
> locations.getRegionLocation().getRegionInfo().getStartKey();
> ConcurrentMap tableLocations =
> getTableLocations(tableName);
> RegionLocations oldLocation = tableLocations.putIfAbsent(startKey,
> locations);
> boolean isNewCacheEntry = (oldLocation == null);
> if (isNewCacheEntry) {
> if (LOG.isTraceEnabled()) {
> LOG.trace("Cached location: " + locations);
> }
> addToCachedServers(locations);
> return;
> }
> {code}
> It will collect all regions into one RegionLocations object and only cache
> the first not null region location and then when we put or get to hbase, we
> do getCacheLocation()
> {code:java}
> // code placeholder
> public RegionLocations getCachedLocation(final TableName tableName, final
> byte [] row) {
> ConcurrentNavigableMap tableLocations =
> getTableLocations(tableName);
> Entry e = tableLocations.floorEntry(row);
> if (e == null) {
> if (metrics!= null) metrics.incrMetaCacheMiss();
> return null;
> }
> RegionLocations possibleRegion = e.getValue();
> // make sure that the end key is greater than the row we're looking
> // for, otherwise the row actually belongs in the next region, not
> // this one. the exception case is when the endkey is
> // HConstants.EMPTY_END_ROW, signifying that the region we're
> // checking is actually the last region in the table.
> byte[] endKey =
> possibleRegion.getRegionLocation().getRegionInfo().getEndKey();
> if (Bytes.equals(endKey, HConstants.EMPTY_END_ROW) ||
> getRowComparator(tableName).compareRows(
> endKey, 0, endKey.length, row, 0, row.length) > 0) {
> if (metrics != null) metrics.incr
[jira] [Commented] (HBASE-20257) hbase-spark should not depend on com.google.code.findbugs.jsr305
[
https://issues.apache.org/jira/browse/HBASE-20257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540606#comment-16540606
]
Sean Busbey commented on HBASE-20257:
-
sweet. checking again.
> hbase-spark should not depend on com.google.code.findbugs.jsr305
>
>
> Key: HBASE-20257
> URL: https://issues.apache.org/jira/browse/HBASE-20257
> Project: HBase
> Issue Type: Task
> Components: build, spark
>Affects Versions: 3.0.0
>Reporter: Ted Yu
>Assignee: Artem Ervits
>Priority: Minor
> Labels: beginner
> Attachments: HBASE-20257.v01.patch, HBASE-20257.v02.patch,
> HBASE-20257.v03.patch, HBASE-20257.v04.patch, HBASE-20257.v05.patch
>
>
> The following can be observed in the build output of master branch:
> {code}
> [WARNING] Rule 0: org.apache.maven.plugins.enforcer.BannedDependencies failed
> with message:
> We don't allow the JSR305 jar from the Findbugs project, see HBASE-16321.
> Found Banned Dependency: com.google.code.findbugs:jsr305:jar:1.3.9
> Use 'mvn dependency:tree' to locate the source of the banned dependencies.
> {code}
> Here is related snippet from hbase-spark/pom.xml:
> {code}
>
> com.google.code.findbugs
> jsr305
> {code}
> Dependency on jsr305 should be dropped.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20651) Master, prevents hbck or shell command to reassign the split parent region
[ https://issues.apache.org/jira/browse/HBASE-20651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540598#comment-16540598 ] huaxiang sun commented on HBASE-20651: -- Yes, this applies to 1.2, 1.3 and 1.4. Thanks [~apurtell] for taking care of it. > Master, prevents hbck or shell command to reassign the split parent region > -- > > Key: HBASE-20651 > URL: https://issues.apache.org/jira/browse/HBASE-20651 > Project: HBase > Issue Type: Improvement > Components: master >Affects Versions: 1.2.6 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Minor > Fix For: 1.5.0 > > Attachments: HBASE-20651-branch-1-v001.patch, > HBASE-20651-branch-1-v002.patch, HBASE-20651-branch-1-v003.patch > > > We are seeing that hbck brings back split parent region and this causes > region inconsistency. More details will be filled as reproduce is still > ongoing. Might need to do something at hbck or master to prevent this from > happening. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-18477) Umbrella JIRA for HBase Read Replica clusters
[
https://issues.apache.org/jira/browse/HBASE-18477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540590#comment-16540590
]
Hudson commented on HBASE-18477:
Results for branch HBASE-18477
[build #261 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/261/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/261//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/261//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/261//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-18477/261//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> Umbrella JIRA for HBase Read Replica clusters
> -
>
> Key: HBASE-18477
> URL: https://issues.apache.org/jira/browse/HBASE-18477
> Project: HBase
> Issue Type: New Feature
>Reporter: Zach York
>Assignee: Zach York
>Priority: Major
> Attachments: HBase Read-Replica Clusters Scope doc.docx, HBase
> Read-Replica Clusters Scope doc.pdf, HBase Read-Replica Clusters Scope
> doc_v2.docx, HBase Read-Replica Clusters Scope doc_v2.pdf
>
>
> Recently, changes (such as HBASE-17437) have unblocked HBase to run with a
> root directory external to the cluster (such as in Amazon S3). This means
> that the data is stored outside of the cluster and can be accessible after
> the cluster has been terminated. One use case that is often asked about is
> pointing multiple clusters to one root directory (sharing the data) to have
> read resiliency in the case of a cluster failure.
>
> This JIRA is an umbrella JIRA to contain all the tasks necessary to create a
> read-replica HBase cluster that is pointed at the same root directory.
>
> This requires making the Read-Replica cluster Read-Only (no metadata
> operation or data operations).
> Separating the hbase:meta table for each cluster (Otherwise HBase gets
> confused with multiple clusters trying to update the meta table with their ip
> addresses)
> Adding refresh functionality for the meta table to ensure new metadata is
> picked up on the read replica cluster.
> Adding refresh functionality for HFiles for a given table to ensure new data
> is picked up on the read replica cluster.
>
> This can be used with any existing cluster that is backed by an external
> filesystem.
>
> Please note that this feature is still quite manual (with the potential for
> automation later).
>
> More information on this particular feature can be found here:
> https://aws.amazon.com/blogs/big-data/setting-up-read-replica-clusters-with-hbase-on-amazon-s3/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20860) Merged region's RIT state may not be cleaned after master restart
[ https://issues.apache.org/jira/browse/HBASE-20860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540546#comment-16540546 ] stack commented on HBASE-20860: --- +1 Looks great. +1 for branch-2.0 too please (Or just say and I'll backport). Thanks [~allan163] > Merged region's RIT state may not be cleaned after master restart > - > > Key: HBASE-20860 > URL: https://issues.apache.org/jira/browse/HBASE-20860 > Project: HBase > Issue Type: Sub-task >Affects Versions: 3.0.0, 2.1.0, 2.0.1 >Reporter: Allan Yang >Assignee: Allan Yang >Priority: Major > Fix For: 3.0.0, 2.0.2, 2.1.1 > > Attachments: HBASE-20860.branch-2.0.002.patch, > HBASE-20860.branch-2.0.003.patch, HBASE-20860.branch-2.0.004.patch, > HBASE-20860.branch-2.0.005.patch, HBASE-20860.branch-2.0.patch > > > In MergeTableRegionsProcedure, we issue UnassignProcedures to offline regions > to merge. But if we restart master just after MergeTableRegionsProcedure > finished these two UnassignProcedure and before it can delete their meta > entries. The new master will found these two region is CLOSED but no > procedures are attached to them. They will be regard as RIT regions and > nobody will clean the RIT state for them later. > A quick way to resolve this stuck situation in the production env is > restarting master again, since the meta entries are deleted in > MergeTableRegionsProcedure. Here, I offer a fix for this problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20257) hbase-spark should not depend on com.google.code.findbugs.jsr305
[
https://issues.apache.org/jira/browse/HBASE-20257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540547#comment-16540547
]
Hadoop QA commented on HBASE-20257:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
5s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
7s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 1s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m
42s{color} | {color:green} hbase-spark in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20257 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12931202/HBASE-20257.v05.patch
|
| Optional Tests | asflicense javac javadoc unit shadedjars hadoopcheck
xml compile |
| uname | Linux 53b859710d0c 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14
08:53:28 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 0d33caa39a |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_171 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13596/testReport/ |
| Max. process+thread count | 841 (vs. ulimit of 1) |
| modules | C: hbase-spark U: hbase-spark |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/13596/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> hbase-spark should not depend on com.google.code.findbugs.jsr305
>
>
> Key: HBASE-20257
> URL: https://issues.apache.org/jira/browse/HBASE-20257
> Project: HBase
> Issue Type: Task
> Components: build, spark
>Affects Versions: 3.0.0
>Reporter: Ted Yu
>Assignee: Artem Ervits
>Priority: Minor
> Labels: beginner
[jira] [Commented] (HBASE-20846) Table's shared lock is not held by sub-procedures after master restart
[
https://issues.apache.org/jira/browse/HBASE-20846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540528#comment-16540528
]
stack commented on HBASE-20846:
---
Updating procedure store is expensive...takes time.
Do we need a new boolean? What cases do we need to handle on load post-crash?
To my mind, the procedure that holds a lock for the life of the procedure is
the only case that needs re-institution?
When we read the store WALs on startup, we read in order. Can we not just look
for Procedures with holdLock=true and if procedure is not yet finished -- it
has sub-procedures or not -- just re-create their lock before starting the
procedure workers?
> Table's shared lock is not held by sub-procedures after master restart
> --
>
> Key: HBASE-20846
> URL: https://issues.apache.org/jira/browse/HBASE-20846
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.1.0
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Major
> Fix For: 3.0.0, 2.0.2, 2.1.1
>
> Attachments: HBASE-20846.branch-2.0.002.patch,
> HBASE-20846.branch-2.0.patch
>
>
> Found this one when investigating ModifyTableProcedure got stuck while there
> was a MoveRegionProcedure going on after master restart.
> Though this issue can be solved by HBASE-20752. But I discovered something
> else.
> Before a MoveRegionProcedure can execute, it will hold the table's shared
> lock. so,, when a UnassignProcedure was spwaned, it will not check the
> table's shared lock since it is sure that its parent(MoveRegionProcedure) has
> aquired the table's lock.
> {code:java}
> // If there is parent procedure, it would have already taken xlock, so no
> need to take
> // shared lock here. Otherwise, take shared lock.
> if (!procedure.hasParent()
> && waitTableQueueSharedLock(procedure, table) == null) {
> return true;
> }
> {code}
> But, it is not the case when Master was restarted. The child
> procedure(UnassignProcedure) will be executed first after restart. Though it
> has a parent(MoveRegionProcedure), but apprently the parent didn't hold the
> table's lock.
> So, since it began to execute without hold the table's shared lock. A
> ModifyTableProcedure can aquire the table's exclusive lock and execute at the
> same time. Which is not possible if the master was not restarted.
> This will cause a stuck before HBASE-20752. But since HBASE-20752 has fixed,
> I wrote a simple UT to repo this case.
> I think we don't have to check the parent for table's shared lock. It is a
> shared lock, right? I think we can acquire it every time we need it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20175) hbase-spark needs scala dependency convergance
[
https://issues.apache.org/jira/browse/HBASE-20175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540529#comment-16540529
]
Artem Ervits commented on HBASE-20175:
--
ping [~mdrob] [~yuzhih...@gmail.com] updated patch with latest
scala.maven.version
> hbase-spark needs scala dependency convergance
> --
>
> Key: HBASE-20175
> URL: https://issues.apache.org/jira/browse/HBASE-20175
> Project: HBase
> Issue Type: Bug
> Components: dependencies, spark
>Reporter: Mike Drob
>Assignee: Artem Ervits
>Priority: Major
> Attachments: HBASE-20175.v01.patch, HBASE-20175.v04.patch
>
>
> This is a follow-on to HBASE-16179 - I think we might need to specify an
> exclude in the dependency management.
> {noformat}
> [INFO] --- scala-maven-plugin:3.2.0:compile (scala-compile-first) @
> hbase-spark ---
> [WARNING] Expected all dependencies to require Scala version: 2.11.8
> [WARNING] org.apache.hbase:hbase-spark:3.0.0-SNAPSHOT requires scala
> version: 2.11.8
> [WARNING] org.apache.spark:spark-streaming_2.11:2.1.1 requires scala
> version: 2.11.8
> [WARNING] org.apache.spark:spark-streaming_2.11:2.1.1 requires scala
> version: 2.11.8
> [WARNING] org.scalatest:scalatest_2.11:2.2.4 requires scala version: 2.11.2
> {noformat}
> [~tedyu] - since you're already fiddling in this area, do you want to take a
> look?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20175) hbase-spark needs scala dependency convergance
[
https://issues.apache.org/jira/browse/HBASE-20175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Artem Ervits updated HBASE-20175:
-
Attachment: HBASE-20175.v04.patch
> hbase-spark needs scala dependency convergance
> --
>
> Key: HBASE-20175
> URL: https://issues.apache.org/jira/browse/HBASE-20175
> Project: HBase
> Issue Type: Bug
> Components: dependencies, spark
>Reporter: Mike Drob
>Assignee: Artem Ervits
>Priority: Major
> Attachments: HBASE-20175.v01.patch, HBASE-20175.v04.patch
>
>
> This is a follow-on to HBASE-16179 - I think we might need to specify an
> exclude in the dependency management.
> {noformat}
> [INFO] --- scala-maven-plugin:3.2.0:compile (scala-compile-first) @
> hbase-spark ---
> [WARNING] Expected all dependencies to require Scala version: 2.11.8
> [WARNING] org.apache.hbase:hbase-spark:3.0.0-SNAPSHOT requires scala
> version: 2.11.8
> [WARNING] org.apache.spark:spark-streaming_2.11:2.1.1 requires scala
> version: 2.11.8
> [WARNING] org.apache.spark:spark-streaming_2.11:2.1.1 requires scala
> version: 2.11.8
> [WARNING] org.scalatest:scalatest_2.11:2.2.4 requires scala version: 2.11.2
> {noformat}
> [~tedyu] - since you're already fiddling in this area, do you want to take a
> look?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
