[GitHub] [hbase] gkanade opened a new pull request #2134: Cherry pick HBASE-24713 RS startup with FSHLog throws NPE after HBASE-21751
gkanade opened a new pull request #2134: URL: https://github.com/apache/hbase/pull/2134 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] gkanade opened a new pull request #2133: Hbase 24713 branch 2.3
gkanade opened a new pull request #2133: URL: https://github.com/apache/hbase/pull/2133 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] anoopsjohn commented on a change in pull request #2021: HBASE-24665 all wal of RegionGroupingProvider together roll
anoopsjohn commented on a change in pull request #2021:
URL: https://github.com/apache/hbase/pull/2021#discussion_r459857809
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/wal/AbstractWALRoller.java
##
@@ -58,31 +58,31 @@
protected static final String WAL_ROLL_PERIOD_KEY =
"hbase.regionserver.logroll.period";
- protected final ConcurrentMap walNeedsRoll = new
ConcurrentHashMap<>();
+ protected final ConcurrentMap wals = new
ConcurrentHashMap<>();
protected final T abortable;
- private volatile long lastRollTime = System.currentTimeMillis();
// Period to roll log.
private final long rollPeriod;
private final int threadWakeFrequency;
// The interval to check low replication on hlog's pipeline
- private long checkLowReplicationInterval;
+ private final long checkLowReplicationInterval;
private volatile boolean running = true;
public void addWAL(WAL wal) {
// check without lock first
-if (walNeedsRoll.containsKey(wal)) {
+if (wals.containsKey(wal)) {
return;
}
// this is to avoid race between addWAL and requestRollAll.
synchronized (this) {
- if (walNeedsRoll.putIfAbsent(wal, Boolean.FALSE) == null) {
+ if (wals.putIfAbsent(wal, new RollController(wal)) == null) {
wal.registerWALActionsListener(new WALActionsListener() {
@Override
public void logRollRequested(WALActionsListener.RollRequestReason
reason) {
// TODO logs will contend with each other here, replace with e.g.
DelayedQueue
synchronized (AbstractWALRoller.this) {
- walNeedsRoll.put(wal, Boolean.TRUE);
+ RollController controller = wals.computeIfAbsent(wal, rc -> new
RollController(wal));
Review comment:
Good Q. In fact I also thought when reviewed this. Ideally speaking we
should get the addWAL call 1st which will add the instance to the Map. When we
get call here the wal should be in the map already. But if u see the cur impl,
there is no such contract enforcing. It just add the WAL with True value. So
believe while making patch, @WenFeiYi went with similar lines.
We can consider this.. Need to see any chance we get a rollReq before
adding.. While RS start, we do some rollReq on WALs.. This introduced some bug
in the past. We need to see that closely.. If we can confirm that we can
add that contract enforcing and so what u suggested. I would say add a TODO
here and raise another issue. This went through multiple cycles of changes. :-)
U ok Viraj?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HBASE-24749) Direct insert HFiles and Persist in-memory HFile tracking
[
https://issues.apache.org/jira/browse/HBASE-24749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164147#comment-17164147
]
Michael Stack commented on HBASE-24749:
---
Just a thought; keeping the hfile set in hbase:meta is going to up the
read/write load on this table significantly; every flush and compaction will
result in an update inline w/ the flush/compaction completion – if it fails,
the flush/compaction fail? – and every open will be an hbase:meta read to find
set of files to use. Currently Master only writes hbase:meta so RS will tell
Master the compaction result – fine-by-me because master should be running
compactions anyways – or the new flush file added, and Master would update, or
RS writes meta, a violation of a simplification we made trying to ensure
one-writer. Just a note.
> Direct insert HFiles and Persist in-memory HFile tracking
> -
>
> Key: HBASE-24749
> URL: https://issues.apache.org/jira/browse/HBASE-24749
> Project: HBase
> Issue Type: Umbrella
> Components: Compaction, HFile
>Affects Versions: 3.0.0-alpha-1
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Labels: design, discussion, objectstore, storeFile, storeengine
> Attachments: 1B100m-25m25m-performance.pdf, Apache HBase - Direct
> insert HFiles and Persist in-memory HFile tracking.pdf
>
>
> We propose a new feature (a new store engine) to remove the {{.tmp}}
> directory used in the commit stage for common HFile operations such as flush
> and compaction to improve the write throughput and latency on object stores.
> Specifically for S3 filesystems, this will also mitigate read-after-write
> inconsistencies caused by immediate HFiles validation after moving the
> HFile(s) to data directory.
> Please see attached for this proposal and the initial result captured with
> 25m (25m operations) and 1B (100m operations) YCSB workload A LOAD and RUN,
> and workload C RUN result.
> The goal of this JIRA is to discuss with the community if the proposed
> improvement on the object stores use case makes senses and if we miss
> anything should be included.
> Improvement Highlights
> 1. Lower write latency, especially the p99+
> 2. Higher write throughput on flush and compaction
> 3. Lower MTTR on region (re)open or assignment
> 4. Remove consistent check dependencies (e.g. DynamoDB) supported by file
> system implementation
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] ramkrish86 commented on pull request #2125: HBASE-24713 RS startup with FSHLog throws NPE after HBASE-21751
ramkrish86 commented on pull request #2125: URL: https://github.com/apache/hbase/pull/2125#issuecomment-663348076 Think that the test case failures are unrelated. Will commit this in a day or two unless there are any more comments. @anoopsjohn , @saintstack . This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-24754) Bulk load performance is degraded in HBase 2
[ https://issues.apache.org/jira/browse/HBASE-24754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajeet Rai updated HBASE-24754: -- Description: in our Test,It is observed that Bulk load performance is degraded in HBase 2 . Test Input: 1: Table with 500 region 2: data =2 TB 3: Cluster: 7 node(2 master+5 Region Server) 4: No of Container Launched are same in both case HBase 2 took 10% more time then HBase 1.3 where test input is same for both cluster |Feature|HBase 2.2.3 Time(Sec)|HBase 1.3.1 Time(Sec)|Diff%|Snappy lib: | |BulkLoad|21837|19686.16|-10.93|Snappy lib: FI8: 1.4 FI651: 1.4| was: in our Test,It is observed that Bulk load performance is degraded in HBase 2 . Test Input: 1: Table with 500 region 2: data =2 TB 3: Cluster: 7 node(2 master+5 Region Server) HBase 2 took 10% more time then HBase 1.3 where test input is same for both cluster > Bulk load performance is degraded in HBase 2 > - > > Key: HBASE-24754 > URL: https://issues.apache.org/jira/browse/HBASE-24754 > Project: HBase > Issue Type: Bug > Components: Performance >Affects Versions: 2.2.3 >Reporter: Ajeet Rai >Priority: Major > > in our Test,It is observed that Bulk load performance is degraded in HBase 2 . > Test Input: > 1: Table with 500 region > 2: data =2 TB > 3: Cluster: 7 node(2 master+5 Region Server) > 4: No of Container Launched are same in both case > HBase 2 took 10% more time then HBase 1.3 where test input is same for both > cluster > > |Feature|HBase 2.2.3 > Time(Sec)|HBase 1.3.1 > Time(Sec)|Diff%|Snappy lib: > | > |BulkLoad|21837|19686.16|-10.93|Snappy lib: > FI8: 1.4 > FI651: 1.4| -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23634) Enable "Split WAL to HFile" by default
[ https://issues.apache.org/jira/browse/HBASE-23634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164140#comment-17164140 ] ramkrishna.s.vasudevan commented on HBASE-23634: bq.Or else write directly under this but when the split attempt failed, the next one's 1st job will be to clean the existing result HFiles. Ya this is what I meant. > Enable "Split WAL to HFile" by default > -- > > Key: HBASE-23634 > URL: https://issues.apache.org/jira/browse/HBASE-23634 > Project: HBase > Issue Type: Task >Affects Versions: 3.0.0-alpha-1, 2.3.0 >Reporter: Guanghao Zhang >Priority: Blocker > Fix For: 3.0.0-alpha-1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23634) Enable "Split WAL to HFile" by default
[ https://issues.apache.org/jira/browse/HBASE-23634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164139#comment-17164139 ] ramkrishna.s.vasudevan commented on HBASE-23634: bq.. Say if HFile was placed under region/cf/recovered.edits/ dir, Instead on a retry we can clear the files under region/cf/recovered.edits and then start over? that will also ensure that the partially written files are removed correct? > Enable "Split WAL to HFile" by default > -- > > Key: HBASE-23634 > URL: https://issues.apache.org/jira/browse/HBASE-23634 > Project: HBase > Issue Type: Task >Affects Versions: 3.0.0-alpha-1, 2.3.0 >Reporter: Guanghao Zhang >Priority: Blocker > Fix For: 3.0.0-alpha-1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24749) Direct insert HFiles and Persist in-memory HFile tracking
[
https://issues.apache.org/jira/browse/HBASE-24749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164138#comment-17164138
]
ramkrishna.s.vasudevan commented on HBASE-24749:
WAL based approach one advantage is that because of the fact that we anyway
replay the WALs on RS crash and we can easily identify this fact of which were
the files that were flushed/compacted completely.
HBASE-20704 was to remove the files that were actually compacted but not yet
removed by the discharger. This WAL marker is just to ensure on a restart of an
RS and region opening we know the file in that region:cf path came out of a
completed compaction or not. I think atleast in region open case prevoiusly we
were reading that marker but we never validated the compacted file under
region:cf with that .
> Direct insert HFiles and Persist in-memory HFile tracking
> -
>
> Key: HBASE-24749
> URL: https://issues.apache.org/jira/browse/HBASE-24749
> Project: HBase
> Issue Type: Umbrella
> Components: Compaction, HFile
>Affects Versions: 3.0.0-alpha-1
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Labels: design, discussion, objectstore, storeFile, storeengine
> Attachments: 1B100m-25m25m-performance.pdf, Apache HBase - Direct
> insert HFiles and Persist in-memory HFile tracking.pdf
>
>
> We propose a new feature (a new store engine) to remove the {{.tmp}}
> directory used in the commit stage for common HFile operations such as flush
> and compaction to improve the write throughput and latency on object stores.
> Specifically for S3 filesystems, this will also mitigate read-after-write
> inconsistencies caused by immediate HFiles validation after moving the
> HFile(s) to data directory.
> Please see attached for this proposal and the initial result captured with
> 25m (25m operations) and 1B (100m operations) YCSB workload A LOAD and RUN,
> and workload C RUN result.
> The goal of this JIRA is to discuss with the community if the proposed
> improvement on the object stores use case makes senses and if we miss
> anything should be included.
> Improvement Highlights
> 1. Lower write latency, especially the p99+
> 2. Higher write throughput on flush and compaction
> 3. Lower MTTR on region (re)open or assignment
> 4. Remove consistent check dependencies (e.g. DynamoDB) supported by file
> system implementation
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24749) Direct insert HFiles and Persist in-memory HFile tracking
[
https://issues.apache.org/jira/browse/HBASE-24749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164129#comment-17164129
]
Anoop Sam John commented on HBASE-24749:
So the diff is this. In case of HFiles list in META or a system table, that
info is abt the valid files and always that has to be examined (when new
cluster is created or cluster is restarted). In WAL based what we rely on the
status of not committed files alone. By default all files are committed files.
Only those which are on going will have markers in the WAL. So that is the
basic difference.
> Direct insert HFiles and Persist in-memory HFile tracking
> -
>
> Key: HBASE-24749
> URL: https://issues.apache.org/jira/browse/HBASE-24749
> Project: HBase
> Issue Type: Umbrella
> Components: Compaction, HFile
>Affects Versions: 3.0.0-alpha-1
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Labels: design, discussion, objectstore, storeFile, storeengine
> Attachments: 1B100m-25m25m-performance.pdf, Apache HBase - Direct
> insert HFiles and Persist in-memory HFile tracking.pdf
>
>
> We propose a new feature (a new store engine) to remove the {{.tmp}}
> directory used in the commit stage for common HFile operations such as flush
> and compaction to improve the write throughput and latency on object stores.
> Specifically for S3 filesystems, this will also mitigate read-after-write
> inconsistencies caused by immediate HFiles validation after moving the
> HFile(s) to data directory.
> Please see attached for this proposal and the initial result captured with
> 25m (25m operations) and 1B (100m operations) YCSB workload A LOAD and RUN,
> and workload C RUN result.
> The goal of this JIRA is to discuss with the community if the proposed
> improvement on the object stores use case makes senses and if we miss
> anything should be included.
> Improvement Highlights
> 1. Lower write latency, especially the p99+
> 2. Higher write throughput on flush and compaction
> 3. Lower MTTR on region (re)open or assignment
> 4. Remove consistent check dependencies (e.g. DynamoDB) supported by file
> system implementation
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24766) Document Remote Procedure Execution
[ https://issues.apache.org/jira/browse/HBASE-24766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164130#comment-17164130 ] Michael Stack commented on HBASE-24766: --- This is a duplicate of HBASE-24596 > Document Remote Procedure Execution > --- > > Key: HBASE-24766 > URL: https://issues.apache.org/jira/browse/HBASE-24766 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: Michael Stack >Priority: Major > > The doc in ServerRemoteProcedure, the main class is good. I said I'd give it > a tuneup now we've a few impls of this feature. I want to doc in the code > rather than in refguide since it devs who would use it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24596) Document general framework to execute remote procedure on RegionServer
[ https://issues.apache.org/jira/browse/HBASE-24596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack resolved HBASE-24596. --- Resolution: Duplicate Resolving as duplicate of HBASE-24596 (I thought I'd filed one already...) > Document general framework to execute remote procedure on RegionServer > -- > > Key: HBASE-24596 > URL: https://issues.apache.org/jira/browse/HBASE-24596 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: Michael Stack >Priority: Major > > Document -- probably best in javadoc -- the system added by HBASE-19216 > 'Implement a general framework to execute remote procedure on RS ' for having > procedures run tasks on remote regionservers. The system evolved a good while > ago in HBASE-19216 to manage replication peers. It was then adopted by the > new procedure-based distributed WAL splitting feature with some worry that > the latter didn't use the framework properly (so far, all looks good). > This issue is about vetting the distributed WAL splitters usage and doc'ing > the framework as I go. This framework is useful. Others will want to exploit > it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24749) Direct insert HFiles and Persist in-memory HFile tracking
[
https://issues.apache.org/jira/browse/HBASE-24749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164126#comment-17164126
]
Anoop Sam John commented on HBASE-24749:
Thought abt that also.. In case of a cluster drop and later recreate based on
the cluster FS, we wont have any WALs. So no wal replay. That means
automatically all the files are valid. All files will come as valid HFiles for
that region:cf.
In case of RS crash only the WAL replay come into pic. We split all WAL files
and replay and once replay also over, the region will come online in next RS.
So during this replay, if we do the tracking of the files also, end of that we
can find uncommitted files and throw them away.
In case when we have to store the HFiles list in META, every write op
(flush/compaction) from every RS is depending on the META region and a write to
that. If the META region is not available for some time, all the
flushes/compaction is blocked from completion. That is one worry I am having.
And then the next is how to store the META table's file list itself. In case
of cluster recreate, the zk data also lost right [~zyork]? So that also says
clearly that storing at zk is not possible.
> Direct insert HFiles and Persist in-memory HFile tracking
> -
>
> Key: HBASE-24749
> URL: https://issues.apache.org/jira/browse/HBASE-24749
> Project: HBase
> Issue Type: Umbrella
> Components: Compaction, HFile
>Affects Versions: 3.0.0-alpha-1
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Labels: design, discussion, objectstore, storeFile, storeengine
> Attachments: 1B100m-25m25m-performance.pdf, Apache HBase - Direct
> insert HFiles and Persist in-memory HFile tracking.pdf
>
>
> We propose a new feature (a new store engine) to remove the {{.tmp}}
> directory used in the commit stage for common HFile operations such as flush
> and compaction to improve the write throughput and latency on object stores.
> Specifically for S3 filesystems, this will also mitigate read-after-write
> inconsistencies caused by immediate HFiles validation after moving the
> HFile(s) to data directory.
> Please see attached for this proposal and the initial result captured with
> 25m (25m operations) and 1B (100m operations) YCSB workload A LOAD and RUN,
> and workload C RUN result.
> The goal of this JIRA is to discuss with the community if the proposed
> improvement on the object stores use case makes senses and if we miss
> anything should be included.
> Improvement Highlights
> 1. Lower write latency, especially the p99+
> 2. Higher write throughput on flush and compaction
> 3. Lower MTTR on region (re)open or assignment
> 4. Remove consistent check dependencies (e.g. DynamoDB) supported by file
> system implementation
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164124#comment-17164124 ] Reid Chan commented on HBASE-20312: --- I just push a new feature branch named "ccsmap", based on branch-1, where I'm going to push the first commit which focus only on the data structure (what HBASE-20717 intends) > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: Chance Li >Priority: Major > Fix For: 3.0.0-alpha-1 > > Attachments: 1.1.2-ccsmap-number.png, HBASE-20312-1.3.2.patch, > HBASE-20312-master.v1.patch, HBASE-20312-master.v2.patch, > HBASE-20312-master.v3.patch, HBASE-20312-master.v4.patch, > HBASE-20312-master.v5.patch, HBASE-20312-master.v6.patch, > HBASE-20312-master.v7.patch, HBASE-20312-master.v8.patch, > HBASE-20312-master.v9.patch, ccsmap-branch-1.1.patch, hits.png, jira1.png, > jira2.png, jira3.png, off-heap-test-put-master.png, > on-heap-test-put-master.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% > better read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2130: HBASE-24765: Dynamic master discovery
Apache-HBase commented on pull request #2130: URL: https://github.com/apache/hbase/pull/2130#issuecomment-663337641 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 14s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 20s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 55s | master passed | | +1 :green_heart: | compile | 2m 8s | master passed | | +1 :green_heart: | shadedjars | 6m 20s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 15s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 1s | the patch passed | | +1 :green_heart: | compile | 2m 21s | the patch passed | | +1 :green_heart: | javac | 2m 21s | the patch passed | | +1 :green_heart: | shadedjars | 6m 12s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 11s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 48s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 1m 16s | hbase-client in the patch passed. | | +1 :green_heart: | unit | 207m 50s | hbase-server in the patch passed. | | | | 241m 15s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2130/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2130 | | JIRA Issue | HBASE-24765 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux a40a8f1ed21b 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / e7963f6486 | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2130/2/testReport/ | | Max. process+thread count | 4005 (vs. ulimit of 12500) | | modules | C: hbase-protocol-shaded hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2130/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-24665) MultiWAL : Avoid rolling of ALL WALs when one of the WAL needs a roll
[ https://issues.apache.org/jira/browse/HBASE-24665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wenfeiyi666 updated HBASE-24665: Component/s: wal > MultiWAL : Avoid rolling of ALL WALs when one of the WAL needs a roll > -- > > Key: HBASE-24665 > URL: https://issues.apache.org/jira/browse/HBASE-24665 > Project: HBase > Issue Type: Bug > Components: wal >Affects Versions: 2.3.0, master, 2.1.10, 1.4.14, 2.2.6 >Reporter: wenfeiyi666 >Assignee: wenfeiyi666 >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.1, 2.1.10, 1.4.14, 2.2.7 > > > when use multiwal, any a wal request roll, all wal will be together roll. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23834) HBase fails to run on Hadoop 3.3.0/3.2.2/3.1.4 due to jetty version mismatch
[ https://issues.apache.org/jira/browse/HBASE-23834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-23834: -- Component/s: dependencies > HBase fails to run on Hadoop 3.3.0/3.2.2/3.1.4 due to jetty version mismatch > > > Key: HBASE-23834 > URL: https://issues.apache.org/jira/browse/HBASE-23834 > Project: HBase > Issue Type: Bug > Components: dependencies >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > > HBase master branch is currently on Jetty 9.3, and latest Hadoop 3 > (unreleased branches trunk, branch-3.2 and branch-3.1) bumped Jetty to 9.4 to > address a vulnerability CVE-2017-9735. > (1) Jetty 9.3 and 9.4 are quite different (there are incompatible API > changes) and HBase won't start on the latest Hadoop 3. > (2) In any case, HBase should update its Jetty dependency to address the > vulnerability. > Fortunately for HBase, updating to Jetty 9.4 requires no code change other > than the maven version string. > More tests are needed to verify if HBase can run on older Hadoop versions if > its Jetty is updated. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24762) Purge protobuf java 2.5.0 dependency
[ https://issues.apache.org/jira/browse/HBASE-24762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-24762. --- Hadoop Flags: Reviewed Resolution: Fixed Merged to master. On branch-2, we still have hbase-protocol which depends on protobuf 2.5.0 so whether to apply this patch is not very important. > Purge protobuf java 2.5.0 dependency > > > Key: HBASE-24762 > URL: https://issues.apache.org/jira/browse/HBASE-24762 > Project: HBase > Issue Type: Sub-task > Components: dependencies, Protobufs >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-1 > > > On master branch, we have removed the hbase-protocol module so in general, we > do not need to depend on protobuf 2.5.0 directl. Especially for hadoop 3.3.0, > hadoop will not depend on 2.5.0 any more, we should make sure hbase do not > introduce protobuf 2.5.0 too. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache9 merged pull request #2128: HBASE-24762 Purge protobuf java 2.5.0 dependency
Apache9 merged pull request #2128: URL: https://github.com/apache/hbase/pull/2128 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (HBASE-24749) Direct insert HFiles and Persist in-memory HFile tracking
[
https://issues.apache.org/jira/browse/HBASE-24749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164094#comment-17164094
]
Zach York edited comment on HBASE-24749 at 7/24/20, 3:37 AM:
-
Yes, I think that is potentially an alternative implementation that could work.
One downside I could see is you would still want to be able to handle bulk
loading/other procedures. If all updates to the state are controlled by the RS,
this approach would work. I wonder what the perf difference might be... since
in this case you would have to replay edits always.
Edit: After thinking it through a bit, the WAL approach has one problem in our
environment (where we expect the HDFS WALs will not be migrated to a new
cluster). Storing the data in a table is more durable for our use case, but the
WAL implementation could be suitable for the ROOT table where it matters less
if the file list needs to fall back to FS listing/validation.
was (Author: zyork):
Yes, I think that is potentially an alternative implementation that could work.
One downside I could see is you would still want to be able to handle bulk
loading/other procedures. If all updates to the state are controlled by the RS,
this approach would work. I wonder what the perf difference might be... since
in this case you would have to replay edits always.
> Direct insert HFiles and Persist in-memory HFile tracking
> -
>
> Key: HBASE-24749
> URL: https://issues.apache.org/jira/browse/HBASE-24749
> Project: HBase
> Issue Type: Umbrella
> Components: Compaction, HFile
>Affects Versions: 3.0.0-alpha-1
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Labels: design, discussion, objectstore, storeFile, storeengine
> Attachments: 1B100m-25m25m-performance.pdf, Apache HBase - Direct
> insert HFiles and Persist in-memory HFile tracking.pdf
>
>
> We propose a new feature (a new store engine) to remove the {{.tmp}}
> directory used in the commit stage for common HFile operations such as flush
> and compaction to improve the write throughput and latency on object stores.
> Specifically for S3 filesystems, this will also mitigate read-after-write
> inconsistencies caused by immediate HFiles validation after moving the
> HFile(s) to data directory.
> Please see attached for this proposal and the initial result captured with
> 25m (25m operations) and 1B (100m operations) YCSB workload A LOAD and RUN,
> and workload C RUN result.
> The goal of this JIRA is to discuss with the community if the proposed
> improvement on the object stores use case makes senses and if we miss
> anything should be included.
> Improvement Highlights
> 1. Lower write latency, especially the p99+
> 2. Higher write throughput on flush and compaction
> 3. Lower MTTR on region (re)open or assignment
> 4. Remove consistent check dependencies (e.g. DynamoDB) supported by file
> system implementation
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2131: HBASE-24766 Document Remote Procedure Execution
Apache-HBase commented on pull request #2131: URL: https://github.com/apache/hbase/pull/2131#issuecomment-663328941 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 13m 25s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 22s | branch-2 passed | | +1 :green_heart: | compile | 1m 17s | branch-2 passed | | +1 :green_heart: | shadedjars | 6m 11s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 48s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 6s | the patch passed | | +1 :green_heart: | compile | 1m 20s | the patch passed | | +1 :green_heart: | javac | 1m 20s | the patch passed | | +1 :green_heart: | shadedjars | 6m 44s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 48s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 196m 19s | hbase-server in the patch passed. | | | | 239m 23s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2131/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2131 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 002316908f2b 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / ce4e692699 | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2131/1/testReport/ | | Max. process+thread count | 2852 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2131/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2132: HBASE-24767 Change default to false for HBASE-15519 per-user metrics
Apache-HBase commented on pull request #2132: URL: https://github.com/apache/hbase/pull/2132#issuecomment-663328453 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 39s | Docker mode activated. | | -0 :warning: | yetus | 0m 7s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 46s | branch-2 passed | | +1 :green_heart: | compile | 0m 53s | branch-2 passed | | +1 :green_heart: | shadedjars | 4m 57s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 20s | the patch passed | | +1 :green_heart: | compile | 0m 55s | the patch passed | | +1 :green_heart: | javac | 0m 55s | the patch passed | | +1 :green_heart: | shadedjars | 5m 6s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 34s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 153m 53s | hbase-server in the patch failed. | | | | 177m 4s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2132/1/artifact/yetus-jdk8-hadoop2-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2132 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 8913a5dab25d 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 70ab0dc324 | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2132/1/artifact/yetus-jdk8-hadoop2-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2132/1/testReport/ | | Max. process+thread count | 3296 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2132/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-19256) [hbase-thirdparty] shade jetty
[ https://issues.apache.org/jira/browse/HBASE-19256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164103#comment-17164103 ] Duo Zhang commented on HBASE-19256: --- The patch is ready, mind taking a look [~weichiu]? Thanks. > [hbase-thirdparty] shade jetty > -- > > Key: HBASE-19256 > URL: https://issues.apache.org/jira/browse/HBASE-19256 > Project: HBase > Issue Type: Task > Components: dependencies, thirdparty >Reporter: Mike Drob >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache9 commented on pull request #2113: HBASE-24286: HMaster won't become healthy after after cloning or crea…
Apache9 commented on pull request #2113: URL: https://github.com/apache/hbase/pull/2113#issuecomment-663328123 In general, if you touch the internal of HBase directly, it may lead to data loss, unexpected behavior, etc. As I said above, the current design is to compare the WAL directories and the region server znodes on zookeeper to detect dead region servers when master starts up. If you just removed the WAL directories then HBase will have unexpected behaviors. Any addendums to solve the problem here should be considered as dangerous operations, which should only be in HBCK. If you want to solve the problem automatically, you should find another way to detect the dead region servers when master starts up, to make sure we do not rely on the WAL directories. But I'm still a bit nervous that when SCP notices that there is no WAL directory for a dead region server, what should it do. It is not the expected behavior in HBase... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2132: HBASE-24767 Change default to false for HBASE-15519 per-user metrics
Apache-HBase commented on pull request #2132: URL: https://github.com/apache/hbase/pull/2132#issuecomment-663327829 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 39s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 27s | branch-2 passed | | +1 :green_heart: | compile | 1m 7s | branch-2 passed | | +1 :green_heart: | shadedjars | 5m 58s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 41s | hbase-server in branch-2 failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 5s | the patch passed | | +1 :green_heart: | compile | 1m 4s | the patch passed | | +1 :green_heart: | javac | 1m 4s | the patch passed | | +1 :green_heart: | shadedjars | 7m 5s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 50s | hbase-server in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 145m 13s | hbase-server in the patch failed. | | | | 173m 31s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2132/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2132 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 31acd52d0b43 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 70ab0dc324 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2132/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2132/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2132/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2132/1/testReport/ | | Max. process+thread count | 3573 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2132/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache9 commented on pull request #2113: HBASE-24286: HMaster won't become healthy after after cloning or crea…
Apache9 commented on pull request #2113: URL: https://github.com/apache/hbase/pull/2113#issuecomment-663325543 > In this scenario, regardless of what we do, there will be dataloss unless the correct WAL directory is (again) specified. In fact, I don't believe you can change WAL dir without restarting servers (I also don't think it works with rolling restart). I don't think this is a valid scenario for this issue. No, there will be silent data loss, the user will notice that no region is online, and the cluster is not in a good state, just the same as what you described here. And again, this is not a normal operation in HBase, we do not expect that the WAL directories can be removed without SCP. I wonder why our SCP can even pass without a WAL directory. We should hang there I suppose. Only HBCKSCP can do the dangerous operation. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2131: HBASE-24766 Document Remote Procedure Execution
Apache-HBase commented on pull request #2131: URL: https://github.com/apache/hbase/pull/2131#issuecomment-663324964 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 15s | Docker mode activated. | | -0 :warning: | yetus | 0m 6s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 45s | branch-2 passed | | +1 :green_heart: | compile | 1m 12s | branch-2 passed | | +1 :green_heart: | shadedjars | 6m 27s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 42s | hbase-server in branch-2 failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 29s | the patch passed | | +1 :green_heart: | compile | 1m 9s | the patch passed | | +1 :green_heart: | javac | 1m 9s | the patch passed | | +1 :green_heart: | shadedjars | 6m 24s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 40s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 188m 9s | hbase-server in the patch passed. | | | | 217m 9s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2131/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2131 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 18ef0cd3812c 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / ce4e692699 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2131/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2131/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2131/1/testReport/ | | Max. process+thread count | 2785 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2131/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2130: HBASE-24765: Dynamic master discovery
Apache-HBase commented on pull request #2130: URL: https://github.com/apache/hbase/pull/2130#issuecomment-663324629 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 33s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 9s | master passed | | +1 :green_heart: | compile | 2m 30s | master passed | | +1 :green_heart: | shadedjars | 5m 42s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 26s | hbase-client in master failed. | | -0 :warning: | javadoc | 0m 39s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 16s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 56s | the patch passed | | +1 :green_heart: | compile | 2m 31s | the patch passed | | +1 :green_heart: | javac | 2m 31s | the patch passed | | +1 :green_heart: | shadedjars | 5m 46s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 25s | hbase-client in the patch failed. | | -0 :warning: | javadoc | 0m 39s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 57s | hbase-protocol-shaded in the patch passed. | | +1 :green_heart: | unit | 1m 11s | hbase-client in the patch passed. | | -1 :x: | unit | 133m 26s | hbase-server in the patch failed. | | | | 166m 21s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2130/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2130 | | JIRA Issue | HBASE-24765 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux a15a5146bc82 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / e7963f6486 | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2130/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2130/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2130/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2130/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2130/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2130/2/testReport/ | | Max. process+thread count | 4758 (vs. ulimit of 12500) | | modules | C: hbase-protocol-shaded hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2130/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24749) Direct insert HFiles and Persist in-memory HFile tracking
[
https://issues.apache.org/jira/browse/HBASE-24749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164094#comment-17164094
]
Zach York commented on HBASE-24749:
---
Yes, I think that is potentially an alternative implementation that could work.
One downside I could see is you would still want to be able to handle bulk
loading/other procedures. If all updates to the state are controlled by the RS,
this approach would work. I wonder what the perf difference might be... since
in this case you would have to replay edits always.
> Direct insert HFiles and Persist in-memory HFile tracking
> -
>
> Key: HBASE-24749
> URL: https://issues.apache.org/jira/browse/HBASE-24749
> Project: HBase
> Issue Type: Umbrella
> Components: Compaction, HFile
>Affects Versions: 3.0.0-alpha-1
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Labels: design, discussion, objectstore, storeFile, storeengine
> Attachments: 1B100m-25m25m-performance.pdf, Apache HBase - Direct
> insert HFiles and Persist in-memory HFile tracking.pdf
>
>
> We propose a new feature (a new store engine) to remove the {{.tmp}}
> directory used in the commit stage for common HFile operations such as flush
> and compaction to improve the write throughput and latency on object stores.
> Specifically for S3 filesystems, this will also mitigate read-after-write
> inconsistencies caused by immediate HFiles validation after moving the
> HFile(s) to data directory.
> Please see attached for this proposal and the initial result captured with
> 25m (25m operations) and 1B (100m operations) YCSB workload A LOAD and RUN,
> and workload C RUN result.
> The goal of this JIRA is to discuss with the community if the proposed
> improvement on the object stores use case makes senses and if we miss
> anything should be included.
> Improvement Highlights
> 1. Lower write latency, especially the p99+
> 2. Higher write throughput on flush and compaction
> 3. Lower MTTR on region (re)open or assignment
> 4. Remove consistent check dependencies (e.g. DynamoDB) supported by file
> system implementation
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-19256) [hbase-thirdparty] shade jetty
[ https://issues.apache.org/jira/browse/HBASE-19256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164093#comment-17164093 ] Wei-Chiu Chuang commented on HBASE-19256: - Go ahead. Thanks for taking it up. > [hbase-thirdparty] shade jetty > -- > > Key: HBASE-19256 > URL: https://issues.apache.org/jira/browse/HBASE-19256 > Project: HBase > Issue Type: Task > Components: dependencies, thirdparty >Reporter: Mike Drob >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-24749) Direct insert HFiles and Persist in-memory HFile tracking
[
https://issues.apache.org/jira/browse/HBASE-24749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164090#comment-17164090
]
Anoop Sam John edited comment on HBASE-24749 at 7/24/20, 2:33 AM:
--
bq. Can you expand on how we can get in a situation where a partial file is
written? I'm trying to see if there are any failure modes we haven't though of.
If the case is a complete file written to the data directory, is there harm in
picking up the new file (even if it hasn't successfully committed to the SFM)?
That point was based on another direction what Stack was saying. Am not sure
whether Stack suggested that for META table alone or for all. ie use the WAL
event markers to know whether a HFile is committed or not. If we see, during
WAL replay that there is a flush begin marker and later flush complete marker,
this means it is a committed file. If no markers at all for a file, this is
an old existing file. If only begin but no end means this is not a committed
file and so while region reopen, we can ignore this. Same with compaction
also. There one issue was what if the WAL file which is having the begin
marker got rolled and deleted. We lost the track. But if that can be
controlled, this is also a direction no? (Dedicated wal for these event
markers) We can avoid the need to store all the files list into META and avoid
the Q of how to handled the META's file list. Storing in zk is not a direction.
was (Author: anoop.hbase):
bq. Can you expand on how we can get in a situation where a partial file is
written? I'm trying to see if there are any failure modes we haven't though of.
If the case is a complete file written to the data directory, is there harm in
picking up the new file (even if it hasn't successfully committed to the SFM)?
That point was based on another direction what Stack was saying. Am not sure
whether Stack suggested that for META table alone or for all. ie use the WAL
event markers to know whether a HFile is committer or not. If we see, during
WAL replay that there is a flush begin marker and later flush complete marker,
this means it is a committed file. If no markers at all for a file, this is
an old existing file. If only begin but no end means this is not a committed
file and so while region reopen, we can ignore this. Same with compaction
also. There one issue was what if the WAL file which is having the begin
marker got rolled and deleted. We lost the track. But if that can be
controlled, this is also a direction no? We can avoid the need to store all
the files list into META and avoid the Q of how to handled the META's file
list. Storing in zk is not a direction.
> Direct insert HFiles and Persist in-memory HFile tracking
> -
>
> Key: HBASE-24749
> URL: https://issues.apache.org/jira/browse/HBASE-24749
> Project: HBase
> Issue Type: Umbrella
> Components: Compaction, HFile
>Affects Versions: 3.0.0-alpha-1
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Labels: design, discussion, objectstore, storeFile, storeengine
> Attachments: 1B100m-25m25m-performance.pdf, Apache HBase - Direct
> insert HFiles and Persist in-memory HFile tracking.pdf
>
>
> We propose a new feature (a new store engine) to remove the {{.tmp}}
> directory used in the commit stage for common HFile operations such as flush
> and compaction to improve the write throughput and latency on object stores.
> Specifically for S3 filesystems, this will also mitigate read-after-write
> inconsistencies caused by immediate HFiles validation after moving the
> HFile(s) to data directory.
> Please see attached for this proposal and the initial result captured with
> 25m (25m operations) and 1B (100m operations) YCSB workload A LOAD and RUN,
> and workload C RUN result.
> The goal of this JIRA is to discuss with the community if the proposed
> improvement on the object stores use case makes senses and if we miss
> anything should be included.
> Improvement Highlights
> 1. Lower write latency, especially the p99+
> 2. Higher write throughput on flush and compaction
> 3. Lower MTTR on region (re)open or assignment
> 4. Remove consistent check dependencies (e.g. DynamoDB) supported by file
> system implementation
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24749) Direct insert HFiles and Persist in-memory HFile tracking
[
https://issues.apache.org/jira/browse/HBASE-24749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164090#comment-17164090
]
Anoop Sam John commented on HBASE-24749:
bq. Can you expand on how we can get in a situation where a partial file is
written? I'm trying to see if there are any failure modes we haven't though of.
If the case is a complete file written to the data directory, is there harm in
picking up the new file (even if it hasn't successfully committed to the SFM)?
That point was based on another direction what Stack was saying. Am not sure
whether Stack suggested that for META table alone or for all. ie use the WAL
event markers to know whether a HFile is committer or not. If we see, during
WAL replay that there is a flush begin marker and later flush complete marker,
this means it is a committed file. If no markers at all for a file, this is
an old existing file. If only begin but no end means this is not a committed
file and so while region reopen, we can ignore this. Same with compaction
also. There one issue was what if the WAL file which is having the begin
marker got rolled and deleted. We lost the track. But if that can be
controlled, this is also a direction no? We can avoid the need to store all
the files list into META and avoid the Q of how to handled the META's file
list. Storing in zk is not a direction.
> Direct insert HFiles and Persist in-memory HFile tracking
> -
>
> Key: HBASE-24749
> URL: https://issues.apache.org/jira/browse/HBASE-24749
> Project: HBase
> Issue Type: Umbrella
> Components: Compaction, HFile
>Affects Versions: 3.0.0-alpha-1
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Labels: design, discussion, objectstore, storeFile, storeengine
> Attachments: 1B100m-25m25m-performance.pdf, Apache HBase - Direct
> insert HFiles and Persist in-memory HFile tracking.pdf
>
>
> We propose a new feature (a new store engine) to remove the {{.tmp}}
> directory used in the commit stage for common HFile operations such as flush
> and compaction to improve the write throughput and latency on object stores.
> Specifically for S3 filesystems, this will also mitigate read-after-write
> inconsistencies caused by immediate HFiles validation after moving the
> HFile(s) to data directory.
> Please see attached for this proposal and the initial result captured with
> 25m (25m operations) and 1B (100m operations) YCSB workload A LOAD and RUN,
> and workload C RUN result.
> The goal of this JIRA is to discuss with the community if the proposed
> improvement on the object stores use case makes senses and if we miss
> anything should be included.
> Improvement Highlights
> 1. Lower write latency, especially the p99+
> 2. Higher write throughput on flush and compaction
> 3. Lower MTTR on region (re)open or assignment
> 4. Remove consistent check dependencies (e.g. DynamoDB) supported by file
> system implementation
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23634) Enable "Split WAL to HFile" by default
[ https://issues.apache.org/jira/browse/HBASE-23634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164085#comment-17164085 ] Anoop Sam John commented on HBASE-23634: bq.So it is ok to have corruption file in the process of split wal? Dont know whether call it corrupted file. But in this case, the HFile under recovered edits can be a partially written one. Any time a valid file can get corrupted. That is issue at storage. Now the prob is we are not able to distinguish these 2 cases. So the aim is in cases of wal split also we need a commit kind of mechanism for the files created under recovered.edits. Or else write directly under this but when the split attempt failed, the next one's 1st job will be to clean the existing result HFiles. bq.IIRC recovered.edits system does the right thing. Ya. > Enable "Split WAL to HFile" by default > -- > > Key: HBASE-23634 > URL: https://issues.apache.org/jira/browse/HBASE-23634 > Project: HBase > Issue Type: Task >Affects Versions: 3.0.0-alpha-1, 2.3.0 >Reporter: Guanghao Zhang >Priority: Blocker > Fix For: 3.0.0-alpha-1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-19256) [hbase-thirdparty] shade jetty
[ https://issues.apache.org/jira/browse/HBASE-19256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reassigned HBASE-19256: - Assignee: Duo Zhang (was: Wei-Chiu Chuang) > [hbase-thirdparty] shade jetty > -- > > Key: HBASE-19256 > URL: https://issues.apache.org/jira/browse/HBASE-19256 > Project: HBase > Issue Type: Task > Components: dependencies, thirdparty >Reporter: Mike Drob >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24768) Clear cached service kerberos ticket in case of SASL failures thrown from server side
[ https://issues.apache.org/jira/browse/HBASE-24768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164083#comment-17164083 ] Sandeep Guggilam commented on HBASE-24768: -- FYI [~apurtell] [~abhishek.chouhan] > Clear cached service kerberos ticket in case of SASL failures thrown from > server side > - > > Key: HBASE-24768 > URL: https://issues.apache.org/jira/browse/HBASE-24768 > Project: HBase > Issue Type: Bug >Reporter: Sandeep Guggilam >Assignee: Sandeep Guggilam >Priority: Major > > We setup a SASL connection using different mechanisms like Digest, Kerberos > from master to RS for various activities like region assignment etc. In case > of SASL connect failures, we try to dispose of the SaslRpcClient and try to > relogin from the keytab on the client side. However the relogin from keytab > method doesn't clear off the service ticket cached in memory unless TGT is > about to expire within a timeframe. > This actually causes an issue where there is a keytab refresh that happens > because of expiry on the RS server and throws a SASL connect error when > Master reaches out to the RS server with the cached service ticket that no > longer works with the new refreshed keytab. We might need to clear off the > service ticket cached as there could be a credential refresh on the RS server > side when handling connect failures -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24768) Clear cached service kerberos ticket in case of SASL failures thrown from server side
[ https://issues.apache.org/jira/browse/HBASE-24768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Guggilam updated HBASE-24768: - Summary: Clear cached service kerberos ticket in case of SASL failures thrown from server side (was: Clear service kerberos ticket in case of SASL failures from server side) > Clear cached service kerberos ticket in case of SASL failures thrown from > server side > - > > Key: HBASE-24768 > URL: https://issues.apache.org/jira/browse/HBASE-24768 > Project: HBase > Issue Type: Bug >Reporter: Sandeep Guggilam >Assignee: Sandeep Guggilam >Priority: Major > > We setup a SASL connection using different mechanisms like Digest, Kerberos > from master to RS for various activities like region assignment etc. In case > of SASL connect failures, we try to dispose of the SaslRpcClient and try to > relogin from the keytab on the client side. However the relogin from keytab > method doesn't clear off the service ticket cached in memory unless TGT is > about to expire within a timeframe. > This actually causes an issue where there is a keytab refresh that happens > because of expiry on the RS server and throws a SASL connect error when > Master reaches out to the RS server with the cached service ticket that no > longer works with the new refreshed keytab. We might need to clear off the > service ticket cached as there could be a credential refresh on the RS server > side when handling connect failures -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24768) Clear service kerberos ticket in case of SASL failures from server side
Sandeep Guggilam created HBASE-24768: Summary: Clear service kerberos ticket in case of SASL failures from server side Key: HBASE-24768 URL: https://issues.apache.org/jira/browse/HBASE-24768 Project: HBase Issue Type: Bug Reporter: Sandeep Guggilam Assignee: Sandeep Guggilam We setup a SASL connection using different mechanisms like Digest, Kerberos from master to RS for various activities like region assignment etc. In case of SASL connect failures, we try to dispose of the SaslRpcClient and try to relogin from the keytab on the client side. However the relogin from keytab method doesn't clear off the service ticket cached in memory unless TGT is about to expire within a timeframe. This actually causes an issue where there is a keytab refresh that happens because of expiry on the RS server and throws a SASL connect error when Master reaches out to the RS server with the cached service ticket that no longer works with the new refreshed keytab. We might need to clear off the service ticket cached as there could be a credential refresh on the RS server side when handling connect failures -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2132: HBASE-24767 Change default to false for HBASE-15519 per-user metrics
Apache-HBase commented on pull request #2132: URL: https://github.com/apache/hbase/pull/2132#issuecomment-663302313 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 31s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 15s | branch-2 passed | | +1 :green_heart: | checkstyle | 1m 17s | branch-2 passed | | +1 :green_heart: | spotbugs | 2m 16s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 50s | the patch passed | | +1 :green_heart: | checkstyle | 1m 11s | the patch passed | | +1 :green_heart: | whitespace | 0m 1s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 13m 50s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 31s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 18s | The patch does not generate ASF License warnings. | | | | 39m 58s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2132/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2132 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 1cc7a9178422 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / 70ab0dc324 | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2132/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2130: HBASE-24765: Dynamic master discovery
Apache-HBase commented on pull request #2130: URL: https://github.com/apache/hbase/pull/2130#issuecomment-663301795 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 25s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | | +0 :ok: | prototool | 0m 1s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 24s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 56s | master passed | | +1 :green_heart: | checkstyle | 1m 52s | master passed | | +1 :green_heart: | spotbugs | 6m 33s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 12s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 42s | the patch passed | | -0 :warning: | checkstyle | 0m 26s | hbase-client: The patch generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 12m 0s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | hbaseprotoc | 2m 0s | the patch passed | | +1 :green_heart: | spotbugs | 7m 5s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. | | | | 48m 29s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2130/2/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2130 | | JIRA Issue | HBASE-24765 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle cc hbaseprotoc prototool | | uname | Linux c0dacdaedae8 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / e7963f6486 | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2130/2/artifact/yetus-general-check/output/diff-checkstyle-hbase-client.txt | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-protocol-shaded hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2130/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164059#comment-17164059 ] Zach York commented on HBASE-11288: --- Since (as Duo pointed out) this is relevant to the work to store File locations in meta, I will get up to speed on the discussion and current status. I won't impede the process, but will be happy to help with reviews/discussions. > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23634) Enable "Split WAL to HFile" by default
[ https://issues.apache.org/jira/browse/HBASE-23634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164054#comment-17164054 ] Michael Stack commented on HBASE-23634: --- IIRC recovered.edits system does the right thing. > Enable "Split WAL to HFile" by default > -- > > Key: HBASE-23634 > URL: https://issues.apache.org/jira/browse/HBASE-23634 > Project: HBase > Issue Type: Task >Affects Versions: 3.0.0-alpha-1, 2.3.0 >Reporter: Guanghao Zhang >Priority: Blocker > Fix For: 3.0.0-alpha-1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23634) Enable "Split WAL to HFile" by default
[
https://issues.apache.org/jira/browse/HBASE-23634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164052#comment-17164052
]
Guanghao Zhang commented on HBASE-23634:
{quote}Say if HFile was placed under region/cf/recovered.edits/ dir, we could have cleaned it up before doing the next attempt.
{quote}
Yes. This can work. So it is ok to have corruption file in the process of
split wal?
> Enable "Split WAL to HFile" by default
> --
>
> Key: HBASE-23634
> URL: https://issues.apache.org/jira/browse/HBASE-23634
> Project: HBase
> Issue Type: Task
>Affects Versions: 3.0.0-alpha-1, 2.3.0
>Reporter: Guanghao Zhang
>Priority: Blocker
> Fix For: 3.0.0-alpha-1
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-24749) Direct insert HFiles and Persist in-memory HFile tracking
[
https://issues.apache.org/jira/browse/HBASE-24749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164048#comment-17164048
]
Guanghao Zhang edited comment on HBASE-24749 at 7/24/20, 12:19 AM:
---
bq. it should be HBASE-20724, so for compaction we can reused that to confirm
if the flushed StoreFile were from a compaction.
Yes. The compaction event marker in WAL is not used anymore.
was (Author: zghaobac):
{quote}{quote} it should be HBASE-20724, so for compaction we can reused that
to confirm if the flushed StoreFile were from a compaction.
{quote}{quote}
Yes. The compaction event marker in WAL is not used anymore.
> Direct insert HFiles and Persist in-memory HFile tracking
> -
>
> Key: HBASE-24749
> URL: https://issues.apache.org/jira/browse/HBASE-24749
> Project: HBase
> Issue Type: Umbrella
> Components: Compaction, HFile
>Affects Versions: 3.0.0-alpha-1
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Labels: design, discussion, objectstore, storeFile, storeengine
> Attachments: 1B100m-25m25m-performance.pdf, Apache HBase - Direct
> insert HFiles and Persist in-memory HFile tracking.pdf
>
>
> We propose a new feature (a new store engine) to remove the {{.tmp}}
> directory used in the commit stage for common HFile operations such as flush
> and compaction to improve the write throughput and latency on object stores.
> Specifically for S3 filesystems, this will also mitigate read-after-write
> inconsistencies caused by immediate HFiles validation after moving the
> HFile(s) to data directory.
> Please see attached for this proposal and the initial result captured with
> 25m (25m operations) and 1B (100m operations) YCSB workload A LOAD and RUN,
> and workload C RUN result.
> The goal of this JIRA is to discuss with the community if the proposed
> improvement on the object stores use case makes senses and if we miss
> anything should be included.
> Improvement Highlights
> 1. Lower write latency, especially the p99+
> 2. Higher write throughput on flush and compaction
> 3. Lower MTTR on region (re)open or assignment
> 4. Remove consistent check dependencies (e.g. DynamoDB) supported by file
> system implementation
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-24749) Direct insert HFiles and Persist in-memory HFile tracking
[
https://issues.apache.org/jira/browse/HBASE-24749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164048#comment-17164048
]
Guanghao Zhang edited comment on HBASE-24749 at 7/24/20, 12:18 AM:
---
{quote}{quote} it should be HBASE-20724, so for compaction we can reused that
to confirm if the flushed StoreFile were from a compaction.
{quote}{quote}
Yes. The compaction event marker in WAL is not used anymore.
was (Author: zghaobac):
{quote}bq. it should be HBASE-20724, so for compaction we can reused that to
confirm if the flushed StoreFile were from a compaction.
{quote}
Yes. The compaction event marker is not used anymore.
> Direct insert HFiles and Persist in-memory HFile tracking
> -
>
> Key: HBASE-24749
> URL: https://issues.apache.org/jira/browse/HBASE-24749
> Project: HBase
> Issue Type: Umbrella
> Components: Compaction, HFile
>Affects Versions: 3.0.0-alpha-1
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Labels: design, discussion, objectstore, storeFile, storeengine
> Attachments: 1B100m-25m25m-performance.pdf, Apache HBase - Direct
> insert HFiles and Persist in-memory HFile tracking.pdf
>
>
> We propose a new feature (a new store engine) to remove the {{.tmp}}
> directory used in the commit stage for common HFile operations such as flush
> and compaction to improve the write throughput and latency on object stores.
> Specifically for S3 filesystems, this will also mitigate read-after-write
> inconsistencies caused by immediate HFiles validation after moving the
> HFile(s) to data directory.
> Please see attached for this proposal and the initial result captured with
> 25m (25m operations) and 1B (100m operations) YCSB workload A LOAD and RUN,
> and workload C RUN result.
> The goal of this JIRA is to discuss with the community if the proposed
> improvement on the object stores use case makes senses and if we miss
> anything should be included.
> Improvement Highlights
> 1. Lower write latency, especially the p99+
> 2. Higher write throughput on flush and compaction
> 3. Lower MTTR on region (re)open or assignment
> 4. Remove consistent check dependencies (e.g. DynamoDB) supported by file
> system implementation
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24749) Direct insert HFiles and Persist in-memory HFile tracking
[
https://issues.apache.org/jira/browse/HBASE-24749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164048#comment-17164048
]
Guanghao Zhang commented on HBASE-24749:
{quote}bq. it should be HBASE-20724, so for compaction we can reused that to
confirm if the flushed StoreFile were from a compaction.
{quote}
Yes. The compaction event marker is not used anymore.
> Direct insert HFiles and Persist in-memory HFile tracking
> -
>
> Key: HBASE-24749
> URL: https://issues.apache.org/jira/browse/HBASE-24749
> Project: HBase
> Issue Type: Umbrella
> Components: Compaction, HFile
>Affects Versions: 3.0.0-alpha-1
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Labels: design, discussion, objectstore, storeFile, storeengine
> Attachments: 1B100m-25m25m-performance.pdf, Apache HBase - Direct
> insert HFiles and Persist in-memory HFile tracking.pdf
>
>
> We propose a new feature (a new store engine) to remove the {{.tmp}}
> directory used in the commit stage for common HFile operations such as flush
> and compaction to improve the write throughput and latency on object stores.
> Specifically for S3 filesystems, this will also mitigate read-after-write
> inconsistencies caused by immediate HFiles validation after moving the
> HFile(s) to data directory.
> Please see attached for this proposal and the initial result captured with
> 25m (25m operations) and 1B (100m operations) YCSB workload A LOAD and RUN,
> and workload C RUN result.
> The goal of this JIRA is to discuss with the community if the proposed
> improvement on the object stores use case makes senses and if we miss
> anything should be included.
> Improvement Highlights
> 1. Lower write latency, especially the p99+
> 2. Higher write throughput on flush and compaction
> 3. Lower MTTR on region (re)open or assignment
> 4. Remove consistent check dependencies (e.g. DynamoDB) supported by file
> system implementation
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-24743) Reject to add a peer which replicate to itself earlier
[
https://issues.apache.org/jira/browse/HBASE-24743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang resolved HBASE-24743.
Resolution: Fixed
All ut passed. Pushed to branch-2 and master.
> Reject to add a peer which replicate to itself earlier
> --
>
> Key: HBASE-24743
> URL: https://issues.apache.org/jira/browse/HBASE-24743
> Project: HBase
> Issue Type: Improvement
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.4.0
>
>
> Now there are one check in ReplicationSource#initialize method
> {code:java}
> // In rare case, zookeeper setting may be messed up. That leads to the
> incorrect
> // peerClusterId value, which is the same as the source clusterId
> if (clusterId.equals(peerClusterId) &&
> !replicationEndpoint.canReplicateToSameCluster()) {
> this.terminate("ClusterId " + clusterId + " is replicating to itself:
> peerClusterId "
> + peerClusterId + " which is not allowed by ReplicationEndpoint:"
> + replicationEndpoint.getClass().getName(), null, false);
> this.manager.removeSource(this);
> return;
> }
> {code}
> This check should move to AddPeerProcedure's precheck.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] infraio commented on pull request #2124: HBASE-24743 Reject to add a peer which replicate to itself earlier
infraio commented on pull request #2124: URL: https://github.com/apache/hbase/pull/2124#issuecomment-663291501 Ok. All ut passed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] infraio merged pull request #2124: HBASE-24743 Reject to add a peer which replicate to itself earlier
infraio merged pull request #2124: URL: https://github.com/apache/hbase/pull/2124 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] infraio merged pull request #2122: HBASE-24743 Reject to add a peer which replicate to itself earlier
infraio merged pull request #2122: URL: https://github.com/apache/hbase/pull/2122 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] infraio commented on pull request #2122: HBASE-24743 Reject to add a peer which replicate to itself earlier
infraio commented on pull request #2122: URL: https://github.com/apache/hbase/pull/2122#issuecomment-663291310 Ok. All UT passed. Let me merge it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] saintstack opened a new pull request #2132: HBASE-24767 Change default to false for HBASE-15519 per-user metrics
saintstack opened a new pull request #2132: URL: https://github.com/apache/hbase/pull/2132 Set hbase.regionserver.user.metrics.enabled default to false; i.e. off. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24675) On Master restart all servers are assigned to default rsgroup.
[
https://issues.apache.org/jira/browse/HBASE-24675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164042#comment-17164042
]
Hudson commented on HBASE-24675:
Results for branch branch-2.2
[build #918 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/918/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/918//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/918//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/918//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> On Master restart all servers are assigned to default rsgroup.
> --
>
> Key: HBASE-24675
> URL: https://issues.apache.org/jira/browse/HBASE-24675
> Project: HBase
> Issue Type: Bug
> Components: rsgroup
>Affects Versions: 2.2.3
>Reporter: Mohammad Arshad
>Assignee: Mohammad Arshad
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0, 2.2.7
>
>
> Steps to reproduce:
> # Install a HBase cluster with three RS(rs1,rs2 and rs3) and one Master
> # Create two rsgroups r1 and r2 and move rs1 to r1 and rs2 to r2
> {code:java}
> add_rsgroup 'r1';add_rsgroup 'r2';move_servers_rsgroup
> 'r1',['host1:16020'];move_servers_rsgroup 'r2',['host2:16020']
> {code}
> # Restart Master
> # Run list_rsgroups for hbase shell, all region servers are assigned to
> default regroup.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-24767) Change default to false for HBASE-15519 per-user metrics
Michael Stack created HBASE-24767: - Summary: Change default to false for HBASE-15519 per-user metrics Key: HBASE-24767 URL: https://issues.apache.org/jira/browse/HBASE-24767 Project: HBase Issue Type: Bug Components: metrics Affects Versions: 2.3.0 Reporter: Michael Stack Assignee: Michael Stack Fix For: 3.0.0-alpha-1, 2.3.1, 2.4.0 HBASE-15519 added a nice feature to show per-user metrics. It needs a bit of work still – see base of HBASE-15519 for notes but in particular, it can spin up lots of threads, more than makes sense – but currently it is enabled by default. Here we disable it by setting default for hbase.regionserver.user.metrics.enabled to false. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24696) Include JVM information on Web UI under "Software Attributes"
[
https://issues.apache.org/jira/browse/HBASE-24696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164041#comment-17164041
]
Hudson commented on HBASE-24696:
Results for branch branch-2.2
[build #918 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/918/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/918//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/918//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/918//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Include JVM information on Web UI under "Software Attributes"
> -
>
> Key: HBASE-24696
> URL: https://issues.apache.org/jira/browse/HBASE-24696
> Project: HBase
> Issue Type: Improvement
> Components: UI
>Reporter: Nick Dimiduk
>Assignee: Mingliang Liu
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0, 2.1.10, 2.2.7
>
> Attachments: Screen Shot 2020-07-17 at 10.55.56 PM.png
>
>
> It's a small thing, but seems like an omission.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24749) Direct insert HFiles and Persist in-memory HFile tracking
[
https://issues.apache.org/jira/browse/HBASE-24749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164039#comment-17164039
]
Zach York commented on HBASE-24749:
---
[~anoop.hbase] Can you expand on how we can get in a situation where a partial
file is written? I'm trying to see if there are any failure modes we haven't
though of. If the case is a complete file written to the data directory, is
there harm in picking up the new file (even if it hasn't successfully committed
to the SFM)?
In any case, for all user tables, this could be covered by the new store file
list. The only case that is tricky/of concern is the file list for the root.
> On HBASE-14090, it is old but still cool, virtuous, aiming to hit a bigger
> target.
[~stack] We've definitely looked at it and gained some inspiration :) I think
at this point, we want to keep this in a manageable scope to be able to deliver
something. However, this approach should help break some of the reliance on FS
structure and make it easier to accomplish the goal in the future.
> Direct insert HFiles and Persist in-memory HFile tracking
> -
>
> Key: HBASE-24749
> URL: https://issues.apache.org/jira/browse/HBASE-24749
> Project: HBase
> Issue Type: Umbrella
> Components: Compaction, HFile
>Affects Versions: 3.0.0-alpha-1
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Labels: design, discussion, objectstore, storeFile, storeengine
> Attachments: 1B100m-25m25m-performance.pdf, Apache HBase - Direct
> insert HFiles and Persist in-memory HFile tracking.pdf
>
>
> We propose a new feature (a new store engine) to remove the {{.tmp}}
> directory used in the commit stage for common HFile operations such as flush
> and compaction to improve the write throughput and latency on object stores.
> Specifically for S3 filesystems, this will also mitigate read-after-write
> inconsistencies caused by immediate HFiles validation after moving the
> HFile(s) to data directory.
> Please see attached for this proposal and the initial result captured with
> 25m (25m operations) and 1B (100m operations) YCSB workload A LOAD and RUN,
> and workload C RUN result.
> The goal of this JIRA is to discuss with the community if the proposed
> improvement on the object stores use case makes senses and if we miss
> anything should be included.
> Improvement Highlights
> 1. Lower write latency, especially the p99+
> 2. Higher write throughput on flush and compaction
> 3. Lower MTTR on region (re)open or assignment
> 4. Remove consistent check dependencies (e.g. DynamoDB) supported by file
> system implementation
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-24749) Direct insert HFiles and Persist in-memory HFile tracking
[
https://issues.apache.org/jira/browse/HBASE-24749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164037#comment-17164037
]
Tak-Lon (Stephen) Wu edited comment on HBASE-24749 at 7/24/20, 12:03 AM:
-
bq. If an HFile is written successfully but no marker in the WAL, then it
doesn't exist, right? As part of the WAL replay you will reconstitute it from
edits in the WAL?
You're right if that happens, any uncommitted HFile should not be taken and
replay from the WALs. (assuming that replay may generate the same content but a
different HFile, I was thinking an store open optimization that but that would
be too far from now.)
bq. you have surveyed the calls to the NN made by HBase on a regular basis?
we don't have how many rename calls captured to NN or even to object stores,
and we will add a measurement survey task on the related milestone. But at one
point we captured while running compaction with rename, the overall wall clock
time only for the rename part were dominating ~60% of overall compaction time
on object stores.
bq. IIRC there is a issue for storing the compacted files in HFile's metadata,
to solve the problem that the wal file contains the compaction marker may be
deleted before wal splitting.
it should be HBASE-20724, so for compaction we can reused that to confirm if
the flushed StoreFile were from a compaction.
bq. Now while replay of wal, we dont have start compaction marker for this wal
file. So we think this is an old valid file but that is wrong. This is a
partial file. This is possible.
don't we only have the `end` compaction event marker only when compacted
HFile(s) has been moved to cf directory right before updating the store file
manager? but yeah if the WAL is rolled, then we lost this even marker.
but this is a good discussion, I will put down the above discussion as
consideration to related milestone.
was (Author: taklwu):
bq. If an HFile is written successfully but no marker in the WAL, then it
doesn't exist, right? As part of the WAL replay you will reconstitute it from
edits in the WAL?
You're right if that happens, any uncommitted HFile should not be taken and
replay from the WALs. (assuming that replay may generate the same content but a
different HFile, I was thinking an store open optimization that but that would
be too far from now.)
bq. you have surveyed the calls to the NN made by HBase on a regular basis?
we don't have how many rename calls captured to NN or even to object stores,
and we will add a measurement survey task on the related milestone. But at one
point we captured while running compaction with rename, the overall wall clock
time only for the rename part were dominating ~60% of overall compaction time
on object stores.
bq. IIRC there is a issue for storing the compacted files in HFile's metadata,
to solve the problem that the wal file contains the compaction marker may be
deleted before wal splitting.
it should be HBASE-20724, so for compaction we can reused that to confirm if
the flushed StoreFile were from a compaction.
bq. Now while replay of wal, we dont have start compaction marker for this wal
file. So we think this is an old valid file but that is wrong. This is a
partial file. This is possible.
don't we only have the `end` compaction event marker only when compacted
HFile(s) has been moved to cf directory right before updating the store file
manager? but yeah if the WAL is rolled, then we lost this even marker.
> Direct insert HFiles and Persist in-memory HFile tracking
> -
>
> Key: HBASE-24749
> URL: https://issues.apache.org/jira/browse/HBASE-24749
> Project: HBase
> Issue Type: Umbrella
> Components: Compaction, HFile
>Affects Versions: 3.0.0-alpha-1
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Labels: design, discussion, objectstore, storeFile, storeengine
> Attachments: 1B100m-25m25m-performance.pdf, Apache HBase - Direct
> insert HFiles and Persist in-memory HFile tracking.pdf
>
>
> We propose a new feature (a new store engine) to remove the {{.tmp}}
> directory used in the commit stage for common HFile operations such as flush
> and compaction to improve the write throughput and latency on object stores.
> Specifically for S3 filesystems, this will also mitigate read-after-write
> inconsistencies caused by immediate HFiles validation after moving the
> HFile(s) to data directory.
> Please see attached for this proposal and the initial result captured with
> 25m (25m operations) and 1B (100m operations) YCSB workload A LOAD and RUN,
> and workload C RUN result.
> The goal of this JIRA is to discuss with the community if the proposed
> improvement on the object stores use
[jira] [Comment Edited] (HBASE-24749) Direct insert HFiles and Persist in-memory HFile tracking
[
https://issues.apache.org/jira/browse/HBASE-24749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164037#comment-17164037
]
Tak-Lon (Stephen) Wu edited comment on HBASE-24749 at 7/24/20, 12:03 AM:
-
bq. If an HFile is written successfully but no marker in the WAL, then it
doesn't exist, right? As part of the WAL replay you will reconstitute it from
edits in the WAL?
You're right if that happens, any uncommitted HFile should not be taken and
replay from the WALs. (assuming that replay may generate the same content but a
different HFile, I was thinking an store open optimization that but that would
be too far from now.)
bq. you have surveyed the calls to the NN made by HBase on a regular basis?
we don't have how many rename calls captured to NN or even to object stores,
and we will add a measurement survey task on the related milestone. But at one
point we captured while running compaction with rename, the overall wall clock
time only for the rename part were dominating ~60% of overall compaction time
on object stores.
bq. IIRC there is a issue for storing the compacted files in HFile's metadata,
to solve the problem that the wal file contains the compaction marker may be
deleted before wal splitting.
it should be HBASE-20724, so for compaction we can reused that to confirm if
the flushed StoreFile were from a compaction.
bq. Now while replay of wal, we dont have start compaction marker for this wal
file. So we think this is an old valid file but that is wrong. This is a
partial file. This is possible.
don't we only have the `end` compaction event marker only when compacted
HFile(s) has been moved to cf directory right before updating the store file
manager? but yeah if the WAL is rolled, then we lost this even marker.
this is a good discussion, I will put down the above discussion as
consideration to related milestone.
was (Author: taklwu):
bq. If an HFile is written successfully but no marker in the WAL, then it
doesn't exist, right? As part of the WAL replay you will reconstitute it from
edits in the WAL?
You're right if that happens, any uncommitted HFile should not be taken and
replay from the WALs. (assuming that replay may generate the same content but a
different HFile, I was thinking an store open optimization that but that would
be too far from now.)
bq. you have surveyed the calls to the NN made by HBase on a regular basis?
we don't have how many rename calls captured to NN or even to object stores,
and we will add a measurement survey task on the related milestone. But at one
point we captured while running compaction with rename, the overall wall clock
time only for the rename part were dominating ~60% of overall compaction time
on object stores.
bq. IIRC there is a issue for storing the compacted files in HFile's metadata,
to solve the problem that the wal file contains the compaction marker may be
deleted before wal splitting.
it should be HBASE-20724, so for compaction we can reused that to confirm if
the flushed StoreFile were from a compaction.
bq. Now while replay of wal, we dont have start compaction marker for this wal
file. So we think this is an old valid file but that is wrong. This is a
partial file. This is possible.
don't we only have the `end` compaction event marker only when compacted
HFile(s) has been moved to cf directory right before updating the store file
manager? but yeah if the WAL is rolled, then we lost this even marker.
but this is a good discussion, I will put down the above discussion as
consideration to related milestone.
> Direct insert HFiles and Persist in-memory HFile tracking
> -
>
> Key: HBASE-24749
> URL: https://issues.apache.org/jira/browse/HBASE-24749
> Project: HBase
> Issue Type: Umbrella
> Components: Compaction, HFile
>Affects Versions: 3.0.0-alpha-1
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Labels: design, discussion, objectstore, storeFile, storeengine
> Attachments: 1B100m-25m25m-performance.pdf, Apache HBase - Direct
> insert HFiles and Persist in-memory HFile tracking.pdf
>
>
> We propose a new feature (a new store engine) to remove the {{.tmp}}
> directory used in the commit stage for common HFile operations such as flush
> and compaction to improve the write throughput and latency on object stores.
> Specifically for S3 filesystems, this will also mitigate read-after-write
> inconsistencies caused by immediate HFiles validation after moving the
> HFile(s) to data directory.
> Please see attached for this proposal and the initial result captured with
> 25m (25m operations) and 1B (100m operations) YCSB workload A LOAD and RUN,
> and workload C RUN result.
> The
[GitHub] [hbase] Apache-HBase commented on pull request #2130: HBASE-24765: Dynamic master discovery
Apache-HBase commented on pull request #2130: URL: https://github.com/apache/hbase/pull/2130#issuecomment-663289529 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 21s | Docker mode activated. | | -0 :warning: | yetus | 0m 2s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 23s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 57s | master passed | | +1 :green_heart: | compile | 2m 11s | master passed | | +1 :green_heart: | shadedjars | 6m 5s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 13s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 47s | the patch passed | | +1 :green_heart: | compile | 2m 10s | the patch passed | | +1 :green_heart: | javac | 2m 10s | the patch passed | | +1 :green_heart: | shadedjars | 6m 5s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 1m 10s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 0m 49s | hbase-protocol-shaded in the patch passed. | | -1 :x: | unit | 0m 56s | hbase-client in the patch failed. | | +1 :green_heart: | unit | 208m 58s | hbase-server in the patch passed. | | | | 241m 29s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2130/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2130 | | JIRA Issue | HBASE-24765 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux d4ae8b4b288e 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 8191fbdd7d | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2130/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-client.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2130/1/testReport/ | | Max. process+thread count | 3936 (vs. ulimit of 12500) | | modules | C: hbase-protocol-shaded hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2130/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24749) Direct insert HFiles and Persist in-memory HFile tracking
[
https://issues.apache.org/jira/browse/HBASE-24749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164037#comment-17164037
]
Tak-Lon (Stephen) Wu commented on HBASE-24749:
--
bq. If an HFile is written successfully but no marker in the WAL, then it
doesn't exist, right? As part of the WAL replay you will reconstitute it from
edits in the WAL?
You're right if that happens, any uncommitted HFile should not be taken and
replay from the WALs. (assuming that replay may generate the same content but a
different HFile, I was thinking an store open optimization that but that would
be too far from now.)
bq. you have surveyed the calls to the NN made by HBase on a regular basis?
we don't have how many rename calls captured to NN or even to object stores,
and we will add a measurement survey task on the related milestone. But at one
point we captured while running compaction with rename, the overall wall clock
time only for the rename part were dominating ~60% of overall compaction time
on object stores.
bq. IIRC there is a issue for storing the compacted files in HFile's metadata,
to solve the problem that the wal file contains the compaction marker may be
deleted before wal splitting.
it should be HBASE-20724, so for compaction we can reused that to confirm if
the flushed StoreFile were from a compaction.
bq. Now while replay of wal, we dont have start compaction marker for this wal
file. So we think this is an old valid file but that is wrong. This is a
partial file. This is possible.
don't we only have the `end` compaction event marker only when compacted
HFile(s) has been moved to cf directory right before updating the store file
manager? but yeah if the WAL is rolled, then we lost this even marker.
> Direct insert HFiles and Persist in-memory HFile tracking
> -
>
> Key: HBASE-24749
> URL: https://issues.apache.org/jira/browse/HBASE-24749
> Project: HBase
> Issue Type: Umbrella
> Components: Compaction, HFile
>Affects Versions: 3.0.0-alpha-1
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Labels: design, discussion, objectstore, storeFile, storeengine
> Attachments: 1B100m-25m25m-performance.pdf, Apache HBase - Direct
> insert HFiles and Persist in-memory HFile tracking.pdf
>
>
> We propose a new feature (a new store engine) to remove the {{.tmp}}
> directory used in the commit stage for common HFile operations such as flush
> and compaction to improve the write throughput and latency on object stores.
> Specifically for S3 filesystems, this will also mitigate read-after-write
> inconsistencies caused by immediate HFiles validation after moving the
> HFile(s) to data directory.
> Please see attached for this proposal and the initial result captured with
> 25m (25m operations) and 1B (100m operations) YCSB workload A LOAD and RUN,
> and workload C RUN result.
> The goal of this JIRA is to discuss with the community if the proposed
> improvement on the object stores use case makes senses and if we miss
> anything should be included.
> Improvement Highlights
> 1. Lower write latency, especially the p99+
> 2. Higher write throughput on flush and compaction
> 3. Lower MTTR on region (re)open or assignment
> 4. Remove consistent check dependencies (e.g. DynamoDB) supported by file
> system implementation
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2131: HBASE-24766 Document Remote Procedure Execution
Apache-HBase commented on pull request #2131: URL: https://github.com/apache/hbase/pull/2131#issuecomment-663288955 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 26s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ branch-2 Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 53s | branch-2 passed | | +1 :green_heart: | checkstyle | 1m 20s | branch-2 passed | | +1 :green_heart: | spotbugs | 2m 22s | branch-2 passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 5s | the patch passed | | +1 :green_heart: | checkstyle | 1m 24s | the patch passed | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 15m 29s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | spotbugs | 2m 42s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 13s | The patch does not generate ASF License warnings. | | | | 43m 48s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2131/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2131 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle | | uname | Linux 481f8ae46983 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | branch-2 / ce4e692699 | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2131/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-15519) Add per-user metrics
[ https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164035#comment-17164035 ] Michael Stack commented on HBASE-15519: --- Let me set the default as off. It seems like the lossy counters can come and go. In a little test cluster I see they rise and fall but even still.. I see instances of 40 threads which seems a bunch for counting. In my prod situation, there's a steady 140+ threads doing lossycounting which is a bit over the top. > Add per-user metrics > - > > Key: HBASE-15519 > URL: https://issues.apache.org/jira/browse/HBASE-15519 > Project: HBase > Issue Type: Sub-task > Components: metrics >Affects Versions: 1.2.0 >Reporter: Enis Soztutar >Assignee: Ankit Singhal >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0 > > Attachments: HBASE-15519.master.003.patch, hbase-15519_v0.patch, > hbase-15519_v1.patch, hbase-15519_v1.patch, hbase-15519_v2.patch > > > Per-user metrics will be useful in multi-tenant cases where we can emit > number of requests, operations, num RPCs etc at the per-user aggregate level > per regionserver. We currently have throttles per user, but no way to monitor > resource usage per-user. > Looking at these metrics, operators can adjust throttles, do capacity > planning, etc per-user. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[ https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164023#comment-17164023 ] Michael Stack commented on HBASE-11288: --- A deadline seems reasonable to me for making a call on how to proceed. I offer to help w/ the ITBLL eval if that'd be of use (on branch-2, what I know). As an aside, pardon if you've seen this already, but I liked this post on the merits of the design process and of the value documenting decisions: [https://www.industrialempathy.com/posts/design-docs-at-google/] > Splittable Meta > --- > > Key: HBASE-11288 > URL: https://issues.apache.org/jira/browse/HBASE-11288 > Project: HBase > Issue Type: Umbrella > Components: meta >Reporter: Francis Christopher Liu >Assignee: Francis Christopher Liu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] saintstack opened a new pull request #2131: HBASE-24766 Document Remote Procedure Execution
saintstack opened a new pull request #2131: URL: https://github.com/apache/hbase/pull/2131 Add outline of how the mechanism works generally. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2130: HBASE-24765: Dynamic master discovery
Apache-HBase commented on pull request #2130: URL: https://github.com/apache/hbase/pull/2130#issuecomment-663269079 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 59s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 5m 35s | master passed | | +1 :green_heart: | compile | 3m 15s | master passed | | +1 :green_heart: | shadedjars | 8m 11s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 28s | hbase-client in master failed. | | -0 :warning: | javadoc | 0m 42s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 12s | the patch passed | | +1 :green_heart: | compile | 2m 32s | the patch passed | | +1 :green_heart: | javac | 2m 32s | the patch passed | | +1 :green_heart: | shadedjars | 5m 47s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 27s | hbase-client in the patch failed. | | -0 :warning: | javadoc | 0m 40s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 0s | hbase-protocol-shaded in the patch passed. | | -1 :x: | unit | 0m 53s | hbase-client in the patch failed. | | +1 :green_heart: | unit | 129m 52s | hbase-server in the patch passed. | | | | 168m 26s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2130/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2130 | | JIRA Issue | HBASE-24765 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 1919cfbddf4f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 8191fbdd7d | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2130/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2130/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2130/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2130/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2130/1/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-client.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2130/1/testReport/ | | Max. process+thread count | 4474 (vs. ulimit of 12500) | | modules | C: hbase-protocol-shaded hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2130/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] z-york commented on pull request #2113: HBASE-24286: HMaster won't become healthy after after cloning or crea…
z-york commented on pull request #2113: URL: https://github.com/apache/hbase/pull/2113#issuecomment-663260010 So the use case here is starting a new cluster in the cloud where HDFS (WAL) data on the previous cluster will not be available. One of the benefits of storing the data off the cluster (in our case, S3), is to not have to replicate data (and just create a new cluster pointed to the same root directory). IMO, in this case we shouldn't need the WAL directories to exist just to tell us to reassign and this is a valid use case. I get that there is pushback for enabling this in the catalog cleaner, and I think that's fine. For this case, it's a one time issue, not something that periodically needs fixing. (there might be other unknown server cases that would require that, but that isn't blocking us at the moment). So, instead maybe a 1-time run to cleanup old servers/schedule SCP for them (this is what the code that was removed in HBASE-20708 actually did) makes the most sense? I understand that it was removed to simplify the assignment, but it has a very different behavior. In fact it looks like we don't even try to read hbase:meta if it is found (without SCP/WAL) and simply just delete the directory[1]. What problem is being solved by deleting instead of trying to assign it if data is there? [1] https://github.com/apache/hbase/blob/bad2d4e409ba57014e0f5931b72e54cc397e268a/hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/InitMetaProcedure.java#L75-L81 > For your scenario, it is OK as you can confirm that there is no data loss as you manually flushed all the data. But what if another user just configs the wrong WAL directory? In this case, if we schedule SCPs automatically, there will be data loss. In this scenario, regardless of what we do, there will be dataloss unless the correct WAL directory is (again) specified. In fact, I don't believe you can change WAL dir without restarting servers (I also don't think it works with rolling restart). I don't think this is a valid scenario for this issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-24632) Enable procedure-based log splitting as default in hbase3
[
https://issues.apache.org/jira/browse/HBASE-24632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163998#comment-17163998
]
Michael Stack commented on HBASE-24632:
---
{quote} So here also once the work is given, the thread wont get blocked but
will yield?
{quote}
Thats right. The RPD will dispatch the split WAL job to the remote RS wrapped
in a ExecuteProceduresRemoteCall. Once the RPC delivers the job to the remote
RS, the RPD dispatcher is done (and its thread from the RPD executor of 128
threads is now idle again). Meanwhile the split WAL starts to run on the RS
side. When it is done, it will do a similar call back to the Master to tell it
success or failure but this is different RPC that originates at the RS.
HBASE-24766 is issue where I'll add some doc around this stuff. The questions
above are good. I said I'd do doc in this area a while ago. Let me do now.
> Enable procedure-based log splitting as default in hbase3
> -
>
> Key: HBASE-24632
> URL: https://issues.apache.org/jira/browse/HBASE-24632
> Project: HBase
> Issue Type: Sub-task
> Components: wal
>Reporter: Michael Stack
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.4.0
>
>
> Means changing this value in HConstants to false:
>public static final boolean DEFAULT_HBASE_SPLIT_COORDINATED_BY_ZK = true;
> Should probably also deprecate the current zk distributed split too so we can
> clear out those classes to.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-24766) Document Remote Procedure Execution
Michael Stack created HBASE-24766: - Summary: Document Remote Procedure Execution Key: HBASE-24766 URL: https://issues.apache.org/jira/browse/HBASE-24766 Project: HBase Issue Type: Bug Components: documentation Reporter: Michael Stack The doc in ServerRemoteProcedure, the main class is good. I said I'd give it a tuneup now we've a few impls of this feature. I want to doc in the code rather than in refguide since it devs who would use it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-11288) Splittable Meta
[
https://issues.apache.org/jira/browse/HBASE-11288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163989#comment-17163989
]
Hudson commented on HBASE-11288:
Results for branch HBASE-11288.splittable-meta
[build #18 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/18/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/18/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/18/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-11288.splittable-meta/18/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Splittable Meta
> ---
>
> Key: HBASE-11288
> URL: https://issues.apache.org/jira/browse/HBASE-11288
> Project: HBase
> Issue Type: Umbrella
> Components: meta
>Reporter: Francis Christopher Liu
>Assignee: Francis Christopher Liu
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24662) Update DumpClusterStatusAction to notice changes in region server count
[ https://issues.apache.org/jira/browse/HBASE-24662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163986#comment-17163986 ] Bharath Vissapragada commented on HBASE-24662: -- > You're saying you've worked around the lack of HBASE-24296 for other patches > in the past Sorry missed your comment. Right, I worked around the conflicts before but thanks for fixing it now. > Update DumpClusterStatusAction to notice changes in region server count > --- > > Key: HBASE-24662 > URL: https://issues.apache.org/jira/browse/HBASE-24662 > Project: HBase > Issue Type: Task > Components: integration tests >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.1, 2.4.0 > > > Sometimes running chaos monkey, I've found that we lose accounting of region > servers. I've taken to a manual process of checking the reported list against > a known reference. It occurs to me that ChaosMonkey has a known reference, > and it can do this accounting for me. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-15519) Add per-user metrics
[ https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163980#comment-17163980 ] Andrew Kyle Purtell commented on HBASE-15519: - Can we make lossy counting housekeeping global to the LossyCounting class (or a helper class) so the overhead is constant no matter how many instances? We already accept imprecision so having one thread service them all should be reasonable... or maybe some small thread pool sized logarithmically with respect to the total number of instances? > Add per-user metrics > - > > Key: HBASE-15519 > URL: https://issues.apache.org/jira/browse/HBASE-15519 > Project: HBase > Issue Type: Sub-task > Components: metrics >Affects Versions: 1.2.0 >Reporter: Enis Soztutar >Assignee: Ankit Singhal >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0 > > Attachments: HBASE-15519.master.003.patch, hbase-15519_v0.patch, > hbase-15519_v1.patch, hbase-15519_v1.patch, hbase-15519_v2.patch > > > Per-user metrics will be useful in multi-tenant cases where we can emit > number of requests, operations, num RPCs etc at the per-user aggregate level > per regionserver. We currently have throttles per user, but no way to monitor > resource usage per-user. > Looking at these metrics, operators can adjust throttles, do capacity > planning, etc per-user. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-15519) Add per-user metrics
[ https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163978#comment-17163978 ] Michael Stack commented on HBASE-15519: --- Thanks for the prompt response [~an...@apache.org] The lossy-count thread spin up seems extreme at least. How we figure user name of incoming client seems like it could get some improvement: i.e. instead of 'hbase', perhaps 'hbase.ip' or 'hbase.hostname'... Metrics have bloated now I look. This is not main offender (though it could grow to be if lots of users?) but I see between 35-45 MBeans and then between 2M and 3M for the list of all metrics... which seems like a lot. This looks like a really useful feature but on by default might be a bit much. Ideal I think would be being able to switch it on when a problem but then off again given its expense But it doesn't seem possible given the way our metrics work it seems. Thanks for entertaining my questions. > Add per-user metrics > - > > Key: HBASE-15519 > URL: https://issues.apache.org/jira/browse/HBASE-15519 > Project: HBase > Issue Type: Sub-task > Components: metrics >Affects Versions: 1.2.0 >Reporter: Enis Soztutar >Assignee: Ankit Singhal >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0 > > Attachments: HBASE-15519.master.003.patch, hbase-15519_v0.patch, > hbase-15519_v1.patch, hbase-15519_v1.patch, hbase-15519_v2.patch > > > Per-user metrics will be useful in multi-tenant cases where we can emit > number of requests, operations, num RPCs etc at the per-user aggregate level > per regionserver. We currently have throttles per user, but no way to monitor > resource usage per-user. > Looking at these metrics, operators can adjust throttles, do capacity > planning, etc per-user. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2127: HBASE-24757 : ReplicationSink should limit row count in batch mutation based on hbase.rpc.rows.warning.threshold
Apache-HBase commented on pull request #2127: URL: https://github.com/apache/hbase/pull/2127#issuecomment-663243730 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 38s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 44s | master passed | | +1 :green_heart: | compile | 1m 20s | master passed | | +1 :green_heart: | shadedjars | 6m 6s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 55s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 14s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 50s | the patch passed | | +1 :green_heart: | compile | 1m 18s | the patch passed | | +1 :green_heart: | javac | 1m 18s | the patch passed | | +1 :green_heart: | shadedjars | 6m 7s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 56s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 39s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 210m 44s | hbase-server in the patch passed. | | | | 240m 36s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2127/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2127 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux de83455c9947 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 8191fbdd7d | | Default Java | 1.8.0_232 | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2127/2/testReport/ | | Max. process+thread count | 2984 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2127/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-15519) Add per-user metrics
[
https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163961#comment-17163961
]
Ankit Singhal commented on HBASE-15519:
---
{quote}Each instance creates an executor which has a single thread to do
lossy-counting. This seems profligate? No?
{quote}
Yes Looks like a miss by me, we could share an executor with more threads and a
non-blocking queue here.
{quote}Does this feature work? I see the unit tests but on a real cluster where
I have many users, in these metrics, the user shows as 'nobody' or 'hbase'...
{quote}
I tested it with few users on my local setup and I was able to see the users
appearing with the right request count when the requests are coming and
disappears as there is no request, Idea was to get only the top users at a
time. Let me recollect my knowledge around lossyCounting and see how the users
are removed.
> Add per-user metrics
> -
>
> Key: HBASE-15519
> URL: https://issues.apache.org/jira/browse/HBASE-15519
> Project: HBase
> Issue Type: Sub-task
> Components: metrics
>Affects Versions: 1.2.0
>Reporter: Enis Soztutar
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0
>
> Attachments: HBASE-15519.master.003.patch, hbase-15519_v0.patch,
> hbase-15519_v1.patch, hbase-15519_v1.patch, hbase-15519_v2.patch
>
>
> Per-user metrics will be useful in multi-tenant cases where we can emit
> number of requests, operations, num RPCs etc at the per-user aggregate level
> per regionserver. We currently have throttles per user, but no way to monitor
> resource usage per-user.
> Looking at these metrics, operators can adjust throttles, do capacity
> planning, etc per-user.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-15519) Add per-user metrics
[ https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163953#comment-17163953 ] Michael Stack commented on HBASE-15519: --- Does this feature work? I see the unit tests but on a real cluster where I have many users, in these metrics, the user shows as 'nobody' or 'hbase'... > Add per-user metrics > - > > Key: HBASE-15519 > URL: https://issues.apache.org/jira/browse/HBASE-15519 > Project: HBase > Issue Type: Sub-task > Components: metrics >Affects Versions: 1.2.0 >Reporter: Enis Soztutar >Assignee: Ankit Singhal >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0 > > Attachments: HBASE-15519.master.003.patch, hbase-15519_v0.patch, > hbase-15519_v1.patch, hbase-15519_v1.patch, hbase-15519_v2.patch > > > Per-user metrics will be useful in multi-tenant cases where we can emit > number of requests, operations, num RPCs etc at the per-user aggregate level > per regionserver. We currently have throttles per user, but no way to monitor > resource usage per-user. > Looking at these metrics, operators can adjust throttles, do capacity > planning, etc per-user. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2130: HBASE-24765: Dynamic master discovery
Apache-HBase commented on pull request #2130: URL: https://github.com/apache/hbase/pull/2130#issuecomment-663229929 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 21s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +0 :ok: | prototool | 0m 0s | prototool was not available. | | +1 :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 26s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 40s | master passed | | +1 :green_heart: | checkstyle | 2m 3s | master passed | | +1 :green_heart: | spotbugs | 8m 19s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 36s | the patch passed | | -0 :warning: | checkstyle | 0m 31s | hbase-client: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) | | -0 :warning: | checkstyle | 1m 24s | hbase-server: The patch generated 2 new + 106 unchanged - 0 fixed = 108 total (was 106) | | +1 :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | +1 :green_heart: | hadoopcheck | 14m 33s | Patch does not cause any errors with Hadoop 3.1.2 3.2.1. | | +1 :green_heart: | hbaseprotoc | 2m 25s | the patch passed | | +1 :green_heart: | spotbugs | 8m 50s | the patch passed | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 35s | The patch does not generate ASF License warnings. | | | | 60m 28s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.9 Server=19.03.9 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2130/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2130 | | JIRA Issue | HBASE-24765 | | Optional Tests | dupname asflicense spotbugs hadoopcheck hbaseanti checkstyle cc hbaseprotoc prototool | | uname | Linux 2e51900bfdca 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 8191fbdd7d | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2130/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-client.txt | | checkstyle | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2130/1/artifact/yetus-general-check/output/diff-checkstyle-hbase-server.txt | | Max. process+thread count | 84 (vs. ulimit of 12500) | | modules | C: hbase-protocol-shaded hbase-client hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2130/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) spotbugs=3.1.12 | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-15519) Add per-user metrics
[ https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163943#comment-17163943 ] Michael Stack commented on HBASE-15519: --- As I read this, we have per-user metrics enabled per Region. Each instance creates an executor which has a single thread to do lossy-counting. This seems profligate? No? It shouldn't be default, do ye think? > Add per-user metrics > - > > Key: HBASE-15519 > URL: https://issues.apache.org/jira/browse/HBASE-15519 > Project: HBase > Issue Type: Sub-task > Components: metrics >Affects Versions: 1.2.0 >Reporter: Enis Soztutar >Assignee: Ankit Singhal >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0 > > Attachments: HBASE-15519.master.003.patch, hbase-15519_v0.patch, > hbase-15519_v1.patch, hbase-15519_v1.patch, hbase-15519_v2.patch > > > Per-user metrics will be useful in multi-tenant cases where we can emit > number of requests, operations, num RPCs etc at the per-user aggregate level > per regionserver. We currently have throttles per user, but no way to monitor > resource usage per-user. > Looking at these metrics, operators can adjust throttles, do capacity > planning, etc per-user. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2118: HBASE-24758 Avoid flooding replication source RSes logs when no sinks…
Apache-HBase commented on pull request #2118: URL: https://github.com/apache/hbase/pull/2118#issuecomment-663228402 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 28s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 20s | master passed | | +1 :green_heart: | compile | 0m 59s | master passed | | +1 :green_heart: | shadedjars | 6m 8s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 37s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 57s | the patch passed | | +1 :green_heart: | compile | 0m 59s | the patch passed | | +1 :green_heart: | javac | 0m 59s | the patch passed | | +1 :green_heart: | shadedjars | 6m 5s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 36s | the patch passed | ||| _ Other Tests _ | | -1 :x: | unit | 231m 11s | hbase-server in the patch failed. | | | | 256m 57s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2118/2/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2118 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux b9f7589c18c2 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 8191fbdd7d | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2118/2/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2118/2/testReport/ | | Max. process+thread count | 2733 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2118/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HBASE-24765) Dynamic master discovery
[
https://issues.apache.org/jira/browse/HBASE-24765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharath Vissapragada updated HBASE-24765:
-
Status: Patch Available (was: Open)
> Dynamic master discovery
>
>
> Key: HBASE-24765
> URL: https://issues.apache.org/jira/browse/HBASE-24765
> Project: HBase
> Issue Type: Sub-task
> Components: Client
>Affects Versions: 2.3.0, 3.0.0-alpha-1
>Reporter: Bharath Vissapragada
>Assignee: Bharath Vissapragada
>Priority: Major
>
> [~stack]'s idea in the design doc for splittable-meta.
> We can keep a live list of masters to query by fetching the list of available
> masters from any of the available masters configured in the seed list. User
> configured list of masters ("hbase.masters") would be used as a seed list.
> The endpoints are refreshed every 5mins or if any of the registry RPCs hit an
> error (which ever happens first).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2118: HBASE-24758 Avoid flooding replication source RSes logs when no sinks…
Apache-HBase commented on pull request #2118: URL: https://github.com/apache/hbase/pull/2118#issuecomment-663224048 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 27s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 5m 10s | master passed | | +1 :green_heart: | compile | 1m 15s | master passed | | +1 :green_heart: | shadedjars | 6m 29s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 46s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 4m 40s | the patch passed | | +1 :green_heart: | compile | 1m 13s | the patch passed | | +1 :green_heart: | javac | 1m 13s | the patch passed | | +1 :green_heart: | shadedjars | 6m 43s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 50s | hbase-server in the patch failed. | ||| _ Other Tests _ | | -1 :x: | unit | 217m 18s | hbase-server in the patch failed. | | | | 246m 25s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2118/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2118 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 230e03c9a750 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 8191fbdd7d | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2118/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2118/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2118/2/artifact/yetus-jdk11-hadoop3-check/output/patch-unit-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2118/2/testReport/ | | Max. process+thread count | 3617 (vs. ulimit of 12500) | | modules | C: hbase-server U: hbase-server | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2118/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-24763) Update 2.1 documentation to 2.1.9 and remove direct link from main page
[ https://issues.apache.org/jira/browse/HBASE-24763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-24763. --- Fix Version/s: 3.0.0-alpha-1 Resolution: Fixed Pushed 2.1.9 docs to hbase-site repository and removed "2.1 Documentation" menu item from website. Thanks for the review [~busbey] and [~janh]! > Update 2.1 documentation to 2.1.9 and remove direct link from main page > --- > > Key: HBASE-24763 > URL: https://issues.apache.org/jira/browse/HBASE-24763 > Project: HBase > Issue Type: Task > Components: documentation, website >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Major > Fix For: 3.0.0-alpha-1 > > > For the 2.1 release the available documentation points to 2.1.5. Since the > release has reached EOL with 2.1.9 the documentation on the website should > reflect this and a direct link to 2.1 should be removed. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] petersomogyi merged pull request #2129: HBASE-24763 Remove 2.1 Documentation direct link from website
petersomogyi merged pull request #2129: URL: https://github.com/apache/hbase/pull/2129 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2129: HBASE-24763 Remove 2.1 Documentation direct link from website
Apache-HBase commented on pull request #2129: URL: https://github.com/apache/hbase/pull/2129#issuecomment-663212529 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | ||| _ Prechecks _ | | +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | | +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | ||| _ master Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 45s | master passed | | +1 :green_heart: | mvnsite | 10m 11s | master passed | ||| _ Patch Compile Tests _ | | +1 :green_heart: | mvninstall | 3m 28s | the patch passed | | +1 :green_heart: | mvnsite | 10m 7s | the patch passed | | +1 :green_heart: | whitespace | 0m 1s | The patch has no whitespace issues. | | +1 :green_heart: | xml | 0m 2s | The patch has no ill-formed XML file. | ||| _ Other Tests _ | | +1 :green_heart: | asflicense | 0m 21s | The patch does not generate ASF License warnings. | | | | 30m 4s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2129/1/artifact/yetus-general-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2129 | | Optional Tests | dupname asflicense mvnsite xml | | uname | Linux 8f9e81af8b7a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 8191fbdd7d | | Max. process+thread count | 89 (vs. ulimit of 12500) | | modules | C: . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2129/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2128: HBASE-24762 Purge protobuf java 2.5.0 dependency
Apache-HBase commented on pull request #2128: URL: https://github.com/apache/hbase/pull/2128#issuecomment-663211195 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 30s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 31s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 3m 41s | master passed | | +1 :green_heart: | compile | 2m 20s | master passed | | +1 :green_heart: | shadedjars | 5m 35s | branch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 3m 35s | master passed | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 21s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 3m 21s | the patch passed | | +1 :green_heart: | compile | 2m 16s | the patch passed | | +1 :green_heart: | javac | 2m 16s | the patch passed | | +1 :green_heart: | shadedjars | 5m 38s | patch has no errors when building our shaded downstream artifacts. | | +1 :green_heart: | javadoc | 0m 20s | hbase-common in the patch passed. | | +1 :green_heart: | javadoc | 0m 22s | hbase-client in the patch passed. | | +1 :green_heart: | javadoc | 0m 36s | hbase-server in the patch passed. | | +1 :green_heart: | javadoc | 0m 20s | hbase-examples generated 0 new + 0 unchanged - 72 fixed = 0 total (was 72) | | +1 :green_heart: | javadoc | 1m 57s | root generated 0 new + 43 unchanged - 72 fixed = 43 total (was 115) | ||| _ Other Tests _ | | -1 :x: | unit | 332m 44s | root in the patch failed. | | | | 367m 8s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2128/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2128 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 66ebe61335cb 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 8191fbdd7d | | Default Java | 1.8.0_232 | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2128/1/artifact/yetus-jdk8-hadoop3-check/output/patch-unit-root.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2128/1/testReport/ | | Max. process+thread count | 5554 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-client hbase-server hbase-examples . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2128/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2127: HBASE-24757 : ReplicationSink should limit row count in batch mutation based on hbase.rpc.rows.warning.threshold
Apache-HBase commented on pull request #2127: URL: https://github.com/apache/hbase/pull/2127#issuecomment-663210758 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 41s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 25s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 21s | master passed | | +1 :green_heart: | compile | 1m 32s | master passed | | +1 :green_heart: | shadedjars | 5m 49s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 20s | hbase-common in master failed. | | -0 :warning: | javadoc | 0m 39s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 15s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 2s | the patch passed | | +1 :green_heart: | compile | 1m 31s | the patch passed | | +1 :green_heart: | javac | 1m 31s | the patch passed | | +1 :green_heart: | shadedjars | 5m 47s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 18s | hbase-common in the patch failed. | | -0 :warning: | javadoc | 0m 39s | hbase-server in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 1m 33s | hbase-common in the patch passed. | | +1 :green_heart: | unit | 132m 38s | hbase-server in the patch passed. | | | | 162m 42s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2127/2/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2127 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 6900b87b01a5 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 8191fbdd7d | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2127/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-common.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2127/2/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2127/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-common.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2127/2/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2127/2/testReport/ | | Max. process+thread count | 4153 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-server U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2127/2/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (HBASE-15519) Add per-user metrics
[ https://issues.apache.org/jira/browse/HBASE-15519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163919#comment-17163919 ] Michael Stack commented on HBASE-15519: --- [~an...@apache.org] There are two users accessing my system but I have 138 instances of a thread whose name is always 'lossy-count-0". This seems like a mistake? [~xucang] ... flagging you because the basis here is your work (though off by default). Thanks. > Add per-user metrics > - > > Key: HBASE-15519 > URL: https://issues.apache.org/jira/browse/HBASE-15519 > Project: HBase > Issue Type: Sub-task > Components: metrics >Affects Versions: 1.2.0 >Reporter: Enis Soztutar >Assignee: Ankit Singhal >Priority: Major > Fix For: 3.0.0-alpha-1, 2.3.0 > > Attachments: HBASE-15519.master.003.patch, hbase-15519_v0.patch, > hbase-15519_v1.patch, hbase-15519_v1.patch, hbase-15519_v2.patch > > > Per-user metrics will be useful in multi-tenant cases where we can emit > number of requests, operations, num RPCs etc at the per-user aggregate level > per regionserver. We currently have throttles per user, but no way to monitor > resource usage per-user. > Looking at these metrics, operators can adjust throttles, do capacity > planning, etc per-user. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2129: HBASE-24763 Remove 2.1 Documentation direct link from website
Apache-HBase commented on pull request #2129: URL: https://github.com/apache/hbase/pull/2129#issuecomment-663201249 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 2m 22s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | ||| _ Patch Compile Tests _ | ||| _ Other Tests _ | | | | 3m 39s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2129/1/artifact/yetus-jdk8-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2129 | | Optional Tests | | | uname | Linux cdafc5166243 4.15.0-101-generic #102-Ubuntu SMP Mon May 11 10:07:26 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 8191fbdd7d | | Max. process+thread count | 42 (vs. ulimit of 12500) | | modules | C: . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2129/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] bharathv opened a new pull request #2130: HBASE-24765: Dynamic master discovery
bharathv opened a new pull request #2130: URL: https://github.com/apache/hbase/pull/2130 This patch adds the ability to discover newly added masters dynamically on the master registry side. The trigger for the re-fetch is either 5mins or any registry RPC failure. I didn't add the method to ZK registry interface since there is a design discussion going on in splittable meta doc. We can add it later if needed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase] Apache-HBase commented on pull request #2129: HBASE-24763 Remove 2.1 Documentation direct link from website
Apache-HBase commented on pull request #2129: URL: https://github.com/apache/hbase/pull/2129#issuecomment-663200572 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 0m 38s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | ||| _ Patch Compile Tests _ | ||| _ Other Tests _ | | | | 1m 58s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2129/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2129 | | Optional Tests | | | uname | Linux 296c30331b2f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 8191fbdd7d | | Max. process+thread count | 51 (vs. ulimit of 12500) | | modules | C: . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2129/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus 0.11.1 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase-site] petersomogyi merged pull request #6: HBASE-24763 Update 2.1 documentation to 2.1.9
petersomogyi merged pull request #6: URL: https://github.com/apache/hbase-site/pull/6 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HBASE-24765) Dynamic master discovery
Bharath Vissapragada created HBASE-24765:
Summary: Dynamic master discovery
Key: HBASE-24765
URL: https://issues.apache.org/jira/browse/HBASE-24765
Project: HBase
Issue Type: Sub-task
Components: Client
Affects Versions: 2.3.0, 3.0.0-alpha-1
Reporter: Bharath Vissapragada
Assignee: Bharath Vissapragada
[~stack]'s idea in the design doc for splittable-meta.
We can keep a live list of masters to query by fetching the list of available
masters from any of the available masters configured in the seed list. User
configured list of masters ("hbase.masters") would be used as a seed list.
The endpoints are refreshed every 5mins or if any of the registry RPCs hit an
error (which ever happens first).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [hbase] petersomogyi opened a new pull request #2129: HBASE-24763 Remove 2.1 Documentation direct link from website
petersomogyi opened a new pull request #2129: URL: https://github.com/apache/hbase/pull/2129 2.1 is EOM so the 2.1 Documentation link can be removed. The content is still available under https://hbase.apache.org/2.1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hbase-site] petersomogyi commented on pull request #6: HBASE-24763 Update 2.1 documentation to 2.1.9
petersomogyi commented on pull request #6: URL: https://github.com/apache/hbase-site/pull/6#issuecomment-663193784 Looks like GitHub is having trouble with the size of this commit. :) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (HBASE-24209) [Hadoop3.3] Add Hadoop-3.3.0-SNAPSHOT to hadoopcheck in our yetus personality
[ https://issues.apache.org/jira/browse/HBASE-24209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HBASE-24209. - Resolution: Won't Fix > [Hadoop3.3] Add Hadoop-3.3.0-SNAPSHOT to hadoopcheck in our yetus personality > - > > Key: HBASE-24209 > URL: https://issues.apache.org/jira/browse/HBASE-24209 > Project: HBase > Issue Type: Task > Components: build >Reporter: Nick Dimiduk >Assignee: Wei-Chiu Chuang >Priority: Major > > Since HBASE-23833 we're paying attention to our builds on Hadoop trunk, > currently 3.3.0-SNAPSHOT. Let's add this version to the version lists in > hadoopcheck so our CI will let us know when things break, at least > compile-time anyway. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24764) Add support of enabling default peer configs via hbase-site.xml for all default replication peers.
Ankit Jain created HBASE-24764: -- Summary: Add support of enabling default peer configs via hbase-site.xml for all default replication peers. Key: HBASE-24764 URL: https://issues.apache.org/jira/browse/HBASE-24764 Project: HBase Issue Type: Improvement Reporter: Ankit Jain Assignee: Ankit Jain Currently, if a user needs to apply some common peer configs to all the default replication peers, the only way is to execute update_peer_config via CLI which requires manual intervention and can be tedious in case of large deployment fleet. As part of this JIRA, we plan to add the support to have default replication peer configs as part of hbase-site.xml like hbase.replication.peer.default.config="k1=v1;k2=v2.." which can be just applied by a rolling restart. Example below: hbase.replication.peer.default.configs hbase.replication.source.custom.walentryfilters=x,y,z;hbase.rpc.protection=abc;hbase.xxx.custom_property=123 This will be empty by default, but one can override to have default configs in place. The final peer configuration would be a merge of this default config + whatever users override during the peer creation/update (if any). Related Jira: https://issues.apache.org/jira/browse/HBASE-17543. HBASE-17543 added the support to add the WALEntryFilters to default endpoint via peer configuration. By this new Jira we are extending the support to update peer configs via hbase-site.xml. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24696) Include JVM information on Web UI under "Software Attributes"
[ https://issues.apache.org/jira/browse/HBASE-24696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163865#comment-17163865 ] Viraj Jasani commented on HBASE-24696: -- (y) > Include JVM information on Web UI under "Software Attributes" > - > > Key: HBASE-24696 > URL: https://issues.apache.org/jira/browse/HBASE-24696 > Project: HBase > Issue Type: Improvement > Components: UI >Reporter: Nick Dimiduk >Assignee: Mingliang Liu >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0, 2.1.10, 2.2.7 > > Attachments: Screen Shot 2020-07-17 at 10.55.56 PM.png > > > It's a small thing, but seems like an omission. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [hbase] Apache-HBase commented on pull request #2128: HBASE-24762 Purge protobuf java 2.5.0 dependency
Apache-HBase commented on pull request #2128: URL: https://github.com/apache/hbase/pull/2128#issuecomment-663172924 :confetti_ball: **+1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | +0 :ok: | reexec | 1m 6s | Docker mode activated. | | -0 :warning: | yetus | 0m 3s | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --whitespace-eol-ignore-list --whitespace-tabs-ignore-list --quick-hadoopcheck | ||| _ Prechecks _ | ||| _ master Compile Tests _ | | +0 :ok: | mvndep | 0m 29s | Maven dependency ordering for branch | | +1 :green_heart: | mvninstall | 4m 40s | master passed | | +1 :green_heart: | compile | 3m 2s | master passed | | +1 :green_heart: | shadedjars | 6m 25s | branch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 15s | root in master failed. | | -0 :warning: | javadoc | 0m 25s | hbase-client in master failed. | | -0 :warning: | javadoc | 0m 15s | hbase-common in master failed. | | -0 :warning: | javadoc | 0m 16s | hbase-examples in master failed. | | -0 :warning: | javadoc | 0m 39s | hbase-server in master failed. | ||| _ Patch Compile Tests _ | | +0 :ok: | mvndep | 0m 19s | Maven dependency ordering for patch | | +1 :green_heart: | mvninstall | 4m 24s | the patch passed | | +1 :green_heart: | compile | 3m 2s | the patch passed | | +1 :green_heart: | javac | 3m 2s | the patch passed | | +1 :green_heart: | shadedjars | 6m 21s | patch has no errors when building our shaded downstream artifacts. | | -0 :warning: | javadoc | 0m 16s | hbase-common in the patch failed. | | -0 :warning: | javadoc | 0m 25s | hbase-client in the patch failed. | | -0 :warning: | javadoc | 0m 40s | hbase-server in the patch failed. | | -0 :warning: | javadoc | 0m 20s | hbase-examples in the patch failed. | | -0 :warning: | javadoc | 0m 14s | root in the patch failed. | ||| _ Other Tests _ | | +1 :green_heart: | unit | 243m 3s | root in the patch passed. | | | | 279m 16s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.12 Server=19.03.12 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2128/1/artifact/yetus-jdk11-hadoop3-check/output/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/2128 | | Optional Tests | javac javadoc unit shadedjars compile | | uname | Linux 9bb4eee29aee 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | dev-support/hbase-personality.sh | | git revision | master / 8191fbdd7d | | Default Java | 2020-01-14 | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2128/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-root.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2128/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2128/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-common.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2128/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-examples.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2128/1/artifact/yetus-jdk11-hadoop3-check/output/branch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2128/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-common.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2128/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-client.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2128/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-server.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2128/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-hbase-examples.txt | | javadoc | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2128/1/artifact/yetus-jdk11-hadoop3-check/output/patch-javadoc-root.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2128/1/testReport/ | | Max. process+thread count | 4332 (vs. ulimit of 12500) | | modules | C: hbase-common hbase-client hbase-server hbase-examples . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-2128/1/console | | versions | git=2.17.1 maven=(cecedd343002696d0abb50b32b541b8a6ba2883f) | | Powered by | Apache Yetus
[jira] [Comment Edited] (HBASE-24696) Include JVM information on Web UI under "Software Attributes"
[ https://issues.apache.org/jira/browse/HBASE-24696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163859#comment-17163859 ] Mingliang Liu edited comment on HBASE-24696 at 7/23/20, 6:44 PM: - Thank you [~psomogyi] Very useful information. I did not think of inferring from the download page or checking the book. was (Author: liuml07): Thank you [~psomogyi] Very useful information. I did not think of inferring from or download page or checking the book. > Include JVM information on Web UI under "Software Attributes" > - > > Key: HBASE-24696 > URL: https://issues.apache.org/jira/browse/HBASE-24696 > Project: HBase > Issue Type: Improvement > Components: UI >Reporter: Nick Dimiduk >Assignee: Mingliang Liu >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0, 2.1.10, 2.2.7 > > Attachments: Screen Shot 2020-07-17 at 10.55.56 PM.png > > > It's a small thing, but seems like an omission. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24696) Include JVM information on Web UI under "Software Attributes"
[ https://issues.apache.org/jira/browse/HBASE-24696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163859#comment-17163859 ] Mingliang Liu commented on HBASE-24696: --- Thank you [~psomogyi] Very useful information. I did not think of inferring from or download page or checking the book. > Include JVM information on Web UI under "Software Attributes" > - > > Key: HBASE-24696 > URL: https://issues.apache.org/jira/browse/HBASE-24696 > Project: HBase > Issue Type: Improvement > Components: UI >Reporter: Nick Dimiduk >Assignee: Mingliang Liu >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0, 2.1.10, 2.2.7 > > Attachments: Screen Shot 2020-07-17 at 10.55.56 PM.png > > > It's a small thing, but seems like an omission. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24763) Update 2.1 documentation to 2.1.9 and remove direct link from main page
Peter Somogyi created HBASE-24763: - Summary: Update 2.1 documentation to 2.1.9 and remove direct link from main page Key: HBASE-24763 URL: https://issues.apache.org/jira/browse/HBASE-24763 Project: HBase Issue Type: Task Components: documentation, website Reporter: Peter Somogyi Assignee: Peter Somogyi For the 2.1 release the available documentation points to 2.1.5. Since the release has reached EOL with 2.1.9 the documentation on the website should reflect this and a direct link to 2.1 should be removed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24696) Include JVM information on Web UI under "Software Attributes"
[ https://issues.apache.org/jira/browse/HBASE-24696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163850#comment-17163850 ] Peter Somogyi commented on HBASE-24696: --- Not directly but the [Download page|https://hbase.apache.org/downloads.html] lists only active releases and there is a [list of Release Managers|https://hbase.apache.org/book.html#_release_managers] for the active releases. When a release line reaches End of Life it is communicated in the user and dev mailing lists. > Include JVM information on Web UI under "Software Attributes" > - > > Key: HBASE-24696 > URL: https://issues.apache.org/jira/browse/HBASE-24696 > Project: HBase > Issue Type: Improvement > Components: UI >Reporter: Nick Dimiduk >Assignee: Mingliang Liu >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0, 2.1.10, 2.2.7 > > Attachments: Screen Shot 2020-07-17 at 10.55.56 PM.png > > > It's a small thing, but seems like an omission. -- This message was sent by Atlassian Jira (v8.3.4#803005)
