[jira] [Created] (HBASE-11172) Cancal a backup process
Demai Ni created HBASE-11172: Summary: Cancal a backup process Key: HBASE-11172 URL: https://issues.apache.org/jira/browse/HBASE-11172 Project: HBase Issue Type: New Feature Affects Versions: 0.99.0 Reporter: Demai Ni Fix For: 0.99.0 h2. Feature Description the jira is part of [HBASE-7912|https://issues.apache.org/jira/browse/HBASE-7912], and depend on full backup [HBASE-10900| https://issues.apache.org/jira/browse/HBASE-10900] and incremental backup [HBASE-11085| https://issues.apache.org/jira/browse/HBASE-11085]. for the detail layout and frame work, please reference to [HBASE-10900| https://issues.apache.org/jira/browse/HBASE-10900]. A backup operation may need to move handreds/thousands GB of data, and takes hours. Sometimes, the operation may take longer than the original maintenance time window planned by the administration. So it is necessary to have the functionality to cancel the operation and reset all the history/manifest info whenever necessary. so that we can have a clean backup in the next time window -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11134) Add a -list-snapshots option to SnapshotInfo
[
https://issues.apache.org/jira/browse/HBASE-11134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996125#comment-13996125
]
Lars Hofhansl commented on HBASE-11134:
---
+1 on v1
Add a -list-snapshots option to SnapshotInfo
Key: HBASE-11134
URL: https://issues.apache.org/jira/browse/HBASE-11134
Project: HBase
Issue Type: Improvement
Components: snapshots
Affects Versions: 0.99.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Trivial
Fix For: 0.99.0, 0.96.3, 0.94.20, 0.98.3
Attachments: HBASE-11134-v0.patch, HBASE-11134-v1.patch
Add a -list-snapshots option to SnapshotInfo to show all the snapshots
available. Also add a -remote-dir option to simplify the usage of
SnapshotInfo in case the snapshot dir is not the one of the current hbase
cluster
{code}
$ hbase org.apache.hadoop.hbase.snapshot.SnapshotInfo -list-snapshots
SNAPSHOT | CREATION TIME| TABLE NAME
foo | 2014-05-07T22:40:13 | testtb
bar | 2014-05-07T22:40:16 | testtb
$ hbase org.apache.hadoop.hbase.snapshot.SnapshotInfo -remote-dir
file:///backup/ -snapshot my_local_snapshot
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HBASE-11176) Make /src/main/xslt/configuration_to_docbook_section.xsl produce better Docbook
Misty Stanley-Jones created HBASE-11176: --- Summary: Make /src/main/xslt/configuration_to_docbook_section.xsl produce better Docbook Key: HBASE-11176 URL: https://issues.apache.org/jira/browse/HBASE-11176 Project: HBase Issue Type: Bug Affects Versions: 0.98.2 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones The file generates XML that is not valid, though it does build. As an example, glossentry is not allowed to have an ID, because it is an untitled element. Sub-tasks: 1. Research the best way to display the info 2. Fix up the XSLT transform accordingly. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11181) prune/delete old backups
Jerry He created HBASE-11181: Summary: prune/delete old backups Key: HBASE-11181 URL: https://issues.apache.org/jira/browse/HBASE-11181 Project: HBase Issue Type: New Feature Affects Versions: 0.99.0 Reporter: Jerry He Fix For: 0.99.0 Prune and delete old backups when they are no longer needed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10810) LoadTestTool should share the connection and connection pool
[
https://issues.apache.org/jira/browse/HBASE-10810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13998271#comment-13998271
]
Devaraj Das commented on HBASE-10810:
-
+1
LoadTestTool should share the connection and connection pool
Key: HBASE-10810
URL: https://issues.apache.org/jira/browse/HBASE-10810
Project: HBase
Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: hbase-10070
Attachments: hbase-10810_v1.patch
While running the IT test from HBASE-10572, we've noticed that the number of
threads jumps to 4K's when CM actions are going on.
Our [~ndimiduk] summarizes the problem quite good:
MultiThreadedReader creates this pool for each HTable:
{code}
ThreadPoolExecutor pool = new ThreadPoolExecutor(1, maxThreads,
keepAliveTime, TimeUnit.SECONDS,
new SynchronousQueueRunnable(),
Threads.newDaemonThreadFactory(htable));
{code}
This comes from the HTable creation
{code}
public HTable(Configuration conf, final TableName tableName)
{code}
As well the javadoc says Recommended.
This is wrong.
In this issue we can change the LTT sub classes to use the shared connection
object and initialize their tables using HConnection.getTable() rather than
new HTable().
This is relevant to trunk as well, but there since there is only one
outstanding RPC per thread, it is not such a big problem.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-6990) Pretty print TTL
[ https://issues.apache.org/jira/browse/HBASE-6990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997894#comment-13997894 ] Esteban Gutierrez commented on HBASE-6990: -- Sorry [~stacke], this was meant for [~stack] Apache JIRA is acting slow. Pretty print TTL Key: HBASE-6990 URL: https://issues.apache.org/jira/browse/HBASE-6990 Project: HBase Issue Type: Improvement Reporter: Jean-Daniel Cryans Assignee: Esteban Gutierrez Priority: Minor Attachments: HBASE-6990.v0.patch, HBASE-6990.v1.patch, HBASE-6990.v2.patch, HBASE-6990.v3.patch I've seen a lot of users getting confused by the TTL configuration and I think that if we just pretty printed it it would solve most of the issues. For example, let's say a user wanted to set a TTL of 90 days. That would be 7776000. But let's say that it was typo'd to 7776 instead, it gives you 900 days! So when we print the TTL we could do something like x days, x hours, x minutes, x seconds (real_ttl_value). This would also help people when they use ms instead of seconds as they would see really big values in there. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11054) Create new hook in StoreScanner to help user creating his own delete tracker
[ https://issues.apache.org/jira/browse/HBASE-11054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13997736#comment-13997736 ] stack commented on HBASE-11054: --- We breaking anyone? Phoenix? I'd say policy should be no changes to method apis across minor versions? Create new hook in StoreScanner to help user creating his own delete tracker Key: HBASE-11054 URL: https://issues.apache.org/jira/browse/HBASE-11054 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 0.99.0, 0.98.2 Attachments: HBASE-11054.patch, HBASE-11054_1.patch In case of Visibility labels, to support deletes we need to consider the visibility expression per version inorder to identify the matching visibility expression. Tracking the visibility tags along with the delete tracker would make things easier rather than handling during scans/compactions. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11171) More doc improvements on block cache options
[ https://issues.apache.org/jira/browse/HBASE-11171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-11171: -- Issue Type: Sub-task (was: Bug) Parent: HBASE-10403 More doc improvements on block cache options Key: HBASE-11171 URL: https://issues.apache.org/jira/browse/HBASE-11171 Project: HBase Issue Type: Sub-task Components: documentation Reporter: stack Assignee: stack Attachments: 11171.txt, 11171v2.txt, package-summary.html I have more doc improvement (no code changes for sure this time). Follow on from HBASE-11098. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HBASE-11145) Issue with HLog sync
[
https://issues.apache.org/jira/browse/HBASE-11145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John reassigned HBASE-11145:
--
Assignee: stack
Assigning to Stack as per our offline discussion.
Issue with HLog sync
Key: HBASE-11145
URL: https://issues.apache.org/jira/browse/HBASE-11145
Project: HBase
Issue Type: Bug
Reporter: Anoop Sam John
Assignee: stack
Fix For: 0.99.0
Got the below Exceptions Log in case of a write heavy test
{code}
2014-05-07 11:29:56,417 ERROR [main.append-pool1-t1]
wal.FSHLog$RingBufferEventHandler(1882): UNEXPECTED!!!
java.lang.IllegalStateException: Queue full
at java.util.AbstractQueue.add(Unknown Source)
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.offer(FSHLog.java:1227)
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1878)
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:133)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
2014-05-07 11:29:56,418 ERROR [main.append-pool1-t1]
wal.FSHLog$RingBufferEventHandler(1882): UNEXPECTED!!!
java.lang.ArrayIndexOutOfBoundsException: 5
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1838)
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:133)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
2014-05-07 11:29:56,419 ERROR [main.append-pool1-t1]
wal.FSHLog$RingBufferEventHandler(1882): UNEXPECTED!!!
java.lang.ArrayIndexOutOfBoundsException: 6
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1838)
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:133)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
2014-05-07 11:29:56,419 ERROR [main.append-pool1-t1]
wal.FSHLog$RingBufferEventHandler(1882): UNEXPECTED!!!
java.lang.ArrayIndexOutOfBoundsException: 7
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1838)
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:133)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
{code}
In FSHLog$SyncRunner.offer we do BlockingQueue.add() which throws Exception
as it is full. The problem is the below shown catch() we do not do any
cleanup.
{code}
this.syncRunners[index].offer(sequence, this.syncFutures,
this.syncFuturesCount);
attainSafePoint(sequence);
this.syncFuturesCount = 0;
} catch (Throwable t) {
LOG.error(UNEXPECTED!!!, t);
}
{code}
syncFuturesCount is not getting reset to 0 and so the subsequent onEvent()
handling throws ArrayIndexOutOfBoundsException.
I think we should do the below
1. Handle the Exception and call cleanupOutstandingSyncsOnException() as in
other cases of Exception handling
2. Instead of BlockingQueue.add() use offer() (?)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11143) ageOfLastShippedOp metric is confusing
[ https://issues.apache.org/jira/browse/HBASE-11143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-11143: -- Summary: ageOfLastShippedOp metric is confusing (was: ageOfLastShippedOp confusing) ageOfLastShippedOp metric is confusing -- Key: HBASE-11143 URL: https://issues.apache.org/jira/browse/HBASE-11143 Project: HBase Issue Type: Bug Components: Replication Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.94.20 Attachments: 11143-0.94.txt We are trying to report on replication lag and find that there is no good single metric to do that. ageOfLastShippedOp is close, but unfortunately it is increased even when there is nothing to ship on a particular RegionServer. I would like discuss a few options here: Add a new metric: replicationQueueTime (or something) with the above meaning. I.e. if we have something to ship we set the age of that last shipped edit, if we fail we increment that last time (just like we do now). But if there is nothing to replicate we set it to current time (and hence that metric is reported to close to 0). Alternatively we could change the meaning of ageOfLastShippedOp to mean to do that. That might lead to surprises, but the current behavior is clearly weird when there is nothing to replicate. Comments? [~jdcryans], [~stack]. If approach sounds good, I'll make a patch for all branches. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-3696) No durability when running with LocalFileSystem
[ https://issues.apache.org/jira/browse/HBASE-3696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated HBASE-3696: --- Assignee: (was: Harsh J) Status: Open (was: Patch Available) No durability when running with LocalFileSystem --- Key: HBASE-3696 URL: https://issues.apache.org/jira/browse/HBASE-3696 Project: HBase Issue Type: Bug Components: documentation, wal Reporter: Todd Lipcon LocalFileSystem in Hadoop doesn't currently implement sync(), so when we're running in that case, we don't have any durability. This isn't a huge deal since it isn't a realistic deployment scenario, but it's probably worth documenting. It caused some confusion for a user when a table disappeared after killing a standalone instance that was hosting its data in the local FS. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11137) Add mapred.TableSnapshotInputFormat
[ https://issues.apache.org/jira/browse/HBASE-11137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-11137: - Attachment: HBASE-11137.02.patch HBASE-11137.02-0.98.patch Attaching what was committed. Add mapred.TableSnapshotInputFormat --- Key: HBASE-11137 URL: https://issues.apache.org/jira/browse/HBASE-11137 Project: HBase Issue Type: Improvement Components: mapreduce, Performance Affects Versions: 0.98.0, 0.96.2 Reporter: Nick Dimiduk Assignee: Nick Dimiduk Fix For: 0.99.0, 0.98.3 Attachments: HBASE-11137.00-0.98.patch, HBASE-11137.00.patch, HBASE-11137.01.patch, HBASE-11137.01_rerun.patch, HBASE-11137.02-0.98.patch, HBASE-11137.02.patch We should have feature parity between mapreduce and mapred implementations. This is important for Hive. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11172) Cancel a backup process
[ https://issues.apache.org/jira/browse/HBASE-11172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-11172: --- Summary: Cancel a backup process (was: Cancal a backup process ) Cancel a backup process Key: HBASE-11172 URL: https://issues.apache.org/jira/browse/HBASE-11172 Project: HBase Issue Type: New Feature Affects Versions: 0.99.0 Reporter: Demai Ni Fix For: 0.99.0 h2. Feature Description the jira is part of [HBASE-7912|https://issues.apache.org/jira/browse/HBASE-7912], and depend on full backup [HBASE-10900| https://issues.apache.org/jira/browse/HBASE-10900] and incremental backup [HBASE-11085| https://issues.apache.org/jira/browse/HBASE-11085]. for the detail layout and frame work, please reference to [HBASE-10900| https://issues.apache.org/jira/browse/HBASE-10900]. A backup operation may need to move handreds/thousands GB of data, and takes hours. Sometimes, the operation may take longer than the original maintenance time window planned by the administration. So it is necessary to have the functionality to cancel the operation and reset all the history/manifest info whenever necessary. so that we can have a clean backup in the next time window -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11096) stop method of Master and RegionServer coprocessor is not invoked
[ https://issues.apache.org/jira/browse/HBASE-11096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qiang Tian updated HBASE-11096: --- Status: Patch Available (was: Open) stop method of Master and RegionServer coprocessor is not invoked -- Key: HBASE-11096 URL: https://issues.apache.org/jira/browse/HBASE-11096 Project: HBase Issue Type: Bug Affects Versions: 0.94.19, 0.98.1, 0.96.2 Reporter: Qiang Tian Assignee: Qiang Tian Fix For: 0.99.0, 0.96.3, 0.94.20, 0.98.3 Attachments: HBASE-11096-0.94.patch, HBASE-11096-0.96.patch, HBASE-11096-0.98.patch, HBASE-11096-trunk-v0.patch, HBASE-11096-trunk-v0.patch, HBASE-11096-trunk-v1.patch, HBASE-11096-trunk-v2.patch, HBASE-11096-trunk-v3.patch the stop method of coprocessor specified by hbase.coprocessor.master.classes and hbase.coprocessor.regionserver.classes is not invoked. If coprocessor allocates OS resources, it could cause master/regionserver resource leak or hang during exit. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10504) Define Replication Interface
[ https://issues.apache.org/jira/browse/HBASE-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13999301#comment-13999301 ] Enis Soztutar commented on HBASE-10504: --- bq. This seems to me like work that is difficult to justify for the SEP because it makes use of existing HBase RPC and replication failover handling, and currently works Of course RPC and RS discovery etc are internal to HBase. The main advantage would be that you will be protected against changes there. bq. Obviously these comments are purely from the standpoint of SEP maintenance. As I mentioned in a previous comment, I think that this proposal would be a great addition to HBase and very useful – I just don't see it's usefulness to the SEP yet. Fair enough. HBase's own inner workings for inter-cluster replication is not changing with this patch, so you should be able to have SEP as it is. I think I don't fully understand why would you want to have a proxy layer as a separate service (other than isolation issues). Define Replication Interface Key: HBASE-10504 URL: https://issues.apache.org/jira/browse/HBASE-10504 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Priority: Blocker Fix For: 0.99.0 Attachments: hbase-10504_wip1.patch HBase has replication. Fellas have been hijacking the replication apis to do all kinds of perverse stuff like indexing hbase content (hbase-indexer https://github.com/NGDATA/hbase-indexer) and our [~toffer] just showed up w/ overrides that replicate via an alternate channel (over a secure thrift channel between dcs over on HBASE-9360). This issue is about surfacing these APIs as public with guarantees to downstreamers similar to those we have on our public client-facing APIs (and so we don't break them for downstreamers). Any input [~phunt] or [~gabriel.reid] or [~toffer]? Thanks. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11123) Upgrade instructions from 0.94 to 0.98
[ https://issues.apache.org/jira/browse/HBASE-11123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13999379#comment-13999379 ] Hudson commented on HBASE-11123: ABORTED: Integrated in HBase-TRUNK #5132 (See [https://builds.apache.org/job/HBase-TRUNK/5132/]) HBASE-11123 Upgrade instructions from 0.94 to 0.98 (Misty Stanley-Jones) (stack: rev 1594360) * /hbase/trunk/src/main/docbkx/upgrading.xml Upgrade instructions from 0.94 to 0.98 -- Key: HBASE-11123 URL: https://issues.apache.org/jira/browse/HBASE-11123 Project: HBase Issue Type: Improvement Components: documentation Affects Versions: 0.98.2 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Priority: Minor Fix For: 0.99.0 Attachments: HBASE-11123-1.patch, HBASE-11123.patch I cloned this from the 0.96 upgrade docs task. It was suggested that we need upgrade instructions from 0.94 to 0.98. I will need source material to even prioritize this. Assuming this is Minor. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10212) New rpc metric: number of active handler
[ https://issues.apache.org/jira/browse/HBASE-10212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13999363#comment-13999363 ] Hudson commented on HBASE-10212: ABORTED: Integrated in HBase-TRUNK #5132 (See [https://builds.apache.org/job/HBase-TRUNK/5132/]) Amend HBASE-10561 Forward port: HBASE-10212 New rpc metric: number of active handler (liangxie: rev 1594126) * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSource.java * /hbase/trunk/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSourceImpl.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerWrapperStub.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestRpcMetrics.java HBASE-10561 Forward port: HBASE-10212 New rpc metric: number of active handler (liangxie: rev 1594117) * /hbase/trunk/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerWrapper.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/FifoRpcScheduler.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerWrapperImpl.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcScheduler.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerWrapperStub.java New rpc metric: number of active handler Key: HBASE-10212 URL: https://issues.apache.org/jira/browse/HBASE-10212 Project: HBase Issue Type: Improvement Components: IPC/RPC Reporter: Chao Shi Assignee: Chao Shi Fix For: 0.94.17 Attachments: hbase-10212.patch The attached patch adds a new metric: number of active handler threads. We found this is a good metric to measure how busy of a server. If this number is too high (compared to the total number of handlers), the server has risks in getting call queue full. We used to monitor # reads or # writes. However we found this often produce false alerts, because a read touching HDFS will produce much high workload than a block-cached read. The attached patch is based on our internal 0.94 branch, but I think it pretty easy to port to rebase to other branches if you think it is useful. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11171) More doc improvements on block cache options
[ https://issues.apache.org/jira/browse/HBASE-11171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-11171: -- Attachment: 11171v4.txt Fix conflict More doc improvements on block cache options Key: HBASE-11171 URL: https://issues.apache.org/jira/browse/HBASE-11171 Project: HBase Issue Type: Sub-task Components: documentation Reporter: stack Assignee: stack Attachments: 11171.txt, 11171v2.txt, 11171v2.txt, 11171v3.txt, 11171v4.txt, 11171v5.txt, 11171v6.txt, 11171v7.txt, 11171v8.txt, package-summary.html I have more doc improvement (no code changes for sure this time). Follow on from HBASE-11098. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11167) Avoid usage of java.rmi package Exception in MemStore
[
https://issues.apache.org/jira/browse/HBASE-11167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13999561#comment-13999561
]
Hadoop QA commented on HBASE-11167:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12644785/HBASE-11167.patch
against trunk revision .
ATTACHMENT ID: 12644785
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
58 warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9520//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9520//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9520//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9520//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9520//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9520//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9520//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9520//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9520//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9520//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9520//console
This message is automatically generated.
Avoid usage of java.rmi package Exception in MemStore
-
Key: HBASE-11167
URL: https://issues.apache.org/jira/browse/HBASE-11167
Project: HBase
Issue Type: Improvement
Components: regionserver
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 0.99.0
Attachments: HBASE-11167.patch
This Exception was in use already. While making MemStore into Interface I
have not looked closely into it (whether class from RMI or not). So it went
into Interface as well.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11151) move tracing modules from hbase-server to hbase-common
[ https://issues.apache.org/jira/browse/HBASE-11151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13999382#comment-13999382 ] Hudson commented on HBASE-11151: ABORTED: Integrated in HBase-TRUNK #5132 (See [https://builds.apache.org/job/HBase-TRUNK/5132/]) HBASE-11151 move tracing modules from hbase-server to hbase-common (Masatake Iwasaki) (stack: rev 1594110) * /hbase/trunk/hbase-common/pom.xml * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/trace * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/trace/HBaseHTraceConfiguration.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/trace/SpanReceiverHost.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/trace * /hbase/trunk/src/main/docbkx/tracing.xml move tracing modules from hbase-server to hbase-common -- Key: HBASE-11151 URL: https://issues.apache.org/jira/browse/HBASE-11151 Project: HBase Issue Type: Improvement Components: documentation, util Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki Priority: Minor Fix For: 0.99.0 Attachments: HBASE-11151-0.patch Not only servers but also clients using tracing needs SpanReceiverHost. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11155) Fix Validation Errors in Ref Guide
[
https://issues.apache.org/jira/browse/HBASE-11155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13999421#comment-13999421
]
Hadoop QA commented on HBASE-11155:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12644955/HBASE-11155.patch
against trunk revision .
ATTACHMENT ID: 12644955
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
58 warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+listitemparahbase - system namespace, used to contain
hbase internal tables/para/listitem
+ indextermprimarycolumn family
qualifier/primarysecondaryColumn Family Qualifier/secondary/indexterm.
+paraSee link
xlink:href=http://hbase.apache.org/acid-semantics.html;ACID Semantics/link.
+link
xlink:href=http://hadoop-hbase.blogspot.com/2012/03/acid-in-hbase.html;ACID
in HBase/link./para
+ paraAn explanation is required of what
classnameTableMapReduceUtil/classname is doing, especially with the reducer.
+ These could be set by the programmer on the job and conf, but
classnameTableMapReduceUtil/classname tries to make things easier./para
+As for the Reducer, it is a generic Reducer instead of extending
TableMapper and emitting Puts./para
+ listitemparaStrongly consistent reads/writes: HBase is not
an eventually consistent DataStore. This
+ makes it very suitable for tasks such as high-speed counter
aggregation./para /listitem
+ listitemparaAutomatic sharding: HBase tables are
distributed on the cluster via regions, and regions are
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.mapreduce.TestTableMapReduceBase.testMultiRegionTable(TestTableMapReduceBase.java:96)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9515//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9515//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9515//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9515//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9515//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9515//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9515//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9515//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9515//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9515//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9515//console
This message is automatically generated.
Fix Validation Errors in Ref Guide
--
Key: HBASE-11155
URL: https://issues.apache.org/jira/browse/HBASE-11155
Project: HBase
Issue Type: Bug
Components: documentation
Affects Versions: 0.98.2
Reporter: Misty Stanley-Jones
Assignee: Misty Stanley-Jones
Attachments: HBASE-11155.patch
Before I do serious documentation work, I have to fix all of the validation
errors that are somehow not causing the Ref Guide to break the builds. I will
attach one patch per file -- that's the easiest way I know how to do it. I
will try not to make any content
[jira] [Commented] (HBASE-11158) bin/hbase upgrade -check should also check compression codecs
[ https://issues.apache.org/jira/browse/HBASE-11158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13998690#comment-13998690 ] Jean-Marc Spaggiari commented on HBASE-11158: - Ok. I thought about it a bit and it's not the best way to solve this. There is already a command to validate compression codecs. So I think a better way to inform user about this is to list the codes used when checking the upgrade, and document in the upgrade process to recommend to check the available codecs when upgrade to a new cluster by pointing to the right documentation. If you are fine with this approach I will close this JIRA, open one to add the codecs dump in the -check portion, and another to update the documentation. bin/hbase upgrade -check should also check compression codecs - Key: HBASE-11158 URL: https://issues.apache.org/jira/browse/HBASE-11158 Project: HBase Issue Type: Bug Affects Versions: 0.99.0 Reporter: Jean-Marc Spaggiari When upgrading insitu from 0.94 to 0.96 or 0.98 codecs are usually already there in the servers so it's all fine. But when doing an upgrade by moving the data from one cluster to a brand new one, compression codecs might be missing. bin/hbase upgrade -check will not report any missing codec, but HBase will not be able to start after the upgrade because the codes are missing. I think bin/hbase upgrade -check should check the compression codecs configured on the tables and make sure they are available on the new cluster. f not, it should be reported. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11137) Add mapred.TableSnapshotInputFormat
[ https://issues.apache.org/jira/browse/HBASE-11137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13999364#comment-13999364 ] Hudson commented on HBASE-11137: ABORTED: Integrated in HBase-TRUNK #5132 (See [https://builds.apache.org/job/HBase-TRUNK/5132/]) HBASE-11137 Add mapred.TableSnapshotInputFormat (ndimiduk: rev 1594982) * /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/mapreduce/IntegrationTestTableSnapshotInputFormat.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableSnapshotInputFormat.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormat.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormatImpl.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapred/TestTableSnapshotInputFormat.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormatTestBase.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableSnapshotInputFormat.java Add mapred.TableSnapshotInputFormat --- Key: HBASE-11137 URL: https://issues.apache.org/jira/browse/HBASE-11137 Project: HBase Issue Type: Improvement Components: mapreduce, Performance Affects Versions: 0.98.0, 0.96.2 Reporter: Nick Dimiduk Assignee: Nick Dimiduk Fix For: 0.99.0, 0.98.3 Attachments: HBASE-11137.00-0.98.patch, HBASE-11137.00.patch, HBASE-11137.01.patch, HBASE-11137.01_rerun.patch, HBASE-11137.02-0.98.patch, HBASE-11137.02.patch We should have feature parity between mapreduce and mapred implementations. This is important for Hive. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-9345) Add support for specifying filters in scan
[ https://issues.apache.org/jira/browse/HBASE-9345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-9345: Assignee: Virag Kothari (was: Vandana Ayyalasomayajula) Add support for specifying filters in scan -- Key: HBASE-9345 URL: https://issues.apache.org/jira/browse/HBASE-9345 Project: HBase Issue Type: Improvement Components: REST Affects Versions: 0.94.11 Reporter: Vandana Ayyalasomayajula Assignee: Virag Kothari Priority: Minor Attachments: HBASE-9345_trunk.patch, HBASE-9345_trunk.patch In the implementation of stateless scanner from HBase-9343, the support for specifying filters is missing. This JIRA aims to implement support for filter specification. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11185) Parallelize Snapshot operations
[ https://issues.apache.org/jira/browse/HBASE-11185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-11185: Status: Patch Available (was: Open) Parallelize Snapshot operations --- Key: HBASE-11185 URL: https://issues.apache.org/jira/browse/HBASE-11185 Project: HBase Issue Type: Bug Components: snapshots Affects Versions: 0.99.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 0.99.0 Attachments: HBASE-11185-v0.patch when SnapshotInfo or snapshot verification is executed against a remote path, it may takes a while since all the code is mainly composed by sequential calls to the fs. This patch will parallelize all the snapshot operations using a thread pool to dispatch requests. The size of the pool is tunable by using hbase.snapshot.thread.pool.max -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10835) DBE encode path improvements
[ https://issues.apache.org/jira/browse/HBASE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-10835: --- Status: Patch Available (was: Open) DBE encode path improvements Key: HBASE-10835 URL: https://issues.apache.org/jira/browse/HBASE-10835 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10835.patch, HBASE-10835_V2.patch, HBASE-10835_V3.patch, HBASE-10835_V4.patch Here 1st we write KVs (Cells) into a buffer and then passed to DBE encoder. Encoder again reads kvs one by one from the buffer and encodes and creates a new buffer. There is no need to have this model now. Previously we had option of no encode in disk and encode only in cache. At that time the read buffer from a HFile block was passed to this and encodes. So encode cell by cell can be done now. Making this change will need us to have a NoOp DBE impl which just do the write of a cell as it is with out any encoding. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10885) Support visibility expressions on Deletes
[
https://issues.apache.org/jira/browse/HBASE-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13999717#comment-13999717
]
ramkrishna.s.vasudevan commented on HBASE-10885:
{code}
58 warnings
[WARNING] Javadoc Warnings
[WARNING] javadoc: warning - Multiple sources of package comments found for
package org.apache.hadoop.hbase.io.hfile
[WARNING]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java:120:
warning - Tag @link: reference not found: BUCKET_CACHE_COMBINED_KEY
[WARNING]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java:88:
warning - Tag @link: reference not found: BUCKET_CACHE_SIZE_KEY
[WARNING]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java:99:
warning - Tag @link: reference not found: BUCKET_CACHE_COMBINED_KEY
[WARNING]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java:99:
warning - Tag @link: reference not found: BUCKET_CACHE_COMBINED_PERCENTAGE_KEY
[WARNING]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java:120:
warning - Tag @link: reference not found: BUCKET_CACHE_COMBINED_KEY
[WARNING]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CombinedBlockCache.java:40:
warning - Missing closing '}' character for inline tag: {@link
#getBlock(BlockCacheKey, boolean, boolean) reads
{code}
This is not part of the patch.
Support visibility expressions on Deletes
-
Key: HBASE-10885
URL: https://issues.apache.org/jira/browse/HBASE-10885
Project: HBase
Issue Type: Improvement
Affects Versions: 0.98.1
Reporter: Andrew Purtell
Assignee: ramkrishna.s.vasudevan
Priority: Blocker
Fix For: 0.99.0, 0.98.3
Attachments: HBASE-10885_1.patch, HBASE-10885_2.patch,
HBASE-10885_new_tag_type_1.patch, HBASE-10885_new_tag_type_2.patch
Accumulo can specify visibility expressions for delete markers. During
compaction the cells covered by the tombstone are determined in part by
matching the visibility expression. This is useful for the use case of data
set coalescing, where entries from multiple data sets carrying different
labels are combined into one common large table. Later, a subset of entries
can be conveniently removed using visibility expressions.
Currently doing the same in HBase would only be possible with a custom
coprocessor. Otherwise, a Delete will affect all cells covered by the
tombstone regardless of any visibility expression scoping. This is correct
behavior in that no data spill is possible, but certainly could be
surprising, and is only meant to be transitional. We decided not to support
visibility expressions on Deletes to control the complexity of the initial
implementation.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11134) Add a -list-snapshots option to SnapshotInfo
[
https://issues.apache.org/jira/browse/HBASE-11134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13999376#comment-13999376
]
Hudson commented on HBASE-11134:
FAILURE: Integrated in HBase-0.94-on-Hadoop-2 #84 (See
[https://builds.apache.org/job/HBase-0.94-on-Hadoop-2/84/])
HBASE-11134 Add a -list-snapshots option to SnapshotInfo (mbertozzi: rev
1594857)
*
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
Add a -list-snapshots option to SnapshotInfo
Key: HBASE-11134
URL: https://issues.apache.org/jira/browse/HBASE-11134
Project: HBase
Issue Type: Improvement
Components: snapshots
Affects Versions: 0.99.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Trivial
Fix For: 0.99.0, 0.96.3, 0.94.20, 0.98.3
Attachments: HBASE-11134-v0.patch, HBASE-11134-v1.patch
Add a -list-snapshots option to SnapshotInfo to show all the snapshots
available. Also add a -remote-dir option to simplify the usage of
SnapshotInfo in case the snapshot dir is not the one of the current hbase
cluster
{code}
$ hbase org.apache.hadoop.hbase.snapshot.SnapshotInfo -list-snapshots
SNAPSHOT | CREATION TIME| TABLE NAME
foo | 2014-05-07T22:40:13 | testtb
bar | 2014-05-07T22:40:16 | testtb
$ hbase org.apache.hadoop.hbase.snapshot.SnapshotInfo -remote-dir
file:///backup/ -snapshot my_local_snapshot
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11137) Add mapred.TableSnapshotInputFormat
[ https://issues.apache.org/jira/browse/HBASE-11137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-11137: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to 0.98 and trunk. Thanks for the reviews [~stack], [~enis]. Add mapred.TableSnapshotInputFormat --- Key: HBASE-11137 URL: https://issues.apache.org/jira/browse/HBASE-11137 Project: HBase Issue Type: Improvement Components: mapreduce, Performance Affects Versions: 0.98.0, 0.96.2 Reporter: Nick Dimiduk Assignee: Nick Dimiduk Fix For: 0.99.0, 0.98.3 Attachments: HBASE-11137.00-0.98.patch, HBASE-11137.00.patch, HBASE-11137.01.patch, HBASE-11137.01_rerun.patch, HBASE-11137.02-0.98.patch, HBASE-11137.02.patch We should have feature parity between mapreduce and mapred implementations. This is important for Hive. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10504) Define Replication Interface
[
https://issues.apache.org/jira/browse/HBASE-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13998525#comment-13998525
]
Gabriel Reid commented on HBASE-10504:
--
{quote}Even in that case, the server-client implementation will be much simpler
than what you have in SEP I think, which does hbase RPC + zookeeper + zk
mockup. Only a server discovery + simple one method thrift / PB RPC is what is
needed from my understanding. It would be great to have a skeleton for this
living in HBase proper as well.{quote}
The SEP does indeed make use of HBase RPC and ZK, and does masquerades in ZK as
a region server, but I don't see that much of an advantage of doing a separate
thrift or PB-based RPC. I think that this would just be exchanging one existing
RPC mechanism (hbase) for another (thrift/PB). The service discovery would
still be required, which would also involve ZK. Not having to do the setup of
masquerading as a regionserver would indeed be a plus.
Taking this road for the SEP would then require:
* defining/setting up RPC (which is currently already provided by hbase
* implementing a ReplicationConsumer (and deploying and configuring it in hbase
RS) that does service discovery, RPC endpoint selection, and properly deals
with failing/flaky RPC endpoints (also already provided in hbase replication)
* reworking the SEP RPC server to use the new RPC mechanism
This seems to me like work that is difficult to justify for the SEP because it
makes use of existing HBase RPC and replication failover handling, and
currently works. I may very well be missing something, but I don't see how
reimplementing this stuff in the SEP would simplify things very much.
The one big potential advantage that I do see is that it would protect the SEP
from changes in the internal HBase replication interfaces.
Obviously these comments are purely from the standpoint of SEP maintenance. As
I mentioned in a previous comment, I think that this proposal would be a great
addition to HBase and very useful -- I just don't see it's usefulness to the
SEP yet.
Define Replication Interface
Key: HBASE-10504
URL: https://issues.apache.org/jira/browse/HBASE-10504
Project: HBase
Issue Type: Task
Reporter: stack
Assignee: stack
Priority: Blocker
Fix For: 0.99.0
Attachments: hbase-10504_wip1.patch
HBase has replication. Fellas have been hijacking the replication apis to do
all kinds of perverse stuff like indexing hbase content (hbase-indexer
https://github.com/NGDATA/hbase-indexer) and our [~toffer] just showed up w/
overrides that replicate via an alternate channel (over a secure thrift
channel between dcs over on HBASE-9360). This issue is about surfacing these
APIs as public with guarantees to downstreamers similar to those we have on
our public client-facing APIs (and so we don't break them for downstreamers).
Any input [~phunt] or [~gabriel.reid] or [~toffer]?
Thanks.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11171) More doc improvements on block cache options
[ https://issues.apache.org/jira/browse/HBASE-11171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-11171: -- Attachment: 11171v6.txt More doc improvements on block cache options Key: HBASE-11171 URL: https://issues.apache.org/jira/browse/HBASE-11171 Project: HBase Issue Type: Sub-task Components: documentation Reporter: stack Assignee: stack Attachments: 11171.txt, 11171v2.txt, 11171v2.txt, 11171v3.txt, 11171v4.txt, 11171v5.txt, 11171v6.txt, 11171v7.txt, 11171v8.txt, package-summary.html I have more doc improvement (no code changes for sure this time). Follow on from HBASE-11098. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11168) [docs] Remove references to RowLocks in post 0.96 docs.
[
https://issues.apache.org/jira/browse/HBASE-11168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13999191#comment-13999191
]
Hadoop QA commented on HBASE-11168:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12644843/hbase-11168.patch
against trunk revision .
ATTACHMENT ID: 12644843
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
58 warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9513//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9513//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9513//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9513//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9513//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9513//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9513//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9513//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9513//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9513//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9513//console
This message is automatically generated.
[docs] Remove references to RowLocks in post 0.96 docs.
---
Key: HBASE-11168
URL: https://issues.apache.org/jira/browse/HBASE-11168
Project: HBase
Issue Type: Bug
Components: documentation
Affects Versions: 0.95.0
Reporter: Jonathan Hsieh
Assignee: Jonathan Hsieh
Fix For: 0.99.0, 0.96.3, 0.98.3
Attachments: hbase-11168.patch
Row locks were removed in 0.95 by HBASE-7315 / HBASE-2332. There are a few
vestiges of them in the docs. Remove.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11171) More doc improvements on block cache options
[ https://issues.apache.org/jira/browse/HBASE-11171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-11171: -- Attachment: 11171v3.txt More doc again in the package-info file. Thanks for the +1 Nick. Let me run tests locally to make sure all pass then apply. More doc improvements on block cache options Key: HBASE-11171 URL: https://issues.apache.org/jira/browse/HBASE-11171 Project: HBase Issue Type: Sub-task Components: documentation Reporter: stack Assignee: stack Attachments: 11171.txt, 11171v2.txt, 11171v2.txt, 11171v3.txt, 11171v4.txt, 11171v5.txt, 11171v6.txt, 11171v7.txt, 11171v8.txt, package-summary.html I have more doc improvement (no code changes for sure this time). Follow on from HBASE-11098. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10504) Define Replication Interface
[
https://issues.apache.org/jira/browse/HBASE-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13999804#comment-13999804
]
Gabriel Reid commented on HBASE-10504:
--
{quote}I think I don't fully understand why would you want to have a proxy
layer as a separate service (other than isolation issues).{quote}
Process isolation and deployment isolation are indeed the two main issues. The
process isolation side of things probably speaks for itself. Deployment
isolation is another issue however. Keeping dependencies of hbase-indexer in
line with all deps in hbase is an issue (as all hbase-indexer jars would need
to be deployed into the HBase lib for this approach to work), and all
configuration, config changes, and restarting would also become directly linked
to HBase.
Define Replication Interface
Key: HBASE-10504
URL: https://issues.apache.org/jira/browse/HBASE-10504
Project: HBase
Issue Type: Task
Reporter: stack
Assignee: stack
Priority: Blocker
Fix For: 0.99.0
Attachments: hbase-10504_wip1.patch
HBase has replication. Fellas have been hijacking the replication apis to do
all kinds of perverse stuff like indexing hbase content (hbase-indexer
https://github.com/NGDATA/hbase-indexer) and our [~toffer] just showed up w/
overrides that replicate via an alternate channel (over a secure thrift
channel between dcs over on HBASE-9360). This issue is about surfacing these
APIs as public with guarantees to downstreamers similar to those we have on
our public client-facing APIs (and so we don't break them for downstreamers).
Any input [~phunt] or [~gabriel.reid] or [~toffer]?
Thanks.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11137) Add mapred.TableSnapshotInputFormat
[
https://issues.apache.org/jira/browse/HBASE-11137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13994039#comment-13994039
]
Nick Dimiduk commented on HBASE-11137:
--
Attaching patch for trunk. Will post on RB once it's back up. Starting on a
backport for 0.98. [~stack]: you want this for 0.96 as well?
This patch extracts implementation into TableSnapshotInputFormatBase.
{{mapred}} and {{mapreduce}} implementations delegate functions to that
implementation. Test is similarly structured. One big difference I noticed is
that {{mapred}} does not support providing a {{Scan}} for limiting the data
processed by mappers. For this reason, {{mapred.TestTableSnapshotInputFormat}}
tests deviate slightly from the base. That functionality should be added in a
followup ticket.
ping [~enis], [~lhofhansl], [~stack], [~apurtell].
Add mapred.TableSnapshotInputFormat
---
Key: HBASE-11137
URL: https://issues.apache.org/jira/browse/HBASE-11137
Project: HBase
Issue Type: Improvement
Components: mapreduce, Performance
Affects Versions: 0.98.0, 0.96.2
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk
Attachments: HBASE-11137.00.patch
We should have feature parity between mapreduce and mapred implementations.
This is important for Hive.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10936) Add zeroByte encoding test
[ https://issues.apache.org/jira/browse/HBASE-10936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13999396#comment-13999396 ] Hudson commented on HBASE-10936: FAILURE: Integrated in HBase-0.94 #1367 (See [https://builds.apache.org/job/HBase-0.94/1367/]) HBASE-10936 Add zeroByte encoding test. (larsh: rev 1594139) * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/encoding/TestDataBlockEncoders.java Add zeroByte encoding test -- Key: HBASE-10936 URL: https://issues.apache.org/jira/browse/HBASE-10936 Project: HBase Issue Type: Sub-task Components: test Reporter: Lars Hofhansl Priority: Minor Fix For: 0.96.3, 0.94.20 Attachments: 10936-0.94.txt, 10936-0.96.txt, 10936-0.98.txt -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (HBASE-11158) bin/hbase upgrade -check should also check compression codecs
[
https://issues.apache.org/jira/browse/HBASE-11158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13998293#comment-13998293
]
Jean-Marc Spaggiari edited comment on HBASE-11158 at 5/15/14 11:28 AM:
---
{quote}
It can't check the whole cluster though JMS, right?
{quote}
I agree. Just trying to find a way to bring the alert back to the user without
having to check the whole cluster. I have not found any good idea to achieve
that for now. But I got caught already twice because of that, so trying to
figure if there is anything doable to alert user about it. Maybe with another
command line? Still thinking about it.
was (Author: jmspaggi):
[quote]
It can't check the whole cluster though JMS, right?
[quote]
I agree. Just trying to find a way to bring the alert back to the user without
having to check the whole cluster. I have not found any good idea to achieve
that for now. But I got caught already twice because of that, so trying to
figure if there is anything doable to alert user about it. Maybe with another
command line? Still thinking about it.
bin/hbase upgrade -check should also check compression codecs
-
Key: HBASE-11158
URL: https://issues.apache.org/jira/browse/HBASE-11158
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0
Reporter: Jean-Marc Spaggiari
When upgrading insitu from 0.94 to 0.96 or 0.98 codecs are usually already
there in the servers so it's all fine.
But when doing an upgrade by moving the data from one cluster to a brand new
one, compression codecs might be missing.
bin/hbase upgrade -check will not report any missing codec, but HBase will
not be able to start after the upgrade because the codes are missing.
I think bin/hbase upgrade -check should check the compression codecs
configured on the tables and make sure they are available on the new cluster.
f not, it should be reported.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-8763) [BRAINSTORM] Combine MVCC and SeqId
[ https://issues.apache.org/jira/browse/HBASE-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey Zhong updated HBASE-8763: - Attachment: hbase-8763-v2.patch Full test suite passed locally. TestAcidGuarantees passed 10 times in one loop. [~saint@gmail.com] I've moved the sequenceId wait for assignment from ring buffer consumer to rpc handlers. Hopefully we can restore the performance. After mvcc log sequence combining, one idea I come up today is that we can introduce a client read flushed changes only mode. In this mode, a client only read changes are flushed. During recovery we can set its scanner read point to last flushed sequence id while the region is still under recovery. The total recovery time for those clients are failure detection time + region assignment time. I also attached a write up on this JIRA. [BRAINSTORM] Combine MVCC and SeqId --- Key: HBASE-8763 URL: https://issues.apache.org/jira/browse/HBASE-8763 Project: HBase Issue Type: Improvement Components: regionserver Reporter: Enis Soztutar Assignee: Jeffrey Zhong Priority: Critical Attachments: HBase MVCC LogSeqId Combined.pdf, hbase-8736-poc.patch, hbase-8763-poc-v1.patch, hbase-8763-v1.patch, hbase-8763-v2.patch, hbase-8763_wip1.patch HBASE-8701 and a lot of recent issues include good discussions about mvcc + seqId semantics. It seems that having mvcc and the seqId complicates the comparator semantics a lot in regards to flush + WAL replay + compactions + delete markers and out of order puts. Thinking more about it I don't think we need a MVCC write number which is different than the seqId. We can keep the MVCC semantics, read point and smallest read points intact, but combine mvcc write number and seqId. This will allow cleaner semantics + implementation + smaller data files. We can do some brainstorming for 0.98. We still have to verify that this would be semantically correct, it should be so by my current understanding. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-6990) Pretty print TTL
[
https://issues.apache.org/jira/browse/HBASE-6990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Esteban Gutierrez updated HBASE-6990:
-
Attachment: HBASE-6990.v4.patch
Fixed typo that caused unit tests to fail after refactoring, new version
handles TTL with day, hour, minute precision.
[~stack] now we show TTLs with extra precision
{code}
TTL = '259200 SECONDS (3 DAYS)'
TTL = '259203 SECONDS (3 DAYS 3 SECONDS)'
TTL = '90059 SECONDS (1 DAY 1 HOUR 59 SECONDS)'
{code}
Didn't add years or months due the leap year and month lengths.
Pretty print TTL
Key: HBASE-6990
URL: https://issues.apache.org/jira/browse/HBASE-6990
Project: HBase
Issue Type: Improvement
Components: Usability
Reporter: Jean-Daniel Cryans
Assignee: Esteban Gutierrez
Priority: Minor
Attachments: HBASE-6990.v0.patch, HBASE-6990.v1.patch,
HBASE-6990.v2.patch, HBASE-6990.v3.patch, HBASE-6990.v4.patch
I've seen a lot of users getting confused by the TTL configuration and I
think that if we just pretty printed it it would solve most of the issues.
For example, let's say a user wanted to set a TTL of 90 days. That would be
7776000. But let's say that it was typo'd to 7776 instead, it gives you
900 days!
So when we print the TTL we could do something like x days, x hours, x
minutes, x seconds (real_ttl_value). This would also help people when they
use ms instead of seconds as they would see really big values in there.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11189) Subprocedure should be marked as complete upon failure
[
https://issues.apache.org/jira/browse/HBASE-11189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13999613#comment-13999613
]
Hadoop QA commented on HBASE-11189:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12645106/11189-v1.txt
against trunk revision .
ATTACHMENT ID: 12645106
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
58 warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestMultiParallel
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9521//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9521//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9521//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9521//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9521//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9521//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9521//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9521//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9521//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9521//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9521//console
This message is automatically generated.
Subprocedure should be marked as complete upon failure
--
Key: HBASE-11189
URL: https://issues.apache.org/jira/browse/HBASE-11189
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 0.99.0, 0.98.3
Attachments: 11189-v1.txt
ProcedureMember#submitSubprocedure() uses the following check:
{code}
if (!rsub.isComplete()) {
LOG.error(Subproc ' + procName + ' is already running. Bailing
out);
return false;
}
{code}
If a subprocedure of that name previously ran but failed, its complete field
would stay false, leading to early bailout.
A failed subprocedure should mark itself complete.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-10625) Remove unnecessary key compare from AbstractScannerV2.reseekTo
[
https://issues.apache.org/jira/browse/HBASE-10625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-10625:
--
Resolution: Invalid
Status: Resolved (was: Patch Available)
Remove unnecessary key compare from AbstractScannerV2.reseekTo
--
Key: HBASE-10625
URL: https://issues.apache.org/jira/browse/HBASE-10625
Project: HBase
Issue Type: Bug
Reporter: Lars Hofhansl
Attachments: 10625-0.94-experimental.txt, 10625-0.94.txt,
10625-trunk-experimental.txt, 10625-trunk.txt
In reseekTo we find this
{code}
...
compared = compareKey(reader.getComparator(), key, offset, length);
if (compared 1) {
// If the required key is less than or equal to current key, then
// don't do anything.
return compared;
} else {
...
return loadBlockAndSeekToKey(this.block, this.nextIndexedKey,
false, key, offset, length, false);
...
{code}
loadBlockAndSeekToKey already does the right thing when a we pass a key that
sorts before the current key. It's less efficient than this early check, but
in the vast (all?) cases we pass forward keys (as required by the reseek
contract). We're optimizing the wrong thing.
Scanning with the ExplicitColumnTracker is 20-30% faster.
(I tested with rows of 5 short KVs selected the 2nd and or 4th column)
I propose simply removing that check.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11150) Images in website are broken
[
https://issues.apache.org/jira/browse/HBASE-11150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13999381#comment-13999381
]
Hudson commented on HBASE-11150:
ABORTED: Integrated in HBase-TRUNK #5132 (See
[https://builds.apache.org/job/HBase-TRUNK/5132/])
HBASE-11150 Images in website are broken (enis: rev 1593655)
* /hbase/trunk/src/main/site/resources/images/architecture.gif
* /hbase/trunk/src/main/site/resources/images/big_h_logo.png
* /hbase/trunk/src/main/site/resources/images/favicon.ico
* /hbase/trunk/src/main/site/resources/images/hadoop-logo.jpg
* /hbase/trunk/src/main/site/resources/images/hfile.png
* /hbase/trunk/src/main/site/resources/images/hfilev2.png
* /hbase/trunk/src/main/site/resources/images/replication_overview.png
Images in website are broken
Key: HBASE-11150
URL: https://issues.apache.org/jira/browse/HBASE-11150
Project: HBase
Issue Type: Bug
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 0.99.0
It seems that most of the images in the website are broken. Somehow the image
files are lost from 0.94 - 0.96.
https://hbase.apache.org/replication.html
{code}
HW10676:hbase-0.94$ find -name *.png
./src/main/resources/hbase-webapps/static/hbase_logo.png
./src/site/resources/images/big_h_logo.png
./src/site/resources/images/hbase_logo.png
./src/site/resources/images/hfile.png
./src/site/resources/images/hfilev2.png
./src/site/resources/images/replication_overview.png
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-10835) DBE encode path improvements
[ https://issues.apache.org/jira/browse/HBASE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-10835: --- Status: Open (was: Patch Available) DBE encode path improvements Key: HBASE-10835 URL: https://issues.apache.org/jira/browse/HBASE-10835 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10835.patch, HBASE-10835_V2.patch, HBASE-10835_V3.patch, HBASE-10835_V4.patch Here 1st we write KVs (Cells) into a buffer and then passed to DBE encoder. Encoder again reads kvs one by one from the buffer and encodes and creates a new buffer. There is no need to have this model now. Previously we had option of no encode in disk and encode only in cache. At that time the read buffer from a HFile block was passed to this and encodes. So encode cell by cell can be done now. Making this change will need us to have a NoOp DBE impl which just do the write of a cell as it is with out any encoding. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11167) Avoid usage of java.rmi package Exception in MemStore
[ https://issues.apache.org/jira/browse/HBASE-11167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-11167: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to Trunk. Thanks for the reviews. Avoid usage of java.rmi package Exception in MemStore - Key: HBASE-11167 URL: https://issues.apache.org/jira/browse/HBASE-11167 Project: HBase Issue Type: Improvement Components: regionserver Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-11167.patch This Exception was in use already. While making MemStore into Interface I have not looked closely into it (whether class from RMI or not). So it went into Interface as well. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-4920) We need a mascot, a totem
[ https://issues.apache.org/jira/browse/HBASE-4920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13998870#comment-13998870 ] Nick Dimiduk commented on HBASE-4920: - bq. https://issues.apache.org/jira/secure/attachment/12511412/HBase%20Orca%20Logo.jpg +1 as well. We need a mascot, a totem - Key: HBASE-4920 URL: https://issues.apache.org/jira/browse/HBASE-4920 Project: HBase Issue Type: Task Reporter: stack Attachments: Apache_HBase_Orca_Logo_1.jpg, Apache_HBase_Orca_Logo_Mean_version-3.pdf, Apache_HBase_Orca_Logo_Mean_version-4.pdf, HBase Orca Logo.jpg, Orca_479990801.jpg, Screen shot 2011-11-30 at 4.06.17 PM.png, apache hbase orca logo_Proof 3.pdf, apache logo_Proof 8.pdf, krake.zip, more_orcas.png, more_orcas2.png, photo (2).JPG, plus_orca.png We need a totem for our t-shirt that is yet to be printed. O'Reilly owns the Clyesdale. We need something else. We could have a fluffy little duck that quacks 'hbase!' when you squeeze it and we could order boxes of them from some off-shore sweatshop that subcontracts to a contractor who employs child labor only. Or we could have an Orca (Big!, Fast!, Killer!, and in a poem that Marcy from Salesforce showed me, that was a bit too spiritual for me to be seen quoting here, it had the Orca as the 'Guardian of the Cosmic Memory': i.e. in translation, bigdata). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11171) More doc improvements on block cache options
[
https://issues.apache.org/jira/browse/HBASE-11171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13999031#comment-13999031
]
Hadoop QA commented on HBASE-11171:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12644969/11171v2.txt
against trunk revision .
ATTACHMENT ID: 12644969
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9512//console
This message is automatically generated.
More doc improvements on block cache options
Key: HBASE-11171
URL: https://issues.apache.org/jira/browse/HBASE-11171
Project: HBase
Issue Type: Sub-task
Components: documentation
Reporter: stack
Assignee: stack
Attachments: 11171.txt, 11171v2.txt, 11171v2.txt, 11171v3.txt,
11171v4.txt, 11171v5.txt, 11171v6.txt, 11171v7.txt, 11171v8.txt,
package-summary.html
I have more doc improvement (no code changes for sure this time). Follow on
from HBASE-11098.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11119) Update ExportSnapShot to optionally not use a tmp file on external file system
[ https://issues.apache.org/jira/browse/HBASE-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13992887#comment-13992887 ] Matteo Bertozzi commented on HBASE-9: - +1 patch looks good to me, it applies cleanly to 96, 98 and trunk and almost cleanly to 94. I'll fix the long lines problem on commit. (ok to have a -Dsnapshot.export.skip.tmp=true instead of a -skip-tmp because you must know what you are doing. This may speedup a bit where the rename operation is slow but you have to manually handle the in-progress or failure case, since for the tools the snapshot will be available as soon as the manifest is copied) Update ExportSnapShot to optionally not use a tmp file on external file system -- Key: HBASE-9 URL: https://issues.apache.org/jira/browse/HBASE-9 Project: HBase Issue Type: Improvement Components: snapshots Reporter: Ted Malaska Assignee: Ted Malaska Priority: Minor Fix For: 0.99.0, 0.96.3, 0.94.20, 0.98.3 Attachments: HBASE-9.patch There are FileSystem like S3 where renaming is extremely expensive. This patch will add a parameter that says something like use.tmp.folder It will be defaulted to true. So default behavior is the same. If false is set them the files will land in the final destination with no need for a rename. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11171) More doc improvements on block cache options
[ https://issues.apache.org/jira/browse/HBASE-11171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-11171: -- Attachment: 11171v5.txt Categorize test. More doc improvements on block cache options Key: HBASE-11171 URL: https://issues.apache.org/jira/browse/HBASE-11171 Project: HBase Issue Type: Sub-task Components: documentation Reporter: stack Assignee: stack Attachments: 11171.txt, 11171v2.txt, 11171v2.txt, 11171v3.txt, 11171v4.txt, 11171v5.txt, 11171v6.txt, 11171v7.txt, 11171v8.txt, package-summary.html I have more doc improvement (no code changes for sure this time). Follow on from HBASE-11098. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10936) Add zeroByte encoding test
[ https://issues.apache.org/jira/browse/HBASE-10936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13999273#comment-13999273 ] Hudson commented on HBASE-10936: SUCCESS: Integrated in HBase-0.94-security #483 (See [https://builds.apache.org/job/HBase-0.94-security/483/]) HBASE-10936 Add zeroByte encoding test. (larsh: rev 1594139) * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/encoding/TestDataBlockEncoders.java Add zeroByte encoding test -- Key: HBASE-10936 URL: https://issues.apache.org/jira/browse/HBASE-10936 Project: HBase Issue Type: Sub-task Components: test Reporter: Lars Hofhansl Priority: Minor Fix For: 0.96.3, 0.94.20 Attachments: 10936-0.94.txt, 10936-0.96.txt, 10936-0.98.txt -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11143) Improve replication metrics
[ https://issues.apache.org/jira/browse/HBASE-11143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13999302#comment-13999302 ] Hudson commented on HBASE-11143: SUCCESS: Integrated in HBase-0.94-JDK7 #135 (See [https://builds.apache.org/job/HBase-0.94-JDK7/135/]) HBASE-11143 Improve replication metrics. (larsh: rev 1594471) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceMetrics.java Improve replication metrics --- Key: HBASE-11143 URL: https://issues.apache.org/jira/browse/HBASE-11143 Project: HBase Issue Type: Bug Components: Replication Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.99.0, 0.94.20, 0.98.3 Attachments: 11143-0.94-v2.txt, 11143-0.94-v3.txt, 11143-0.94.txt, 11143-trunk.txt We are trying to report on replication lag and find that there is no good single metric to do that. ageOfLastShippedOp is close, but unfortunately it is increased even when there is nothing to ship on a particular RegionServer. I would like discuss a few options here: Add a new metric: replicationQueueTime (or something) with the above meaning. I.e. if we have something to ship we set the age of that last shipped edit, if we fail we increment that last time (just like we do now). But if there is nothing to replicate we set it to current time (and hence that metric is reported to close to 0). Alternatively we could change the meaning of ageOfLastShippedOp to mean to do that. That might lead to surprises, but the current behavior is clearly weird when there is nothing to replicate. Comments? [~jdcryans], [~stack]. If approach sounds good, I'll make a patch for all branches. Edit: Also adds a new shippedKBs metric to track the amount of data that is shipped via replication. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11098) Improve documentation around our blockcache options
[ https://issues.apache.org/jira/browse/HBASE-11098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13999385#comment-13999385 ] Hudson commented on HBASE-11098: ABORTED: Integrated in HBase-TRUNK #5132 (See [https://builds.apache.org/job/HBase-TRUNK/5132/]) HBASE-11098 Improve documentation around our blockcache options - addendum (nkeywal: rev 1594640) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/FileIOEngine.java HBASE-11098 Improve documentation around our blockcache options (stack: rev 1594413) * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/ByteBufferArray.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheConfig.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/Cacheable.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CombinedBlockCache.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketAllocator.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/ByteBufferIOEngine.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/FileIOEngine.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/IOEngine.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/package-info.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/package-info.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/RandomSeek.java Improve documentation around our blockcache options --- Key: HBASE-11098 URL: https://issues.apache.org/jira/browse/HBASE-11098 Project: HBase Issue Type: Sub-task Components: documentation Reporter: stack Assignee: stack Fix For: 0.99.0 Attachments: 11098.add.patch, 11098.txt, 11098v2.txt, 11098v3.txt Adding as a subtask on [~ndimiduk] necessary cleanup. Trying to make sense of this stuff I started adding doc (package-info files, javadoc). I see this as complementary to the parent task work. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11193) hbase web UI show wrong Catalog Table Description
[
https://issues.apache.org/jira/browse/HBASE-11193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13999540#comment-13999540
]
Guo Ruijing commented on HBASE-11193:
-
/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon:
%javaTableName tableName = systemTable.getTableName();/%java
tda href=table.jsp?name=% tableName %% tableName %/a/td
%if (frags != null)%
td align=center% frags.get(tableName.getNameAsString()) != null ?
frags.get(tableName.getNameAsString())
.intValue() + % : n/a %/td
/%if
%javaString description = null;
if (tableName.equals(TableName.META_TABLE_NAME))
{ description = The hbase:meta table holds references to all User Table
regions; }
else
{ description = The .NAMESPACE. table holds information about namespaces.; }
/%java
td% description %/td
we should add tableName.equals(TableName.) for ACL
hbase web UI show wrong Catalog Table Description
-
Key: HBASE-11193
URL: https://issues.apache.org/jira/browse/HBASE-11193
Project: HBase
Issue Type: Bug
Components: UI
Reporter: Guo Ruijing
On security cluster, check the hbase master web page and look into 'Catalog
Tables' on 'Tables' Section, the Description for 'hbase:acl' table is not
expected:
–
Table Name Description
*hbase:acl The .NAMESPACE. table holds information about namespaces.*
hbase:meta The hbase:meta table holds references to all User Table regions
hbase:namespace The .NAMESPACE. table holds information about namespaces
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-8763) [BRAINSTORM] Combine MVCC and SeqId
[ https://issues.apache.org/jira/browse/HBASE-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey Zhong updated HBASE-8763: - Attachment: hbase-8763-v2.patch [BRAINSTORM] Combine MVCC and SeqId --- Key: HBASE-8763 URL: https://issues.apache.org/jira/browse/HBASE-8763 Project: HBase Issue Type: Improvement Components: regionserver Reporter: Enis Soztutar Assignee: Jeffrey Zhong Priority: Critical Attachments: HBase MVCC LogSeqId Combined.pdf, hbase-8736-poc.patch, hbase-8763-poc-v1.patch, hbase-8763-v1.patch, hbase-8763-v2.patch, hbase-8763_wip1.patch HBASE-8701 and a lot of recent issues include good discussions about mvcc + seqId semantics. It seems that having mvcc and the seqId complicates the comparator semantics a lot in regards to flush + WAL replay + compactions + delete markers and out of order puts. Thinking more about it I don't think we need a MVCC write number which is different than the seqId. We can keep the MVCC semantics, read point and smallest read points intact, but combine mvcc write number and seqId. This will allow cleaner semantics + implementation + smaller data files. We can do some brainstorming for 0.98. We still have to verify that this would be semantically correct, it should be so by my current understanding. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11128) Add -target option to ExportSnapshot to export with a different name
[ https://issues.apache.org/jira/browse/HBASE-11128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13999366#comment-13999366 ] Hudson commented on HBASE-11128: ABORTED: Integrated in HBase-TRUNK #5132 (See [https://builds.apache.org/job/HBase-TRUNK/5132/]) HBASE-11128 Add -target option to ExportSnapshot to export with a different name (mbertozzi: rev 1593775) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java Add -target option to ExportSnapshot to export with a different name Key: HBASE-11128 URL: https://issues.apache.org/jira/browse/HBASE-11128 Project: HBase Issue Type: Improvement Components: snapshots Affects Versions: 0.99.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Trivial Fix For: 0.99.0, 0.96.3, 0.94.20, 0.98.3 Attachments: HBASE-11128-v0.patch Add a -target option to export the snapshot using a different name -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11194) [AccessController] issue with covering permission check in case of concurrent op on same row
Anoop Sam John created HBASE-11194: -- Summary: [AccessController] issue with covering permission check in case of concurrent op on same row Key: HBASE-11194 URL: https://issues.apache.org/jira/browse/HBASE-11194 Project: HBase Issue Type: Bug Components: security Affects Versions: 0.98.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0, 0.98.3 The issue is the hook where we do check in which we have not acquired rowlock. Take case of delete, we do the check in the preDelete() hook. We do get the covering cells and check against their acls. At the point of the preDelete hook, we have not acquired the row lock on the deleting row. Consider 2 parallel threads one doing put and other delete both dealing with same row. Thread 1 acquired the rowlock and decided the TS (HRS time) and doing the memstore write and HLog sync but the mvcc read point is NOT advanced. Thread 2 at same time, doing the delete of the row (Say with latest TS . The intent is to delete entire row) and in place of preDelete hook. There is no row locking happening at this point. As part of covering permission check, it doing a Get. But as said above, the put is not complete and the mvcc advance has not happened. So the Get won’t return the new cell. It will return the old cells. And the check pass for the old cells. Now suppose the new cell ACL is not matching for the deleting user. But the cell was not read, the check has not happened. So the ACL check will allow the user to delete row.. The flow later comes to HRegion#doMiniBatchMutate() and try acquire row lock and by that time the Thread 1 op was over. So it will get lock and will add the delete tombstone. As a result the cell, for which the deleting user has no acl right, also will get deleted. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10569) Co-locate meta and master
[ https://issues.apache.org/jira/browse/HBASE-10569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13998795#comment-13998795 ] Jonathan Hsieh commented on HBASE-10569: bq. Point of order: If I read it right, you can only do 'agreement' on mailing list (Community decisions must be reached on the mailing list. [1]). Thanks for the reminder. Duly noted. Co-locate meta and master - Key: HBASE-10569 URL: https://issues.apache.org/jira/browse/HBASE-10569 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.99.0 Attachments: Co-locateMetaAndMasterHBASE-10569.pdf, hbase-10569_v1.patch, hbase-10569_v2.patch, hbase-10569_v3.1.patch, hbase-10569_v3.patch, master_rs.pdf I was thinking simplifying/improving the region assignments. The first step is to co-locate the meta and the master as many people agreed on HBASE-5487. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11134) Add a -list-snapshots option to SnapshotInfo
[
https://issues.apache.org/jira/browse/HBASE-11134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13999395#comment-13999395
]
Hudson commented on HBASE-11134:
FAILURE: Integrated in HBase-0.94 #1367 (See
[https://builds.apache.org/job/HBase-0.94/1367/])
HBASE-11134 Add a -list-snapshots option to SnapshotInfo (mbertozzi: rev
1594857)
*
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
Add a -list-snapshots option to SnapshotInfo
Key: HBASE-11134
URL: https://issues.apache.org/jira/browse/HBASE-11134
Project: HBase
Issue Type: Improvement
Components: snapshots
Affects Versions: 0.99.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Trivial
Fix For: 0.99.0, 0.96.3, 0.94.20, 0.98.3
Attachments: HBASE-11134-v0.patch, HBASE-11134-v1.patch
Add a -list-snapshots option to SnapshotInfo to show all the snapshots
available. Also add a -remote-dir option to simplify the usage of
SnapshotInfo in case the snapshot dir is not the one of the current hbase
cluster
{code}
$ hbase org.apache.hadoop.hbase.snapshot.SnapshotInfo -list-snapshots
SNAPSHOT | CREATION TIME| TABLE NAME
foo | 2014-05-07T22:40:13 | testtb
bar | 2014-05-07T22:40:16 | testtb
$ hbase org.apache.hadoop.hbase.snapshot.SnapshotInfo -remote-dir
file:///backup/ -snapshot my_local_snapshot
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11176) Make /src/main/xslt/configuration_to_docbook_section.xsl produce better Docbook
[ https://issues.apache.org/jira/browse/HBASE-11176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13999367#comment-13999367 ] Hudson commented on HBASE-11176: ABORTED: Integrated in HBase-TRUNK #5132 (See [https://builds.apache.org/job/HBase-TRUNK/5132/]) HBASE-11176 Make /src/main/xslt/configuration_to_docbook_section.xsl produce better Docbook (Misty Stanley-Jones) (stack: rev 1594765) * /hbase/trunk/src/main/xslt/configuration_to_docbook_section.xsl Make /src/main/xslt/configuration_to_docbook_section.xsl produce better Docbook --- Key: HBASE-11176 URL: https://issues.apache.org/jira/browse/HBASE-11176 Project: HBase Issue Type: Bug Components: build, documentation Affects Versions: 0.98.2 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Fix For: 0.99.0 Attachments: HBASE-11176.patch The file generates XML that is not valid, though it does build. As an example, the ID needs to be xml:id Sub-tasks: 1. Research the best way to display the info 2. Fix up the XSLT transform accordingly. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HBASE-11189) Subprocedure should be marked as complete upon failure
[
https://issues.apache.org/jira/browse/HBASE-11189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu reassigned HBASE-11189:
--
Assignee: Ted Yu
Subprocedure should be marked as complete upon failure
--
Key: HBASE-11189
URL: https://issues.apache.org/jira/browse/HBASE-11189
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 0.99.0, 0.98.3
Attachments: 11189-v1.txt
ProcedureMember#submitSubprocedure() uses the following check:
{code}
if (!rsub.isComplete()) {
LOG.error(Subproc ' + procName + ' is already running. Bailing
out);
return false;
}
{code}
If a subprocedure of that name previously ran but failed, its complete field
would stay false, leading to early bailout.
A failed subprocedure should mark itself complete.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10985) Decouple Split Transaction from Zookeeper
[
https://issues.apache.org/jira/browse/HBASE-10985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13999574#comment-13999574
]
Hadoop QA commented on HBASE-10985:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12644662/HBASE_10985-v5.patch
against trunk revision .
ATTACHMENT ID: 12644662
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
60 warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ .getSplitTransactionConsensus().waitForSplitTransaction(server,
services, parent, hri_a, hri_b, std);
+ .completeSplitTransaction(server, services, regions.getFirst(),
regions.getSecond(), std, parent);
+ public ZKSplitTransactionConsensus(ConsensusProvider consensusProvider,
ZooKeeperWatcher watcher) {
+ public void completeSplitTransaction(final Server server, final
RegionServerServices services, HRegion a,
+.completeSplitTransaction(server, server, regions.getFirst(),
regions.getSecond(), split.std, region);
+public void completeSplitTransaction(Server server, RegionServerServices
services, HRegion a, HRegion b,
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestSnapshotCloneIndependence
org.apache.hadoop.hbase.client.TestAdmin
org.apache.hadoop.hbase.master.TestTableLockManager
org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient
org.apache.hadoop.hbase.client.TestFromClientSide
org.apache.hadoop.hbase.rest.TestTableResource
org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor
org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction
org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster
org.apache.hadoop.hbase.util.TestHBaseFsck
{color:red}-1 core zombie tests{color}. There are 3 zombie test(s):
at
org.apache.hadoop.hbase.master.TestMaster.testMasterOpsWhileSplitting(TestMaster.java:89)
at
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery.testSplitWhileBulkLoadPhase(TestLoadIncrementalHFilesSplitRecovery.java:361)
at
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery.testSplitWhileBulkLoadPhase(TestLoadIncrementalHFilesSplitRecovery.java:361)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9519//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9519//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9519//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9519//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9519//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9519//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9519//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9519//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9519//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9519//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9519//console
This message is automatically generated.
Decouple Split Transaction from Zookeeper
-
Key: HBASE-10985
URL:
[jira] [Commented] (HBASE-5671) hbase.metrics.showTableName should be true by default
[ https://issues.apache.org/jira/browse/HBASE-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13999172#comment-13999172 ] Jean-Daniel Cryans commented on HBASE-5671: --- For anyone stumbling here because of this message Inconsistent configuration. Previous configuration for using table name in metrics: true, new configuration: false, please see HBASE-11188. hbase.metrics.showTableName should be true by default - Key: HBASE-5671 URL: https://issues.apache.org/jira/browse/HBASE-5671 Project: HBase Issue Type: Improvement Components: metrics Reporter: Enis Soztutar Assignee: Enis Soztutar Priority: Critical Fix For: 0.94.0, 0.95.0 Attachments: HBASE-5671_v1.patch HBASE-4768 added per-cf metrics and a new configuration option hbase.metrics.showTableName. We should switch the conf option to true by default, since it is not intuitive (at least to me) to aggregate per-cf across tables by default, and it seems confusing to report on cf's without table names. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11076) Update refguide on getting 0.94.x to run on hadoop 2.2.0+
[
https://issues.apache.org/jira/browse/HBASE-11076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13999535#comment-13999535
]
Ted Yu commented on HBASE-11076:
{code}
+ paraThe next step is regenerate Protobuf files and assuming that
the Protobuf has been installed:/para
{code}
nit: 'is regenerate' - 'is to regenerate'
Update refguide on getting 0.94.x to run on hadoop 2.2.0+
-
Key: HBASE-11076
URL: https://issues.apache.org/jira/browse/HBASE-11076
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Assignee: Gustavo Anatoly
Attachments: HBASE-11076.patch
http://hbase.apache.org/book.html#d248e643 contains steps for rebuilding 0.94
code base to run on hadoop 2.2.0+
However, the files under
src/main/java/org/apache/hadoop/hbase/protobuf/generated were produced by
protoc 2.4.0
These files need to be regenerated.
See
http://search-hadoop.com/m/DHED4j7Um02/HBase+0.94+on+hadoop+2.2.0subj=Re+HBase+0+94+on+hadoop+2+2+0+2+4+0+
This issue is to update refguide with this regeneration step.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11189) Subprocedure should be marked as complete upon failure
[
https://issues.apache.org/jira/browse/HBASE-11189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13999693#comment-13999693
]
Ted Yu commented on HBASE-11189:
Test failure was not related to the patch. This test sometimes failed in other
QA reports as well.
Javadoc warnings were not introduced by the patch.
Subprocedure should be marked as complete upon failure
--
Key: HBASE-11189
URL: https://issues.apache.org/jira/browse/HBASE-11189
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 0.99.0, 0.98.3
Attachments: 11189-v1.txt
ProcedureMember#submitSubprocedure() uses the following check:
{code}
if (!rsub.isComplete()) {
LOG.error(Subproc ' + procName + ' is already running. Bailing
out);
return false;
}
{code}
If a subprocedure of that name previously ran but failed, its complete field
would stay false, leading to early bailout.
A failed subprocedure should mark itself complete.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11193) hbase web UI show wrong Catalog Table Description
[ https://issues.apache.org/jira/browse/HBASE-11193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1319#comment-1319 ] Ted Yu commented on HBASE-11193: Seems to be dup of HBASE-10611 hbase web UI show wrong Catalog Table Description - Key: HBASE-11193 URL: https://issues.apache.org/jira/browse/HBASE-11193 Project: HBase Issue Type: Bug Components: UI Reporter: Guo Ruijing On security cluster, check the hbase master web page and look into 'Catalog Tables' on 'Tables' Section, the Description for 'hbase:acl' table is not expected: – Table Name Description *hbase:acl The .NAMESPACE. table holds information about namespaces.* hbase:meta The hbase:meta table holds references to all User Table regions hbase:namespace The .NAMESPACE. table holds information about namespaces -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11195) Potentially improve block locality during major compaction for old regions
[ https://issues.apache.org/jira/browse/HBASE-11195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] churro morales updated HBASE-11195: --- Attachment: HBASE-11195-0.94.patch Potentially improve block locality during major compaction for old regions -- Key: HBASE-11195 URL: https://issues.apache.org/jira/browse/HBASE-11195 Project: HBase Issue Type: Improvement Affects Versions: 0.94.19 Reporter: churro morales Attachments: HBASE-11195-0.94.patch This might be a specific use case. But we have some regions which are no longer written to (due to the key). Those regions have 1 store file and they are very old, they haven't been written to in a while. We still use these regions to read from so locality would be nice. I propose putting a configuration option: something like hbase.hstore.min.locality.to.skip.major.compact [between 0 and 1] such that you can decide whether or not to skip major compaction for an old region with a single store file. I'll attach a patch, let me know what you guys think. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11143) Improve replication metrics
[ https://issues.apache.org/jira/browse/HBASE-11143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13999374#comment-13999374 ] Hudson commented on HBASE-11143: FAILURE: Integrated in HBase-0.94-on-Hadoop-2 #84 (See [https://builds.apache.org/job/HBase-0.94-on-Hadoop-2/84/]) HBASE-11143 Improve replication metrics. (larsh: rev 1594471) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceMetrics.java Improve replication metrics --- Key: HBASE-11143 URL: https://issues.apache.org/jira/browse/HBASE-11143 Project: HBase Issue Type: Bug Components: Replication Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.99.0, 0.94.20, 0.98.3 Attachments: 11143-0.94-v2.txt, 11143-0.94-v3.txt, 11143-0.94.txt, 11143-trunk.txt We are trying to report on replication lag and find that there is no good single metric to do that. ageOfLastShippedOp is close, but unfortunately it is increased even when there is nothing to ship on a particular RegionServer. I would like discuss a few options here: Add a new metric: replicationQueueTime (or something) with the above meaning. I.e. if we have something to ship we set the age of that last shipped edit, if we fail we increment that last time (just like we do now). But if there is nothing to replicate we set it to current time (and hence that metric is reported to close to 0). Alternatively we could change the meaning of ageOfLastShippedOp to mean to do that. That might lead to surprises, but the current behavior is clearly weird when there is nothing to replicate. Comments? [~jdcryans], [~stack]. If approach sounds good, I'll make a patch for all branches. Edit: Also adds a new shippedKBs metric to track the amount of data that is shipped via replication. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-8763) [BRAINSTORM] Combine MVCC and SeqId
[ https://issues.apache.org/jira/browse/HBASE-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey Zhong updated HBASE-8763: - Attachment: (was: hbase-8763-v2.patch) [BRAINSTORM] Combine MVCC and SeqId --- Key: HBASE-8763 URL: https://issues.apache.org/jira/browse/HBASE-8763 Project: HBase Issue Type: Improvement Components: regionserver Reporter: Enis Soztutar Assignee: Jeffrey Zhong Priority: Critical Attachments: HBase MVCC LogSeqId Combined.pdf, hbase-8736-poc.patch, hbase-8763-poc-v1.patch, hbase-8763-v1.patch, hbase-8763-v2.patch, hbase-8763_wip1.patch HBASE-8701 and a lot of recent issues include good discussions about mvcc + seqId semantics. It seems that having mvcc and the seqId complicates the comparator semantics a lot in regards to flush + WAL replay + compactions + delete markers and out of order puts. Thinking more about it I don't think we need a MVCC write number which is different than the seqId. We can keep the MVCC semantics, read point and smallest read points intact, but combine mvcc write number and seqId. This will allow cleaner semantics + implementation + smaller data files. We can do some brainstorming for 0.98. We still have to verify that this would be semantically correct, it should be so by my current understanding. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11076) Update refguide on getting 0.94.x to run on hadoop 2.2.0+
[ https://issues.apache.org/jira/browse/HBASE-11076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gustavo Anatoly updated HBASE-11076: Attachment: HBASE-11076-v1.patch Thanks Ted. Update refguide on getting 0.94.x to run on hadoop 2.2.0+ - Key: HBASE-11076 URL: https://issues.apache.org/jira/browse/HBASE-11076 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Gustavo Anatoly Attachments: HBASE-11076-v1.patch, HBASE-11076.patch http://hbase.apache.org/book.html#d248e643 contains steps for rebuilding 0.94 code base to run on hadoop 2.2.0+ However, the files under src/main/java/org/apache/hadoop/hbase/protobuf/generated were produced by protoc 2.4.0 These files need to be regenerated. See http://search-hadoop.com/m/DHED4j7Um02/HBase+0.94+on+hadoop+2.2.0subj=Re+HBase+0+94+on+hadoop+2+2+0+2+4+0+ This issue is to update refguide with this regeneration step. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11189) Subprocedure should be marked as complete upon failure
[
https://issues.apache.org/jira/browse/HBASE-11189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-11189:
---
Attachment: 11189-v1.txt
All Snapshot related tests pass.
Subprocedure should be marked as complete upon failure
--
Key: HBASE-11189
URL: https://issues.apache.org/jira/browse/HBASE-11189
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 0.99.0, 0.98.3
Attachments: 11189-v1.txt
ProcedureMember#submitSubprocedure() uses the following check:
{code}
if (!rsub.isComplete()) {
LOG.error(Subproc ' + procName + ' is already running. Bailing
out);
return false;
}
{code}
If a subprocedure of that name previously ran but failed, its complete field
would stay false, leading to early bailout.
A failed subprocedure should mark itself complete.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11165) Scaling so cluster can host 1M regions and beyond (50M regions?)
[ https://issues.apache.org/jira/browse/HBASE-11165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13999605#comment-13999605 ] Mikhail Antonov commented on HBASE-11165: - Before we had META hosted on master, I'd have said that this situation is different from NN - as for NN it's kind of known that 1Gb or RAM can hold ~1M inodes or so and whole FSImage must be in memory. With META tied to Master it's kind of same issue? Does someone have estimates - how many memory on avg. are needed for META table for each, say, 10k or 100k regions? Do we have some list of bottlenecks on this way? I remember from the hackathon @ Salesforce that one of important bottlenecks is number of splits/assignment ops / sec (do I recall correctly [~toffer]?) Scaling so cluster can host 1M regions and beyond (50M regions?) Key: HBASE-11165 URL: https://issues.apache.org/jira/browse/HBASE-11165 Project: HBase Issue Type: Brainstorming Reporter: stack This discussion issue comes out of Co-locate Meta And Master HBASE-10569 and comments on the doc posted there. A user -- our Francis Liu -- needs to be able to scale a cluster to do 1M regions maybe even 50M later. This issue is about discussing how we will do that (or if not 50M on a cluster, how otherwise we can attain same end). More detail to follow. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11191) HBase ClusterId File Empty Check Loggic
sunjingtao created HBASE-11191:
--
Summary: HBase ClusterId File Empty Check Loggic
Key: HBASE-11191
URL: https://issues.apache.org/jira/browse/HBASE-11191
Project: HBase
Issue Type: Bug
Environment: HBase 0.94+Hadoop2.2.0+Zookeeper3.4.5
Reporter: sunjingtao
if the clusterid file exists but empty ,then the following check logic in the
MasterFileSystem.java has none effects.
if (!FSUtils.checkClusterIdExists(fs, rd, c.getInt(
HConstants.THREAD_WAKE_FREQUENCY, 10 * 1000))) {
FSUtils.setClusterId(fs, rd, UUID.randomUUID().toString(), c.getInt(
HConstants.THREAD_WAKE_FREQUENCY, 10 * 1000));
}
clusterId = FSUtils.getClusterId(fs, rd);
because the checkClusterIdExists method only check the path .
Path filePath = new Path(rootdir, HConstants.CLUSTER_ID_FILE_NAME);
return fs.exists(filePath);
in my case ,the file exists but is empty,so the readed clusterid is null which
cause a nullPointerException:
java.lang.NullPointerException
at org.apache.hadoop.hbase.util.Bytes.toBytes(Bytes.java:441)
at
org.apache.hadoop.hbase.zookeeper.ClusterId.setClusterId(ClusterId.java:72)
at
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:581)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:433)
at java.lang.Thread.run(Thread.java:745)
is this a bug?please make sure!
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HBASE-10990) Running CompressionTest yields unrelated schema metrics configuration errors.
[ https://issues.apache.org/jira/browse/HBASE-10990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Daniel Cryans resolved HBASE-10990. Resolution: Duplicate I'm fixing this in HBASE-11188. Running CompressionTest yields unrelated schema metrics configuration errors. - Key: HBASE-10990 URL: https://issues.apache.org/jira/browse/HBASE-10990 Project: HBase Issue Type: Bug Affects Versions: 0.94.18 Reporter: Srikanth Srungarapu Assignee: Srikanth Srungarapu Priority: Minor Attachments: HBASE-10990.patch In a vanilla configuration, running CompressionTest yields the following error: sudo -u hdfs hbase org.apache.hadoop.hbase.util.CompressionTest /path/to/hfile gz Output: 13/03/07 14:49:40 ERROR metrics.SchemaMetrics: Inconsistent configuration. Previous configuration for using table name in metrics: true, new configuration: false -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11165) Scaling so cluster can host 1M regions and beyond (50M regions?)
[ https://issues.apache.org/jira/browse/HBASE-11165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1325#comment-1325 ] Jimmy Xiang commented on HBASE-11165: - It may take a while for a single master to assign 50 million regions. It is also quite some load to the ZK if we use ZK for region assignment. Scaling so cluster can host 1M regions and beyond (50M regions?) Key: HBASE-11165 URL: https://issues.apache.org/jira/browse/HBASE-11165 Project: HBase Issue Type: Brainstorming Reporter: stack This discussion issue comes out of Co-locate Meta And Master HBASE-10569 and comments on the doc posted there. A user -- our Francis Liu -- needs to be able to scale a cluster to do 1M regions maybe even 50M later. This issue is about discussing how we will do that (or if not 50M on a cluster, how otherwise we can attain same end). More detail to follow. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11171) More doc improvements on block cache options
[ https://issues.apache.org/jira/browse/HBASE-11171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-11171: -- Attachment: 11171v9.txt What I committed. Thanks for review Nick. All tests passed locally for me. More doc improvements on block cache options Key: HBASE-11171 URL: https://issues.apache.org/jira/browse/HBASE-11171 Project: HBase Issue Type: Sub-task Components: documentation Reporter: stack Assignee: stack Attachments: 11171.txt, 11171v2.txt, 11171v2.txt, 11171v3.txt, 11171v4.txt, 11171v5.txt, 11171v6.txt, 11171v7.txt, 11171v8.txt, 11171v9.txt, package-summary.html I have more doc improvement (no code changes for sure this time). Follow on from HBASE-11098. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11188) Inconsistent configuration for SchemaMetrics is always shown
[
https://issues.apache.org/jira/browse/HBASE-11188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13999815#comment-13999815
]
Matteo Bertozzi commented on HBASE-11188:
-
+1 on v2
Inconsistent configuration for SchemaMetrics is always shown
--
Key: HBASE-11188
URL: https://issues.apache.org/jira/browse/HBASE-11188
Project: HBase
Issue Type: Bug
Components: metrics
Affects Versions: 0.94.19
Reporter: Jean-Daniel Cryans
Assignee: Jean-Daniel Cryans
Fix For: 0.94.20
Attachments: HBASE-11188-0.94-v2.patch, HBASE-11188-0.94.patch
Some users have been complaining about this message:
{noformat}
ERROR org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics:
Inconsistent configuration. Previous configuration for using table name in
metrics: true, new configuration: false
{noformat}
The interesting thing is that we see it with default configurations, which
made me think that some code path must have been passing the wrong thing. I
found that if SchemaConfigured is passed a null Configuration in its
constructor that it will then pass null to SchemaMetrics#configureGlobally
which will interpret useTableName as being false:
{code}
public static void configureGlobally(Configuration conf) {
if (conf != null) {
final boolean useTableNameNew =
conf.getBoolean(SHOW_TABLE_NAME_CONF_KEY, false);
setUseTableName(useTableNameNew);
} else {
setUseTableName(false);
}
}
{code}
It should be set to true since that's the new default, meaning we missed it
in HBASE-5671.
I found one code path that passes a null configuration, StoreFile.Reader
extends SchemaConfigured and uses the constructor that only passes a Path, so
the Configuration is set to null.
I'm planning on just passing true instead of false, fixing the problem for
almost everyone (those that disable this feature will get the error message).
IMO it's not worth more efforts since it's a 0.94-only problem and it's not
actually doing anything bad.
I'm closing both HBASE-10990 and HBASE-10946 as duplicates.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HBASE-11188) Inconsistent configuration for SchemaMetrics is always shown
[
https://issues.apache.org/jira/browse/HBASE-11188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jean-Daniel Cryans resolved HBASE-11188.
Resolution: Fixed
Release Note:
Region servers with the default value for hbase.metrics.showTableName will stop
showing the error message ERROR
org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics: Inconsistent
configuration. Previous configuration for using table name in metrics: true,
new configuration: false.
Region servers configured with hbase.metrics.showTableName=false should now get
a message like this one: ERROR
org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics: Inconsistent
configuration. Previous configuration for using table name in metrics: false,
new configuration: true, and it's nothing to be concerned about.
Hadoop Flags: Reviewed
Grazie Matteo, I committed the patch to 0.94
Inconsistent configuration for SchemaMetrics is always shown
--
Key: HBASE-11188
URL: https://issues.apache.org/jira/browse/HBASE-11188
Project: HBase
Issue Type: Bug
Components: metrics
Affects Versions: 0.94.19
Reporter: Jean-Daniel Cryans
Assignee: Jean-Daniel Cryans
Fix For: 0.94.20
Attachments: HBASE-11188-0.94-v2.patch, HBASE-11188-0.94.patch
Some users have been complaining about this message:
{noformat}
ERROR org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics:
Inconsistent configuration. Previous configuration for using table name in
metrics: true, new configuration: false
{noformat}
The interesting thing is that we see it with default configurations, which
made me think that some code path must have been passing the wrong thing. I
found that if SchemaConfigured is passed a null Configuration in its
constructor that it will then pass null to SchemaMetrics#configureGlobally
which will interpret useTableName as being false:
{code}
public static void configureGlobally(Configuration conf) {
if (conf != null) {
final boolean useTableNameNew =
conf.getBoolean(SHOW_TABLE_NAME_CONF_KEY, false);
setUseTableName(useTableNameNew);
} else {
setUseTableName(false);
}
}
{code}
It should be set to true since that's the new default, meaning we missed it
in HBASE-5671.
I found one code path that passes a null configuration, StoreFile.Reader
extends SchemaConfigured and uses the constructor that only passes a Path, so
the Configuration is set to null.
I'm planning on just passing true instead of false, fixing the problem for
almost everyone (those that disable this feature will get the error message).
IMO it's not worth more efforts since it's a 0.94-only problem and it's not
actually doing anything bad.
I'm closing both HBASE-10990 and HBASE-10946 as duplicates.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HBASE-11195) Potentially improve block locality during major compaction for old regions
churro morales created HBASE-11195: -- Summary: Potentially improve block locality during major compaction for old regions Key: HBASE-11195 URL: https://issues.apache.org/jira/browse/HBASE-11195 Project: HBase Issue Type: Improvement Affects Versions: 0.94.19 Reporter: churro morales This might be a specific use case. But we have some regions which are no longer written to (due to the key). Those regions have 1 store file and they are very old, they haven't been written to in a while. We still use these regions to read from so locality would be nice. I propose putting a configuration option: something like hbase.hstore.min.locality.to.skip.major.compact [between 0 and 1] such that you can decide whether or not to skip major compaction for an old region with a single store file. I'll attach a patch, let me know what you guys think. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10247) Client promises about timestamps
[ https://issues.apache.org/jira/browse/HBASE-10247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1462#comment-1462 ] Enis Soztutar commented on HBASE-10247: --- Great. bq. I think something like TS_POLICY = 'state' would be best. Agreed. bq. We need at least two (ideally three) states See my comments at https://issues.apache.org/jira/browse/HBASE-9905?focusedCommentId=13815414page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13815414 We should think about HBASE-9905 and this together as a single configuration. It seems that we need these capabilities: - Know whether or not all timestamps are monotonically increasing (this will allow a lot of nice perf improvements and semantic guarantees. This can be either server side seqIds, or client side based ts oracle kind of setups. Both should be allowed and differentiated. Maybe this is a different table property. - Server side ts setting. - Mixed mode, or something like that to support existing tables. This should be deprecated. Client promises about timestamps Key: HBASE-10247 URL: https://issues.apache.org/jira/browse/HBASE-10247 Project: HBase Issue Type: Brainstorming Reporter: Lars Hofhansl Priority: Minor This is to start a discussion about timestamp promises declared per table of CF. For example if a client promises only monotonically increasing timestamps (or no custom set timestamps) and VERSIONS=1, we can aggressively and easily remove old versions of the same row/fam/col from the memstore before we flush, just by supplying a comparator that ignores the timestamp (i.e. two KV just differing by TS would be considered equal). That would increase the performance of counters significantly. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11185) Parallelize Snapshot operations
Matteo Bertozzi created HBASE-11185: --- Summary: Parallelize Snapshot operations Key: HBASE-11185 URL: https://issues.apache.org/jira/browse/HBASE-11185 Project: HBase Issue Type: Bug Components: snapshots Affects Versions: 0.99.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 0.99.0 Attachments: HBASE-11185-v0.patch when SnapshotInfo or snapshot verification is executed against a remote path, it may takes a while since all the code is mainly composed by sequential calls to the fs. This patch will parallelize all the snapshot operations using a thread pool to dispatch requests. The size of the pool is tunable by using hbase.snapshot.thread.pool.max -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11143) Improve replication metrics
[ https://issues.apache.org/jira/browse/HBASE-11143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13999271#comment-13999271 ] Hudson commented on HBASE-11143: SUCCESS: Integrated in HBase-0.94-security #483 (See [https://builds.apache.org/job/HBase-0.94-security/483/]) HBASE-11143 Improve replication metrics. (larsh: rev 1594471) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceMetrics.java Improve replication metrics --- Key: HBASE-11143 URL: https://issues.apache.org/jira/browse/HBASE-11143 Project: HBase Issue Type: Bug Components: Replication Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.99.0, 0.94.20, 0.98.3 Attachments: 11143-0.94-v2.txt, 11143-0.94-v3.txt, 11143-0.94.txt, 11143-trunk.txt We are trying to report on replication lag and find that there is no good single metric to do that. ageOfLastShippedOp is close, but unfortunately it is increased even when there is nothing to ship on a particular RegionServer. I would like discuss a few options here: Add a new metric: replicationQueueTime (or something) with the above meaning. I.e. if we have something to ship we set the age of that last shipped edit, if we fail we increment that last time (just like we do now). But if there is nothing to replicate we set it to current time (and hence that metric is reported to close to 0). Alternatively we could change the meaning of ageOfLastShippedOp to mean to do that. That might lead to surprises, but the current behavior is clearly weird when there is nothing to replicate. Comments? [~jdcryans], [~stack]. If approach sounds good, I'll make a patch for all branches. Edit: Also adds a new shippedKBs metric to track the amount of data that is shipped via replication. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11165) Scaling so cluster can host 1M regions and beyond (50M regions?)
[ https://issues.apache.org/jira/browse/HBASE-11165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13999672#comment-13999672 ] Mikhail Antonov commented on HBASE-11165: - I think the numbers mentioned for the particular usecase were somewhat like region size around 80gb being an issue. bq. We should also discuss why few, large regions are bad I'd think that one of limitations of overly big regions (besides one-mapper-per-region) is that they're inconvenient to split/move around/compact? bq. Now, maybe only a few regions are being written, in that case we need much less heap for the memstores. Do you think that having an option to share memstore between group of regions (rather than having many small, like 64mb or 32mb, memstores) would help here? Scaling so cluster can host 1M regions and beyond (50M regions?) Key: HBASE-11165 URL: https://issues.apache.org/jira/browse/HBASE-11165 Project: HBase Issue Type: Brainstorming Reporter: stack This discussion issue comes out of Co-locate Meta And Master HBASE-10569 and comments on the doc posted there. A user -- our Francis Liu -- needs to be able to scale a cluster to do 1M regions maybe even 50M later. This issue is about discussing how we will do that (or if not 50M on a cluster, how otherwise we can attain same end). More detail to follow. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11188) Inconsistent configuration for SchemaMetrics is always shown
[
https://issues.apache.org/jira/browse/HBASE-11188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jean-Daniel Cryans updated HBASE-11188:
---
Attachment: HBASE-11188-0.94.patch
Proposing this one-liner patch, passes both TestSchemaConfigured and
TestSchemaMetrics.
Inconsistent configuration for SchemaMetrics is always shown
--
Key: HBASE-11188
URL: https://issues.apache.org/jira/browse/HBASE-11188
Project: HBase
Issue Type: Bug
Components: metrics
Affects Versions: 0.94.19
Reporter: Jean-Daniel Cryans
Assignee: Jean-Daniel Cryans
Fix For: 0.94.20
Attachments: HBASE-11188-0.94-v2.patch, HBASE-11188-0.94.patch
Some users have been complaining about this message:
{noformat}
ERROR org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics:
Inconsistent configuration. Previous configuration for using table name in
metrics: true, new configuration: false
{noformat}
The interesting thing is that we see it with default configurations, which
made me think that some code path must have been passing the wrong thing. I
found that if SchemaConfigured is passed a null Configuration in its
constructor that it will then pass null to SchemaMetrics#configureGlobally
which will interpret useTableName as being false:
{code}
public static void configureGlobally(Configuration conf) {
if (conf != null) {
final boolean useTableNameNew =
conf.getBoolean(SHOW_TABLE_NAME_CONF_KEY, false);
setUseTableName(useTableNameNew);
} else {
setUseTableName(false);
}
}
{code}
It should be set to true since that's the new default, meaning we missed it
in HBASE-5671.
I found one code path that passes a null configuration, StoreFile.Reader
extends SchemaConfigured and uses the constructor that only passes a Path, so
the Configuration is set to null.
I'm planning on just passing true instead of false, fixing the problem for
almost everyone (those that disable this feature will get the error message).
IMO it's not worth more efforts since it's a 0.94-only problem and it's not
actually doing anything bad.
I'm closing both HBASE-10990 and HBASE-10946 as duplicates.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Assigned] (HBASE-11195) Potentially improve block locality during major compaction for old regions
[ https://issues.apache.org/jira/browse/HBASE-11195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Latham reassigned HBASE-11195: --- Assignee: churro morales Potentially improve block locality during major compaction for old regions -- Key: HBASE-11195 URL: https://issues.apache.org/jira/browse/HBASE-11195 Project: HBase Issue Type: Improvement Affects Versions: 0.94.19 Reporter: churro morales Assignee: churro morales Attachments: HBASE-11195-0.94.patch This might be a specific use case. But we have some regions which are no longer written to (due to the key). Those regions have 1 store file and they are very old, they haven't been written to in a while. We still use these regions to read from so locality would be nice. I propose putting a configuration option: something like hbase.hstore.min.locality.to.skip.major.compact [between 0 and 1] such that you can decide whether or not to skip major compaction for an old region with a single store file. I'll attach a patch, let me know what you guys think. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11127) Move security features into core
[ https://issues.apache.org/jira/browse/HBASE-11127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13998990#comment-13998990 ] Jerry He commented on HBASE-11127: -- Sounds good! Some additional suggestions: 1. We probably wan to have security controller flexibility and plug-ability. For example, elevate an interface/abstract class framework of its own. The current controller implementations are default. 2. Split the controller into master side and region server. The current AccessController contains everything, which seems to be crowded. Move security features into core Key: HBASE-11127 URL: https://issues.apache.org/jira/browse/HBASE-11127 Project: HBase Issue Type: Improvement Reporter: Andrew Purtell HBASE-11126 mentions concurrency issues we are running into as the security code increases in sophistication, due to current placement of coprocessor hooks, and proposes a solution to those issues with the expectation that security code remains outside of core in coprocessors. However, as an alternative we could consider moving all AccessController and VisibilityController related code into core. Worth discussing? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11171) More doc improvements on block cache options
[ https://issues.apache.org/jira/browse/HBASE-11171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-11171: -- Attachment: 11171v8.txt Test failed on linux. Fix. More doc improvements on block cache options Key: HBASE-11171 URL: https://issues.apache.org/jira/browse/HBASE-11171 Project: HBase Issue Type: Sub-task Components: documentation Reporter: stack Assignee: stack Attachments: 11171.txt, 11171v2.txt, 11171v2.txt, 11171v3.txt, 11171v4.txt, 11171v5.txt, 11171v6.txt, 11171v7.txt, 11171v8.txt, package-summary.html I have more doc improvement (no code changes for sure this time). Follow on from HBASE-11098. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HBASE-11184) why initialize the thread pool with one thread for closing store files
[ https://issues.apache.org/jira/browse/HBASE-11184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-11184. --- Resolution: Invalid Resolving as invalid. This kind of question best belongs on the user or dev lists. Yeah, the notiion is that you can ask to run with more closers. We don't list every config in the hbase-default.xml because it would overwhelm and confuse. Some configs are in code only. If you have gone to the trouble of reading the code, then you'll likely understand what the particular minor config can do for you. why initialize the thread pool with one thread for closing store files -- Key: HBASE-11184 URL: https://issues.apache.org/jira/browse/HBASE-11184 Project: HBase Issue Type: Improvement Components: HFile Affects Versions: 0.94.17 Reporter: Zhang Jingpeng Priority: Trivial when i read Store close() method i found the thread pool for closing store files with only one Thread. why initialize the thread pool with only one thread for closing store files? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10336) Remove deprecated usage of Hadoop HttpServer in InfoServer
[ https://issues.apache.org/jira/browse/HBASE-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13998630#comment-13998630 ] Ted Yu commented on HBASE-10336: Uploading onto review board would make review process smooth. Thanks Remove deprecated usage of Hadoop HttpServer in InfoServer -- Key: HBASE-10336 URL: https://issues.apache.org/jira/browse/HBASE-10336 Project: HBase Issue Type: Bug Affects Versions: 0.99.0 Reporter: Eric Charles Assignee: Eric Charles Attachments: HBASE-10336-1.patch, HBASE-10336-2.patch, HBASE-10336-3.patch, HBASE-10336-4.patch, HBASE-10336-5.patch, HBASE-10336-6.patch, HBASE-10336-7.patch, HBASE-10336-8.patch, HBASE-10336-9.patch, HBASE-10569-10.patch Recent changes in Hadoop HttpServer give NPE when running on hadoop 3.0.0-SNAPSHOT. This way we use HttpServer is deprecated and will probably be not fixed (see HDFS-5760). We'd better move to the new proposed builder pattern, which means we can no more use inheritance to build our nice InfoServer. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10835) DBE encode path improvements
[ https://issues.apache.org/jira/browse/HBASE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-10835: --- Attachment: (was: HBASE-10835_V4.patch) DBE encode path improvements Key: HBASE-10835 URL: https://issues.apache.org/jira/browse/HBASE-10835 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0 Attachments: HBASE-10835.patch, HBASE-10835_V2.patch, HBASE-10835_V3.patch, HBASE-10835_V4.patch Here 1st we write KVs (Cells) into a buffer and then passed to DBE encoder. Encoder again reads kvs one by one from the buffer and encodes and creates a new buffer. There is no need to have this model now. Previously we had option of no encode in disk and encode only in cache. At that time the read buffer from a HFile block was passed to this and encodes. So encode cell by cell can be done now. Making this change will need us to have a NoOp DBE impl which just do the write of a cell as it is with out any encoding. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HBASE-11139) BoundedPriorityBlockingQueue#poll() should check the return value from awaitNanos()
[
https://issues.apache.org/jira/browse/HBASE-11139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu reassigned HBASE-11139:
--
Assignee: Shengzhe Yao
BoundedPriorityBlockingQueue#poll() should check the return value from
awaitNanos()
---
Key: HBASE-11139
URL: https://issues.apache.org/jira/browse/HBASE-11139
Project: HBase
Issue Type: Bug
Affects Versions: 0.99.0
Reporter: Ted Yu
Assignee: Shengzhe Yao
Priority: Minor
Labels: noob
Attachments: HBASE-11139-v1.patch, HBASE-11139-v2.patch
nanos represents the timeout value.
{code}
while (queue.size() == 0 nanos 0) {
notEmpty.awaitNanos(nanos);
}
{code}
The return value from awaitNanos() should be checked - otherwise we may wait
for period longer than the timeout value.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11076) Update refguide on getting 0.94.x to run on hadoop 2.2.0+
[ https://issues.apache.org/jira/browse/HBASE-11076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gustavo Anatoly updated HBASE-11076: Status: Patch Available (was: Open) Update refguide on getting 0.94.x to run on hadoop 2.2.0+ - Key: HBASE-11076 URL: https://issues.apache.org/jira/browse/HBASE-11076 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Gustavo Anatoly Attachments: HBASE-11076.patch http://hbase.apache.org/book.html#d248e643 contains steps for rebuilding 0.94 code base to run on hadoop 2.2.0+ However, the files under src/main/java/org/apache/hadoop/hbase/protobuf/generated were produced by protoc 2.4.0 These files need to be regenerated. See http://search-hadoop.com/m/DHED4j7Um02/HBase+0.94+on+hadoop+2.2.0subj=Re+HBase+0+94+on+hadoop+2+2+0+2+4+0+ This issue is to update refguide with this regeneration step. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11125) Introduce a higher level interface for registering interest in coprocessor upcalls
[ https://issues.apache.org/jira/browse/HBASE-11125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13999681#comment-13999681 ] Nicolas Liochon commented on HBASE-11125: - I think it would work. It could also be more efficient, we would know when nobody cares of the event. Introduce a higher level interface for registering interest in coprocessor upcalls -- Key: HBASE-11125 URL: https://issues.apache.org/jira/browse/HBASE-11125 Project: HBase Issue Type: New Feature Reporter: Andrew Purtell Priority: Critical Fix For: 0.99.0, 1.0.0 We should introduce a higher level interface for managing the registration of 'user' code for execution from the low level hooks. It should not be necessary for coprocessor implementers to learn the universe of available low level hooks and the subtleties of their placement within HBase core code. Instead the higher level API should allow the implementer to describe their intent and then this API should choose the appropriate low level hook placement. A very desirable side effect is a layer of indirection between coprocessor implementers and the actual hooks. This will address the perennial complaint that the low level hooks change too much from release to release, as recently discussed during the RM panel at HBaseCon. If we try to avoid changing the particular placement and arguments of hook functions in response to those complaints, this can be an onerous constraint on necessary internals evolution. Instead we can direct coprocessor implementers to consider the new API and provide the same interface stability guarantees there as we do for client API, -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10915) Decouple region closing (HM and HRS) from ZK
[ https://issues.apache.org/jira/browse/HBASE-10915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-10915: Status: Patch Available (was: Open) Decouple region closing (HM and HRS) from ZK Key: HBASE-10915 URL: https://issues.apache.org/jira/browse/HBASE-10915 Project: HBase Issue Type: Sub-task Components: Consensus, Zookeeper Affects Versions: 0.99.0 Reporter: Mikhail Antonov Assignee: Mikhail Antonov Attachments: HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch Decouple region closing from ZK. Includes RS side (CloseRegionHandler), HM side (ClosedRegionHandler) and the code using (HRegionServer, RSRpcServices etc). May need small changes in AssignmentManager. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11165) Scaling so cluster can host 1M regions and beyond (50M regions?)
[ https://issues.apache.org/jira/browse/HBASE-11165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13999573#comment-13999573 ] Vladimir Rodionov commented on HBASE-11165: --- Single Master for 50M regions? Either it should be master-less or we need something similar to NN federation. Or FB Cell. Each cell is HBase cell-cluster and META cells create tree hierarchy. May be I am missing something? Scaling so cluster can host 1M regions and beyond (50M regions?) Key: HBASE-11165 URL: https://issues.apache.org/jira/browse/HBASE-11165 Project: HBase Issue Type: Brainstorming Reporter: stack This discussion issue comes out of Co-locate Meta And Master HBASE-10569 and comments on the doc posted there. A user -- our Francis Liu -- needs to be able to scale a cluster to do 1M regions maybe even 50M later. This issue is about discussing how we will do that (or if not 50M on a cluster, how otherwise we can attain same end). More detail to follow. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-8763) [BRAINSTORM] Combine MVCC and SeqId
[ https://issues.apache.org/jira/browse/HBASE-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey Zhong updated HBASE-8763: - Attachment: HBase MVCC LogSeqId Combined.pdf [BRAINSTORM] Combine MVCC and SeqId --- Key: HBASE-8763 URL: https://issues.apache.org/jira/browse/HBASE-8763 Project: HBase Issue Type: Improvement Components: regionserver Reporter: Enis Soztutar Assignee: Jeffrey Zhong Priority: Critical Attachments: HBase MVCC LogSeqId Combined.pdf, hbase-8736-poc.patch, hbase-8763-poc-v1.patch, hbase-8763-v1.patch, hbase-8763-v2.patch, hbase-8763_wip1.patch HBASE-8701 and a lot of recent issues include good discussions about mvcc + seqId semantics. It seems that having mvcc and the seqId complicates the comparator semantics a lot in regards to flush + WAL replay + compactions + delete markers and out of order puts. Thinking more about it I don't think we need a MVCC write number which is different than the seqId. We can keep the MVCC semantics, read point and smallest read points intact, but combine mvcc write number and seqId. This will allow cleaner semantics + implementation + smaller data files. We can do some brainstorming for 0.98. We still have to verify that this would be semantically correct, it should be so by my current understanding. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11168) [docs] Remove references to RowLocks in post 0.96 docs.
[ https://issues.apache.org/jira/browse/HBASE-11168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13999544#comment-13999544 ] Jean-Marc Spaggiari commented on HBASE-11168: - Definitively +1! Got asked recently if it was still bad to use rowlocks or if it was now correct... [docs] Remove references to RowLocks in post 0.96 docs. --- Key: HBASE-11168 URL: https://issues.apache.org/jira/browse/HBASE-11168 Project: HBase Issue Type: Bug Components: documentation Affects Versions: 0.95.0 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: 0.99.0, 0.96.3, 0.98.3 Attachments: hbase-11168.patch Row locks were removed in 0.95 by HBASE-7315 / HBASE-2332. There are a few vestiges of them in the docs. Remove. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11134) Add a -list-snapshots option to SnapshotInfo
[
https://issues.apache.org/jira/browse/HBASE-11134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13999303#comment-13999303
]
Hudson commented on HBASE-11134:
SUCCESS: Integrated in HBase-0.94-JDK7 #135 (See
[https://builds.apache.org/job/HBase-0.94-JDK7/135/])
HBASE-11134 Add a -list-snapshots option to SnapshotInfo (mbertozzi: rev
1594857)
*
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java
* /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
Add a -list-snapshots option to SnapshotInfo
Key: HBASE-11134
URL: https://issues.apache.org/jira/browse/HBASE-11134
Project: HBase
Issue Type: Improvement
Components: snapshots
Affects Versions: 0.99.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Trivial
Fix For: 0.99.0, 0.96.3, 0.94.20, 0.98.3
Attachments: HBASE-11134-v0.patch, HBASE-11134-v1.patch
Add a -list-snapshots option to SnapshotInfo to show all the snapshots
available. Also add a -remote-dir option to simplify the usage of
SnapshotInfo in case the snapshot dir is not the one of the current hbase
cluster
{code}
$ hbase org.apache.hadoop.hbase.snapshot.SnapshotInfo -list-snapshots
SNAPSHOT | CREATION TIME| TABLE NAME
foo | 2014-05-07T22:40:13 | testtb
bar | 2014-05-07T22:40:16 | testtb
$ hbase org.apache.hadoop.hbase.snapshot.SnapshotInfo -remote-dir
file:///backup/ -snapshot my_local_snapshot
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11156) Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
[ https://issues.apache.org/jira/browse/HBASE-11156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13998521#comment-13998521 ] Jiten commented on HBASE-11156: --- That's what i want to know, as you suggest to ask this issue on LIST. Can you please give me URL of list where i can ask it. Or please describe in detail, how to raise this issue on list. And kindly mention path of the log file. I am using HBase 0.96.1.1-cdh5.0.0 and Hadoop 2.3.0-cdh5.0.0. Thanks, Jiten Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available - Key: HBASE-11156 URL: https://issues.apache.org/jira/browse/HBASE-11156 Project: HBase Issue Type: Bug Components: Admin Affects Versions: 0.96.1.1 Reporter: Jiten Priority: Critical # hbase shell 2014-05-13 14:51:41,582 INFO [main] Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available HBase Shell; enter 'helpRETURN' for list of supported commands. Type exitRETURN to leave the HBase Shell Version 0.96.1.1-cdh5.0.0, rUnknown, Thu Mar 27 23:01:59 PDT 2014. Not able to create table in Hbase. Please help -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10247) Client promises about timestamps
[ https://issues.apache.org/jira/browse/HBASE-10247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1467#comment-1467 ] Enis Soztutar commented on HBASE-10247: --- Linking to 1.0 parent issue. It would be great to have this and HBASE-9905 and related HBASE-8763 to have a clear story. Client promises about timestamps Key: HBASE-10247 URL: https://issues.apache.org/jira/browse/HBASE-10247 Project: HBase Issue Type: Brainstorming Reporter: Lars Hofhansl Priority: Minor This is to start a discussion about timestamp promises declared per table of CF. For example if a client promises only monotonically increasing timestamps (or no custom set timestamps) and VERSIONS=1, we can aggressively and easily remove old versions of the same row/fam/col from the memstore before we flush, just by supplying a comparator that ignores the timestamp (i.e. two KV just differing by TS would be considered equal). That would increase the performance of counters significantly. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11186) Improve TestExportSnapshot verifications
[
https://issues.apache.org/jira/browse/HBASE-11186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14000134#comment-14000134
]
stack commented on HBASE-11186:
---
Took a quick look. LGTM. I applied the patch to trunk locally and the
refactored test failed with this:
{code}
Tests run: 9, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 303.551 sec
FAILURE!
testSnapshotWithRefsExportFileSystemState(org.apache.hadoop.hbase.snapshot.TestExportSnapshot)
Time elapsed: 23.841 sec FAILURE!
java.lang.AssertionError: expected:0 but was:1
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testExportFileSystemState(TestExportSnapshot.java:288)
at
org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testExportFileSystemState(TestExportSnapshot.java:260)
at
org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testSnapshotWithRefsExportFileSystemState(TestExportSnapshot.java:254)
at
org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testSnapshotWithRefsExportFileSystemState(TestExportSnapshot.java:234)
...
{code}
Did I do something wrong?
Improve TestExportSnapshot verifications
Key: HBASE-11186
URL: https://issues.apache.org/jira/browse/HBASE-11186
Project: HBase
Issue Type: Bug
Components: snapshots
Affects Versions: 0.99.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
Fix For: 0.99.0
Attachments: HBASE-11186-v0.patch
* Remove some code by using the utils that we already have in
SnapshotTestingUtil
* Add an Export with references for both v1 and v2 format
* add the verification on the actual number of files exported
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11059) ZK-less region assignment
[ https://issues.apache.org/jira/browse/HBASE-11059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13999039#comment-13999039 ] Konstantin Boudnik commented on HBASE-11059: [~jxiang]], could you please share the design doc with me as well? Thanks! ZK-less region assignment - Key: HBASE-11059 URL: https://issues.apache.org/jira/browse/HBASE-11059 Project: HBase Issue Type: Improvement Components: master, Region Assignment Reporter: Jimmy Xiang Assignee: Jimmy Xiang It seems that most people don't like region assignment with ZK (HBASE-5487), which causes many uncertainties. This jira is to support ZK-less region assignment. We need to make sure this patch doesn't break backward compatibility/rolling upgrade. -- This message was sent by Atlassian JIRA (v6.2#6252)
