[jira] [Created] (HBASE-19969) Improve FT in merge operation
Vladimir Rodionov created HBASE-19969: - Summary: Improve FT in merge operation Key: HBASE-19969 URL: https://issues.apache.org/jira/browse/HBASE-19969 Project: HBase Issue Type: Sub-task Reporter: Vladimir Rodionov -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-15227) HBase Backup Phase 3: Fault tolerance (client/server) support
[ https://issues.apache.org/jira/browse/HBASE-15227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov resolved HBASE-15227. --- Resolution: Fixed Done. > HBase Backup Phase 3: Fault tolerance (client/server) support > - > > Key: HBASE-15227 > URL: https://issues.apache.org/jira/browse/HBASE-15227 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Major > Labels: backup > Attachments: HBASE-15227-v3.patch, HBASE-15277-v1.patch > > > System must be tolerant to faults: > # Backup operations MUST be atomic (no partial completion state in the backup > system table) > # Process must detect any type of failures which can result in a data loss > (partial backup or partial restore) > # Proper system table state restore and cleanup must be done in case of a > failure > # Additional utility to repair backup system table and corresponding file > system cleanup must be implemented > h3. Backup > h4. General FT framework implementation > Before actual backup operation starts, snapshot of a backup system table is > taken and system table is updated with *ACTIVE_SNAPSHOT* flag. The flag will > be removed upon backup completion. > In case of *any* server-side failures, client catches errors/exceptions and > handles them: > # Cleans up backup destination (removes partial backup data) > # Cleans up any temporary data > # Deletes any active snapshots of a tables being backed up (during full > backup we snapshot tables) > # Restores backup system table from snapshot > # Deletes backup system table snapshot (we read snapshot name from backup > system table before) > In case of *any* client-side failures: > Before any backup or restore operation run we check backup system table on > *ACTIVE_SNAPSHOT*, if flag is present, operation aborts with a message that > backup repair tool (see below) must be run > h4. Backup repair tool > The command line tool *backup repair* which executes the following steps: > # Reads info of a last failed backup session > # Cleans up backup destination (removes partial backup data) > # Cleans up any temporary data > # Deletes any active snapshots of a tables being backed up (during full > backup we snapshot tables) > # Restores backup system table from snapshot > # Deletes backup system table snapshot (we read snapshot name from backup > system table before) > h4. Detection of a partial loss of data > h5. Full backup > Export snapshot operation (?). > We count files and check sizes before and after DistCp run > h5. Incremental backup > Conversion of WAL to HFiles, when WAL file is moved from active to archive > directory. The code is in place to handle this situation > During DistCp run (same as above) > h3. Restore > This operation does not modify backup system table and is idempotent. No > special FT is required. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20547) Restore from backup will fail if done from a different file system
Vladimir Rodionov created HBASE-20547: - Summary: Restore from backup will fail if done from a different file system Key: HBASE-20547 URL: https://issues.apache.org/jira/browse/HBASE-20547 Project: HBase Issue Type: Bug Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20630) B&R: Delete command enhancements
Vladimir Rodionov created HBASE-20630: - Summary: B&R: Delete command enhancements Key: HBASE-20630 URL: https://issues.apache.org/jira/browse/HBASE-20630 Project: HBase Issue Type: New Feature Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Make the command more useable. Currently, user needs to provide list of backup ids to delete. It would be nice to have more convenient options, such as: deleting all backups which are older than XXX days, etc -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20631) B&R: Merge command enhancements
Vladimir Rodionov created HBASE-20631: - Summary: B&R: Merge command enhancements Key: HBASE-20631 URL: https://issues.apache.org/jira/browse/HBASE-20631 Project: HBase Issue Type: New Feature Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Currently, merge supports only list of backup ids, which users must provide. Date range merges seem more convenient for users. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20729) B&R BackupLogCleaner must ignore ProcV2 WAL files
Vladimir Rodionov created HBASE-20729: - Summary: B&R BackupLogCleaner must ignore ProcV2 WAL files Key: HBASE-20729 URL: https://issues.apache.org/jira/browse/HBASE-20729 Project: HBase Issue Type: Bug Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov These are WAL files B&R does need for backup. The issue does not affect B&R functionality though. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21077) MR job launched by hbase incremental backup command failed with FileNotFoundException
Vladimir Rodionov created HBASE-21077: - Summary: MR job launched by hbase incremental backup command failed with FileNotFoundException Key: HBASE-21077 URL: https://issues.apache.org/jira/browse/HBASE-21077 Project: HBase Issue Type: Bug Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21219) Hbase incremental backup fails with null pointer exception
Vladimir Rodionov created HBASE-21219: - Summary: Hbase incremental backup fails with null pointer exception Key: HBASE-21219 URL: https://issues.apache.org/jira/browse/HBASE-21219 Project: HBase Issue Type: Bug Components: backup&restore Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 3.0.0 hbase backup create incremental hdfs:///bkpHbase_Test/bkpHbase_Test2 -t bkpHbase_Test2 2018-09-21 15:35:31,421 INFO [main] impl.TableBackupClient: Backup backup_1537524313995 started at 1537524331419. 2018-09-21 15:35:31,454 INFO [main] impl.IncrementalBackupManager: Execute roll log procedure for incremental backup ... 2018-09-21 15:35:32,985 ERROR [main] impl.TableBackupClient: Unexpected Exception : java.lang.NullPointerException java.lang.NullPointerException at org.apache.hadoop.hbase.backup.impl.IncrementalBackupManager.getLogFilesForNewBackup(IncrementalBackupManager.java:309) at org.apache.hadoop.hbase.backup.impl.IncrementalBackupManager.getIncrBackupLogFileMap(IncrementalBackupManager.java:103) at org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.execute(IncrementalTableBackupClient.java:276) at org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.backupTables(BackupAdminImpl.java:601) at org.apache.hadoop.hbase.backup.impl.BackupCommands$CreateCommand.execute(BackupCommands.java:347) at org.apache.hadoop.hbase.backup.BackupDriver.parseAndRun(BackupDriver.java:138) at org.apache.hadoop.hbase.backup.BackupDriver.doWork(BackupDriver.java:171) at org.apache.hadoop.hbase.backup.BackupDriver.run(BackupDriver.java:204) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.hbase.backup.BackupDriver.main(BackupDriver.java:179) 2018-09-21 15:35:32,989 ERROR [main] impl.TableBackupClient: BackupId=backup_1537524313995,startts=1537524331419,failedts=1537524332989,failedphase=PREPARE_INCREMENTAL,failedmessage=null 2018-09-21 15:35:57,167 ERROR [main] impl.TableBackupClient: Backup backup_1537524313995 failed. Backup session finished. Status: FAILURE 2018-09-21 15:35:57,175 ERROR [main] backup.BackupDriver: Error running command-line tool java.io.IOException: java.lang.NullPointerException at org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.execute(IncrementalTableBackupClient.java:281) at org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.backupTables(BackupAdminImpl.java:601) at org.apache.hadoop.hbase.backup.impl.BackupCommands$CreateCommand.execute(BackupCommands.java:347) at org.apache.hadoop.hbase.backup.BackupDriver.parseAndRun(BackupDriver.java:138) at org.apache.hadoop.hbase.backup.BackupDriver.doWork(BackupDriver.java:171) at org.apache.hadoop.hbase.backup.BackupDriver.run(BackupDriver.java:204) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.hbase.backup.BackupDriver.main(BackupDriver.java:179) Caused by: java.lang.NullPointerException at org.apache.hadoop.hbase.backup.impl.IncrementalBackupManager.getLogFilesForNewBackup(IncrementalBackupManager.java:309) at org.apache.hadoop.hbase.backup.impl.IncrementalBackupManager.getIncrBackupLogFileMap(IncrementalBackupManager.java:103) at org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.execute(IncrementalTableBackupClient.java:276) ... 7 more -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (HBASE-21457) BackupUtils#getWALFilesOlderThan refers to wrong FileSystem
[ https://issues.apache.org/jira/browse/HBASE-21457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov reopened HBASE-21457: --- Opened for addendum. > BackupUtils#getWALFilesOlderThan refers to wrong FileSystem > --- > > Key: HBASE-21457 > URL: https://issues.apache.org/jira/browse/HBASE-21457 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Janos Gub >Assignee: Ted Yu >Priority: Major > Fix For: 3.0.0 > > Attachments: 21457.v1.txt, 21457.v2.txt, 21457.v3.txt, 21457.v3.txt, > 21457.v4.txt > > > Janos reported seeing backup test failure when testing a local HDFS for WALs > while using WASB/ADLS only for store files. > Janos spotted the code in BackupUtils#getWALFilesOlderThan which uses HBase > root dir for retrieving WAL files. > We should use the helper methods from CommonFSUtils. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21688) Address WAL filesystem issues
Vladimir Rodionov created HBASE-21688: - Summary: Address WAL filesystem issues Key: HBASE-21688 URL: https://issues.apache.org/jira/browse/HBASE-21688 Project: HBase Issue Type: Bug Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-21457) BackupUtils#getWALFilesOlderThan refers to wrong FileSystem
[ https://issues.apache.org/jira/browse/HBASE-21457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov resolved HBASE-21457. --- Resolution: Fixed > BackupUtils#getWALFilesOlderThan refers to wrong FileSystem > --- > > Key: HBASE-21457 > URL: https://issues.apache.org/jira/browse/HBASE-21457 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Janos Gub >Assignee: Ted Yu >Priority: Major > Fix For: 3.0.0 > > Attachments: 21457.v1.txt, 21457.v2.txt, 21457.v3.txt, 21457.v3.txt, > 21457.v4.txt, HBASE-21457.add.patch > > > Janos reported seeing backup test failure when testing a local HDFS for WALs > while using WASB/ADLS only for store files. > Janos spotted the code in BackupUtils#getWALFilesOlderThan which uses HBase > root dir for retrieving WAL files. > We should use the helper methods from CommonFSUtils. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (HBASE-21688) Address WAL filesystem issues
[ https://issues.apache.org/jira/browse/HBASE-21688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov reopened HBASE-21688: --- Opened for amendment. > Address WAL filesystem issues > - > > Key: HBASE-21688 > URL: https://issues.apache.org/jira/browse/HBASE-21688 > Project: HBase > Issue Type: Bug >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Major > Fix For: 3.0.0 > > Attachments: HBASE-21688-v1.patch > > > Scan and fix code base to use new way of instantiating WAL File System. > https://issues.apache.org/jira/browse/HBASE-21457?focusedCommentId=16734688&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16734688 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21936) Disable split/merge of a table before taking snapshot
Vladimir Rodionov created HBASE-21936: - Summary: Disable split/merge of a table before taking snapshot Key: HBASE-21936 URL: https://issues.apache.org/jira/browse/HBASE-21936 Project: HBase Issue Type: Bug Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov https://issues.apache.org/jira/browse/HBASE-17942 has introduced per table split/merge enablement. This new feature should be used during table's snapshot to avoid failing snapshots due to concurrent splits/merges. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-21936) Disable split/merge of a table during snapshot
[ https://issues.apache.org/jira/browse/HBASE-21936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov resolved HBASE-21936. --- Resolution: Invalid > Disable split/merge of a table during snapshot > -- > > Key: HBASE-21936 > URL: https://issues.apache.org/jira/browse/HBASE-21936 > Project: HBase > Issue Type: Bug >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Major > Fix For: 3.0.0, 2.2.0, 2.3.0 > > Attachments: HBASE-21936-master-v1.patch > > > https://issues.apache.org/jira/browse/HBASE-17942 has introduced per table > split/merge enablement. This new feature should be used during table's > snapshot to avoid failure due to concurrent splits/merges. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-11854) MetricsRegion.updateScanNext takes 20% of overall RegionScannerImpl.nextRaw call
Vladimir Rodionov created HBASE-11854: - Summary: MetricsRegion.updateScanNext takes 20% of overall RegionScannerImpl.nextRaw call Key: HBASE-11854 URL: https://issues.apache.org/jira/browse/HBASE-11854 Project: HBase Issue Type: Bug Components: metrics, Performance, Scanners Affects Versions: 0.98.6 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov >From our internal profiling we were surprised by amount of time spent in this >call. Further investigation is needed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11898) CoprocessorHost.Environment should cache class loader instance
Vladimir Rodionov created HBASE-11898: - Summary: CoprocessorHost.Environment should cache class loader instance Key: HBASE-11898 URL: https://issues.apache.org/jira/browse/HBASE-11898 Project: HBase Issue Type: Bug Components: Coprocessors Affects Versions: 0.98.6 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 0.98.7 HBASE-9941 fix causes additional overhead... It looks like for every invocation it does a getClassLoader call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-11936) IsolationLevel must be attribute of a Query not a Scan
Vladimir Rodionov created HBASE-11936: - Summary: IsolationLevel must be attribute of a Query not a Scan Key: HBASE-11936 URL: https://issues.apache.org/jira/browse/HBASE-11936 Project: HBase Issue Type: Bug Components: Client Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov The Get operation is implemented in HBase as a Scan. The default isolation level for Scan is READ_COMMITTED. The API to change the isolation level is part of Scan class and there is no way for Get operation to change this from default. We should move this API up to Query (which is a parent of both: Scan and Get). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-11965) Optimize locking in HRegion
Vladimir Rodionov created HBASE-11965: - Summary: Optimize locking in HRegion Key: HBASE-11965 URL: https://issues.apache.org/jira/browse/HBASE-11965 Project: HBase Issue Type: Bug Components: Performance Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Described in this thread: http://mail-archives.apache.org/mod_mbox/hbase-dev/201409.mbox/%3CCAAg3a2qmWJtQbdAk7PrX%2BW8SZWxsYhNggM5f2RNkGTXUri34YQ%40mail.gmail.com%3E *startRegionOperation* and *closeRegionOperation* in HRegion acquires and releases region-wide lock. The locking happens for every mutation, read and scan next. This is a serious bottleneck if we do: * Mutli - get on a same region. * Run multiple scanners on same region. * Do scan during compaction. * Access region simultaneously from different threads (most generic case) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12020) String formatting on each RPC Invoke
Vladimir Rodionov created HBASE-12020: - Summary: String formatting on each RPC Invoke Key: HBASE-12020 URL: https://issues.apache.org/jira/browse/HBASE-12020 Project: HBase Issue Type: Bug Components: IPC/RPC Affects Versions: 0.94.23 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Priority: Minor Fix For: 0.94.24 ExecRPCInvoker.inoker - debug call needs to be wrapped by a disabled check. For each invocation, the Bytes.toStringBinary is being called. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12022) Payloads on Failure attempt to serialize the byte[] into strings...
Vladimir Rodionov created HBASE-12022: - Summary: Payloads on Failure attempt to serialize the byte[] into strings... Key: HBASE-12022 URL: https://issues.apache.org/jira/browse/HBASE-12022 Project: HBase Issue Type: Bug Components: IPC/RPC Affects Versions: 0.94.23 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Priority: Trivial Fix For: 0.94.24 HBaseServer$HandlerRun LOG.debug(getName()", call "+call": error: " + e, e); errorClass = e.getClass().getName(); error = StringUtils.stringifyException(e); Whether or not we are at a debug level, we will serialize the call... Our call has a shit load of KVPairs as byte[]... These need to be at least wrapped in LOG.isDebugEnabled blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12023) HRegion.applyFamilyMapToMemstore creates too many iterator objects...
Vladimir Rodionov created HBASE-12023: - Summary: HRegion.applyFamilyMapToMemstore creates too many iterator objects... Key: HBASE-12023 URL: https://issues.apache.org/jira/browse/HBASE-12023 Project: HBase Issue Type: Bug Affects Versions: 0.94.23, 0.98.5 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Priority: Minor Attachments: applyFamilyMapToMemstore (1).tiff for ... loop (creates iterator) inside another for loop. Produces a lot of garbage. See attached picture. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12031) Parallel Scanners inside Region
Vladimir Rodionov created HBASE-12031: - Summary: Parallel Scanners inside Region Key: HBASE-12031 URL: https://issues.apache.org/jira/browse/HBASE-12031 Project: HBase Issue Type: New Feature Components: Performance, Scanners Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov This JIRA to improve performance of multiple scanners running on a same region in parallel. The scenarios where we will get the performance benefits: * New TableInputFormat with input splits smaller than HBase Region. * Scanning during compaction (Compaction scanner and application scanner over the same Region). Some JIRAs related to this one: https://issues.apache.org/jira/browse/HBASE-7336 https://issues.apache.org/jira/browse/HBASE-5979 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12061) Ensure LOG.isXXXEnabled() around LOG.XXX() calls
Vladimir Rodionov created HBASE-12061: - Summary: Ensure LOG.isXXXEnabled() around LOG.XXX() calls Key: HBASE-12061 URL: https://issues.apache.org/jira/browse/HBASE-12061 Project: HBase Issue Type: Sub-task Components: Performance Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov The common anti-pattern in Java is using LOG.xxx() calls w/o wrapping them into LOG.isXXXEnabled(). This result in unnecessary object serialization/ string concatenation/ garbage object production even if XXX lof=g level is disabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12090) Bytes: more Unsafe, more Fatster
Vladimir Rodionov created HBASE-12090: - Summary: Bytes: more Unsafe, more Fatster Key: HBASE-12090 URL: https://issues.apache.org/jira/browse/HBASE-12090 Project: HBase Issue Type: Improvement Components: Performance Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Additional optimizations to *org.apache.hadoop.hbase.util.Bytes*: * New version of compareTo method. * New versions for primitive converters : putXXX/toXXX. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-12090) Bytes: more Unsafe, more Faster
[ https://issues.apache.org/jira/browse/HBASE-12090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov reopened HBASE-12090: --- Opened for addendum. > Bytes: more Unsafe, more Faster > > > Key: HBASE-12090 > URL: https://issues.apache.org/jira/browse/HBASE-12090 > Project: HBase > Issue Type: Improvement > Components: Performance >Affects Versions: 0.94.23, 0.98.6 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0, 0.98.7, 0.94.24, 0.99.1 > > Attachments: 12090-v1.1.txt, 12090-v3.txt, HBASE-12090.2.patch, > HBASE-12090.patch > > > Additional optimizations to *org.apache.hadoop.hbase.util.Bytes*: > * New version of compareTo method. > * New versions for primitive converters : putXXX/toXXX. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12190) Split thread pool larger than one results in fatal failure during region splits
Vladimir Rodionov created HBASE-12190:
-
Summary: Split thread pool larger than one results in fatal
failure during region splits
Key: HBASE-12190
URL: https://issues.apache.org/jira/browse/HBASE-12190
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 2.0.0
Reporter: Vladimir Rodionov
I have been playing with different sizes for long/short compaction and split
thread poll sizes. It is definitely clear, that any size of split thread pool
large than 1 (one) results in a fatal errors during region splits.
Here is the log file snippet of a local test case :
{code}
2014-10-07 11:04:48,441
[INFO|org.apache.hadoop.hbase.regionserver.HRegionFileSystem|HRegionFileSystem]
The
hdfs://localhost:50215/user/vrodionov/hbase/data/default/TABLE_A/0ace9f564c05043a45cb44b2867432f8/.splits
directory exists. Hence deleting it to recreate it
put: 23
2014-10-07 11:04:48,697 [INFO|BlockStateChange|BlockManager] BLOCK*
addStoredBlock: blockMap updated: 127.0.0.1:50216 is added to
blk_1073741862_1038{blockUCState=COMMITTED, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[[DISK]DS-0b95bff8-1354-42cc-bd16-a5eafcb5bb27:NORMAL|RBW]]}
size 22106983
2014-10-07 11:04:49,100
[INFO|org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher|DefaultStoreFlusher]
Flushed, sequenceid=495, memsize=63.2 M, hasBloomFilter=true, into tmp file
hdfs://localhost:50215/user/vrodionov/hbase/data/default/TABLE_A/0ace9f564c05043a45cb44b2867432f8/.tmp/8d746dd585df4669a505fe2ba0ddafe1
2014-10-07 11:04:49,110
[INFO|org.apache.hadoop.hbase.regionserver.HStore|HStore] Added
hdfs://localhost:50215/user/vrodionov/hbase/data/default/TABLE_A/0ace9f564c05043a45cb44b2867432f8/fam_a/8d746dd585df4669a505fe2ba0ddafe1,
entries=36, sequenceid=495, filesize=21.1 M
2014-10-07 11:04:49,111
[INFO|org.apache.hadoop.hbase.regionserver.HRegion|HRegion] Finished memstore
flush of ~63.17 MB/6624, currentsize=42.11 MB/4416 for region
TABLE_A,row00037607,1412705075738.0ace9f564c05043a45cb44b2867432f8. in
674ms, sequenceid=495, compaction requested=true
2014-10-07 11:04:49,111
[INFO|org.apache.hadoop.hbase.regionserver.HRegion|HRegion] Started memstore
flush for
TABLE_A,row00037607,1412705075738.0ace9f564c05043a45cb44b2867432f8.,
current region memstore size 42.11 MB
2014-10-07 11:04:49,267 [INFO|BlockStateChange|BlockManager] BLOCK*
addStoredBlock: blockMap updated: 127.0.0.1:50216 is added to
blk_1073741863_1039{blockUCState=COMMITTED, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[[DISK]DS-414e0e79-1c18-4469-9458-6842a0b96131:NORMAL|RBW]]}
size 14745108
2014-10-07 11:04:49,669
[INFO|org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher|DefaultStoreFlusher]
Flushed, sequenceid=514, memsize=42.1 M, hasBloomFilter=true, into tmp file
hdfs://localhost:50215/user/vrodionov/hbase/data/default/TABLE_A/0ace9f564c05043a45cb44b2867432f8/.tmp/785a7048b1b849509e3e622cdaceb033
2014-10-07 11:04:49,682
[INFO|org.apache.hadoop.hbase.regionserver.HStore|HStore] Added
hdfs://localhost:50215/user/vrodionov/hbase/data/default/TABLE_A/0ace9f564c05043a45cb44b2867432f8/fam_a/785a7048b1b849509e3e622cdaceb033,
entries=24, sequenceid=514, filesize=14.1 M
2014-10-07 11:04:49,683
[INFO|org.apache.hadoop.hbase.regionserver.HRegion|HRegion] Finished memstore
flush of ~42.11 MB/4416, currentsize=0 B/0 for region
TABLE_A,row00037607,1412705075738.0ace9f564c05043a45cb44b2867432f8. in
572ms, sequenceid=514, compaction requested=true
2014-10-07 11:04:49,686
[INFO|org.apache.hadoop.hbase.regionserver.HStore|HStore] Closed fam_a
2014-10-07 11:04:49,687
[INFO|org.apache.hadoop.hbase.regionserver.HRegion|HRegion] Closed
TABLE_A,row00037607,1412705075738.0ace9f564c05043a45cb44b2867432f8.
2014-10-07 11:04:49,687
[WARN|org.apache.hadoop.hbase.regionserver.HRegion|HRegion] Region
TABLE_A,row00037607,1412705075738.0ace9f564c05043a45cb44b2867432f8. already
closed
2014-10-07 11:04:49,688
[INFO|org.apache.hadoop.hbase.regionserver.SplitRequest|SplitRequest] Running
rollback/cleanup of failed split of
TABLE_A,row00037607,1412705075738.0ace9f564c05043a45cb44b2867432f8.; Failed
to close region: already closed by another thread
java.io.IOException: Failed to close region: already closed by another thread
at
org.apache.hadoop.hbase.regionserver.SplitTransaction.(SplitTransaction.java:190)
at
org.apache.hadoop.hbase.regionserver.SplitRequest.run(SplitRequest.java:65)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-10-07 11:04:49,690
[ERROR|org.apache.hadoop.hbase.master.AssignmentManager|AssignmentManager]
Failed to transition region from {0ace9f564c0
[jira] [Created] (HBASE-12657) The Region is not being split and far exceeds the desired maximum size.
Vladimir Rodionov created HBASE-12657:
-
Summary: The Region is not being split and far exceeds the desired
maximum size.
Key: HBASE-12657
URL: https://issues.apache.org/jira/browse/HBASE-12657
Project: HBase
Issue Type: Bug
Components: Compaction
Affects Versions: 0.99.2, 0.94.25, 0.98.8
Reporter: Vladimir Rodionov
Assignee: Vladimir Rodionov
Fix For: 1.0.0, 2.0.0, 0.94.26, 0.98.9
We are seeing this behavior when creating indexes in one of our environment.
When an index is being created, most of the "requests" go into a single region.
The amount of time to create an index seems to take longer than usual and it
can take days for the regions to compact and split after the index is created.
Here is a du of the HBase index table:
{code}
-bash-4.1$ sudo -su hdfs hadoop fs -du /hbase/43681
705 /hbase/43681/.tableinfo.01
0/hbase/43681/.tmp
27981697293 /hbase/43681/0492e22092e21d35fca8e779b21ec797
539687093/hbase/43681/832298c4e975fc47210feb6bac3d2f71
560660531/hbase/43681/be9bdb3bdf9365afe5fe90db4247d82c
7081938297 /hbase/43681/cd440e524f96fbe0719b2fe969848560
6297860287 /hbase/43681/dc893a2d8daa08c689dc69e6bb2c5b50
7189607722 /hbase/43681/ffbceaea5e2f142dbe6cd4cbeacc00e8
...
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-12720) Make InternalScan Public
Vladimir Rodionov created HBASE-12720: - Summary: Make InternalScan Public Key: HBASE-12720 URL: https://issues.apache.org/jira/browse/HBASE-12720 Project: HBase Issue Type: Improvement Affects Versions: 0.98.9, 0.94.25 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 1.0.0, 2.0.0, 0.94.26, 0.98.10 This is the request from sophisticated users :) Rationale: We would like the internal scan to be made available so we can see what is just in the MemStore (or in store files only). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12748) RegionCoprocessorHost.execOperation creates too many iterator objects
Vladimir Rodionov created HBASE-12748: - Summary: RegionCoprocessorHost.execOperation creates too many iterator objects Key: HBASE-12748 URL: https://issues.apache.org/jira/browse/HBASE-12748 Project: HBase Issue Type: Sub-task Affects Versions: 0.98.9, 0.94.25 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 1.0.0, 2.0.0, 0.94.26, 0.98.10 This is typical pattern of enhanced for ... loop usage in a hot code path. For every HBase operation it instantiates iterator for coprocessor list twice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13761) Optimize FuzzyRowFilter
Vladimir Rodionov created HBASE-13761: - Summary: Optimize FuzzyRowFilter Key: HBASE-13761 URL: https://issues.apache.org/jira/browse/HBASE-13761 Project: HBase Issue Type: Improvement Components: Filters Affects Versions: 0.98.13, 1.1.0, 2.0.0 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Priority: Minor Fix For: 2.0.0, 0.98.14, 1.1.1 FuzzyRowFilter has some room for improvements: a lot of byte-by-byte arithmetic, non-efficient algorithm of selecting next candidate row etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13865) Default value of hbase.hregion.memstore.block.multiplier in HBase book is wrong
Vladimir Rodionov created HBASE-13865: - Summary: Default value of hbase.hregion.memstore.block.multiplier in HBase book is wrong Key: HBASE-13865 URL: https://issues.apache.org/jira/browse/HBASE-13865 Project: HBase Issue Type: Bug Reporter: Vladimir Rodionov Priority: Trivial Its 4 in the book and 2 in a current master. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13866) Add end point coprocessor to the section hbase.coprocessor.region.classes in HBase book
Vladimir Rodionov created HBASE-13866:
-
Summary: Add end point coprocessor to the section
hbase.coprocessor.region.classes in HBase book
Key: HBASE-13866
URL: https://issues.apache.org/jira/browse/HBASE-13866
Project: HBase
Issue Type: Bug
Components: documentation
Reporter: Vladimir Rodionov
Priority: Trivial
{quote}
hbase.coprocessor.region.classes
Description
A comma-separated list of Coprocessors that are loaded by default on all
tables. For any override coprocessor method, these classes will be called in
order. After implementing your own Coprocessor, just put it in HBase’s
classpath and add the fully qualified class name here. A coprocessor can also
be loaded on demand by setting HTableDescriptor.
{quote}
This must be more specific: not Coprocessors, but Region observers and *end
point coprocessors*.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-13867) Add endpoint coprocessor guide to HBase book
Vladimir Rodionov created HBASE-13867: - Summary: Add endpoint coprocessor guide to HBase book Key: HBASE-13867 URL: https://issues.apache.org/jira/browse/HBASE-13867 Project: HBase Issue Type: Task Components: Coprocessors, documentation Reporter: Vladimir Rodionov Endpoint coprocessors are very poorly documented. Coprocessor section of HBase book must be updated either with its own endpoint coprocessors HOW-TO guide or, at least, with the link(s) to some other guides. There is good description here: http://www.3pillarglobal.com/insights/hbase-coprocessors -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-13868) Correct "Disable automatic splitting" section is HBase book
Vladimir Rodionov created HBASE-13868:
-
Summary: Correct "Disable automatic splitting" section is HBase
book
Key: HBASE-13868
URL: https://issues.apache.org/jira/browse/HBASE-13868
Project: HBase
Issue Type: Task
Components: documentation
Affects Versions: 2.0.0
Reporter: Vladimir Rodionov
Priority: Trivial
This recommendation is not correct for
*IncreasingToUpperBoundRegionSplitPolicy* (which is default now)
{quote}
Disable Automatic Splitting
To disable automatic splitting, set hbase.hregion.max.filesize to a very large
value, such as 100 GB It is not recommended to set it to its absolute maximum
value of Long.MAX_VALUE.
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-13869) Fix typo in HBase book
Vladimir Rodionov created HBASE-13869:
-
Summary: Fix typo in HBase book
Key: HBASE-13869
URL: https://issues.apache.org/jira/browse/HBASE-13869
Project: HBase
Issue Type: Task
Components: documentation
Affects Versions: 2.0.0
Reporter: Vladimir Rodionov
Priority: Trivial
Typo in section's title:
{quote}
42.1.4. Variangle Length or Fixed Length Rowkeys
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-13882) Fix RegionSplitPolicy section in HBase book
Vladimir Rodionov created HBASE-13882:
-
Summary: Fix RegionSplitPolicy section in HBase book
Key: HBASE-13882
URL: https://issues.apache.org/jira/browse/HBASE-13882
Project: HBase
Issue Type: Bug
Components: documentation
Reporter: Vladimir Rodionov
Priority: Trivial
65.4.1. Custom Split Policies
The section starts with the following statement:
{quote}
ou can override the default split policy using a custom RegionSplitPolicy(HBase
0.94+). Typically a custom split policy should extend HBase’s default split
policy: IncreasingToUpperBoundRegionSplitPolicy.
{quote}
There is type above as well.
Then if we scroll down a little bit:
{quote}
The default split policy can be overwritten using a custom
RegionSplitPolicy(HBase 0.94+). Typically a custom split policy should extend
HBase’s default split policy: ConstantSizeRegionSplitPolicy.
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-13883) Fix Memstore Flush section in HBase book
Vladimir Rodionov created HBASE-13883:
-
Summary: Fix Memstore Flush section in HBase book
Key: HBASE-13883
URL: https://issues.apache.org/jira/browse/HBASE-13883
Project: HBase
Issue Type: Bug
Components: documentation
Reporter: Vladimir Rodionov
{quote}
65.7.2. MemStore Flush
A MemStore flush can be triggered under any of the conditions listed below. The
minimum flush unit is per region, not at individual MemStore level.
1. When a MemStore reaches the size specified by
hbase.hregion.memstore.flush.size, all MemStores that belong to its region will
be flushed out to disk.
2. When the overall MemStore usage reaches the value specified by
hbase.regionserver.global.memstore.upperLimit, MemStores from various regions
will be flushed out to disk to reduce overall MemStore usage in a RegionServer.
The flush order is based on the descending order of a region’s MemStore usage.
Regions will have their MemStores flushed until the overall MemStore usage
drops to or slightly below hbase.regionserver.global.memstore.lowerLimit.
3. When the number of WAL per region server reaches the value specified in
hbase.regionserver.max.logs, MemStores from various regions will be flushed out
to disk to reduce WAL count. The flush order is based on time. Regions with the
oldest MemStores are flushed first until WAL count drops below
hbase.regionserver.max.logs.
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-13884) Fix Compactions section in HBase book
Vladimir Rodionov created HBASE-13884:
-
Summary: Fix Compactions section in HBase book
Key: HBASE-13884
URL: https://issues.apache.org/jira/browse/HBASE-13884
Project: HBase
Issue Type: Bug
Components: documentation
Reporter: Vladimir Rodionov
Priority: Trivial
http://hbase.apache.org/book.html#_compaction
{quote}
Being Stuck
When the MemStore gets too large, it needs to flush its contents to a
StoreFile. However, a Store can only have hbase.hstore.blockingStoreFiles
files, so the MemStore needs to wait for the number of StoreFiles to be reduced
by one or more compactions. However, if the MemStore grows larger than
hbase.hregion.memstore.flush.size, it is not able to flush its contents to a
StoreFile. If the MemStore is too large and the number of StoreFiles is also
too high, the algorithm is said to be "stuck". The compaction algorithm checks
for this "stuck" situation and provides mechanisms to alleviate it.
{quote}
According to source code, this "stuck" situation has nothingg to do with
MemStore size.
{code}
// Stuck and not compacting enough (estimate). It is not guaranteed that we
will be
// able to compact more if stuck and compacting, because ratio policy
excludes some
// non-compacting files from consideration during compaction (see
getCurrentEligibleFiles).
int futureFiles = filesCompacting.isEmpty() ? 0 : 1;
boolean mayBeStuck = (candidateFiles.size() - filesCompacting.size() +
futureFiles)
>= storeConfigInfo.getBlockingFileCount();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-13890) Get/Scan from MemStore only (Client API)
Vladimir Rodionov created HBASE-13890: - Summary: Get/Scan from MemStore only (Client API) Key: HBASE-13890 URL: https://issues.apache.org/jira/browse/HBASE-13890 Project: HBase Issue Type: New Feature Components: API, Client, Scanners Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov This is short-circuit read for get/scan when recent data (version) of a cell can be found only in MemStore (with very high probability). Good examples are: Atomic counters and appends. This feature will allow to bypass completely store file scanners and improve performance and latency. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-11182) Store backup information in a manifest file using protobuff format
[ https://issues.apache.org/jira/browse/HBASE-11182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov resolved HBASE-11182. --- Resolution: Won't Fix Closing this JIRA as not relevant one to a current Backup/Restore roadmap. > Store backup information in a manifest file using protobuff format > -- > > Key: HBASE-11182 > URL: https://issues.apache.org/jira/browse/HBASE-11182 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.99.0 >Reporter: Jerry He >Assignee: Enoch Hsu > > A manifest file is used to store information about a backup image such as: > Table Name > Type: Full or Incremental > Size > Timestamp Info > State Info: Converted, Merged, Compacted, etc. > Dependency Lineage -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14030) Backup/Restore Phase 1
Vladimir Rodionov created HBASE-14030: - Summary: Backup/Restore Phase 1 Key: HBASE-14030 URL: https://issues.apache.org/jira/browse/HBASE-14030 Project: HBase Issue Type: Umbrella Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov This is the umbrella ticket for Backup/Restore Phase 1. See HBase-7912 design doc for the phase description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14031) Backup/Restore Phase 1: Abstract DistCp in incremental backup
Vladimir Rodionov created HBASE-14031: - Summary: Backup/Restore Phase 1: Abstract DistCp in incremental backup Key: HBASE-14031 URL: https://issues.apache.org/jira/browse/HBASE-14031 Project: HBase Issue Type: Task Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 2.0.0 Abstract DistCp (incremental backup) to support non-M/R based implementations. Provide M/R implementation. DistCp is used to copy WAL files during incremental backup. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14032) Backup/Restore Phase 1: Abstract SnapshotCopy (full backup)
Vladimir Rodionov created HBASE-14032: - Summary: Backup/Restore Phase 1: Abstract SnapshotCopy (full backup) Key: HBASE-14032 URL: https://issues.apache.org/jira/browse/HBASE-14032 Project: HBase Issue Type: Task Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Abstract SnapshotCopy (full backup) to support non-M/R based implementations. Provide M/R implementation. SnapshotCopy is used to copy snapshot’s data during full backup operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14033) Backup/Restore Phase1: Abstract WALPlayer (incremental restore)
Vladimir Rodionov created HBASE-14033: - Summary: Backup/Restore Phase1: Abstract WALPlayer (incremental restore) Key: HBASE-14033 URL: https://issues.apache.org/jira/browse/HBASE-14033 Project: HBase Issue Type: Task Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 2.0.0 Abstract WALPlayer (incremental restore) to support non-M/R based implementations. Provide M/R implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14034) HBase Backup/Restore Phase 1: Abstract Coordination manager (Zk) operations
Vladimir Rodionov created HBASE-14034: - Summary: HBase Backup/Restore Phase 1: Abstract Coordination manager (Zk) operations Key: HBASE-14034 URL: https://issues.apache.org/jira/browse/HBASE-14034 Project: HBase Issue Type: Task Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 2.0.0 Abstract Coordination manager (Zk) operations. See org.apache.hadoop.hbase.coordination package for references. Provide Zookeeper implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14035) HBase Backup/Restore Phase 1: hbase:backup - backup system table
Vladimir Rodionov created HBASE-14035: - Summary: HBase Backup/Restore Phase 1: hbase:backup - backup system table Key: HBASE-14035 URL: https://issues.apache.org/jira/browse/HBASE-14035 Project: HBase Issue Type: Task Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 2.0.0 *hbase:backup* - move all backup meta info from Zk (coordination manager) to hbase system table. Do not use Zk (coordination manager) as a persistent storage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14036) HBase Backup/Restore Phase 1: Custom WAL archive cleaner
Vladimir Rodionov created HBASE-14036: - Summary: HBase Backup/Restore Phase 1: Custom WAL archive cleaner Key: HBASE-14036 URL: https://issues.apache.org/jira/browse/HBASE-14036 Project: HBase Issue Type: Task Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 2.0.0 Custom WAL archive cleaner (BackupLogCleaner). We need to keep WAL files in archive until they either get copied over to backup destination during an incremental backup or full backup (for ALL tables) happens. This is tricky, but is doable. Backup-aware WAL archiver cleaner should consult hbase:backup to determine if WAL file is safe to purge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14037) Deletion of a table from backup set results int RTE during next backup
Vladimir Rodionov created HBASE-14037: - Summary: Deletion of a table from backup set results int RTE during next backup Key: HBASE-14037 URL: https://issues.apache.org/jira/browse/HBASE-14037 Project: HBase Issue Type: Bug Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 2.0.0 Deletion of a table with backup history (has Zk node) results in RuntimeException on all subsequent backup requests. See: BackupClient.requestBackup. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14038) Incremental backup list set is ignored during backup
Vladimir Rodionov created HBASE-14038: - Summary: Incremental backup list set is ignored during backup Key: HBASE-14038 URL: https://issues.apache.org/jira/browse/HBASE-14038 Project: HBase Issue Type: Bug Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 2.0.0 BUG: during incremental backup, provided table list is ignored and replaced with the set of tables which have been already backuped before. Test case: backup T1, T2, T3, then request incremental backup for T1, T2 => T3 will be included as well. See: BackupClient.requestBackup. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14039) BackupHandler.deleteSnapshot MUST use HBase Snapshot API
Vladimir Rodionov created HBASE-14039: - Summary: BackupHandler.deleteSnapshot MUST use HBase Snapshot API Key: HBASE-14039 URL: https://issues.apache.org/jira/browse/HBASE-14039 Project: HBase Issue Type: Improvement Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 2.0.0 BackupHandler.deleteSnapshot MUST use HBase API for that (HBaseAdmin) - not direct FS access (deleting snapshot folder may be not enough?). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14040) Small refactoring in BackupHandler
Vladimir Rodionov created HBASE-14040: - Summary: Small refactoring in BackupHandler Key: HBASE-14040 URL: https://issues.apache.org/jira/browse/HBASE-14040 Project: HBase Issue Type: Improvement Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 2.0.0 Move distributed log roll procedure call to BackupHandler.call from IncrementalBackupManager.getLogFilesForNewBackup. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-14034) HBase Backup/Restore Phase 1: Abstract Coordination manager (Zk) operations
[ https://issues.apache.org/jira/browse/HBASE-14034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov resolved HBASE-14034. --- Resolution: Invalid Closing this as 'invalid'. There will be no CM- related code, all Zk operations will be moved into HBas (*hbase:backup* table). > HBase Backup/Restore Phase 1: Abstract Coordination manager (Zk) operations > --- > > Key: HBASE-14034 > URL: https://issues.apache.org/jira/browse/HBASE-14034 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > > Abstract Coordination manager (Zk) operations. See > org.apache.hadoop.hbase.coordination package for references. Provide > Zookeeper implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-14034) HBase Backup/Restore Phase 1: Abstract Coordination manager (Zk) operations
[ https://issues.apache.org/jira/browse/HBASE-14034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov reopened HBASE-14034: --- I am reopening this JIRA because there is Zk-related code in distributed log roll procedure which needs to be abstracted. > HBase Backup/Restore Phase 1: Abstract Coordination manager (Zk) operations > --- > > Key: HBASE-14034 > URL: https://issues.apache.org/jira/browse/HBASE-14034 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > > Abstract Coordination manager (Zk) operations. See > org.apache.hadoop.hbase.coordination package for references. Provide > Zookeeper implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-14035) HBase Backup/Restore Phase 1: hbase:backup - backup system table
[ https://issues.apache.org/jira/browse/HBASE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov resolved HBASE-14035. --- Resolution: Fixed The patch is attached to a parent JIRA (HBASE-14030). See v1. > HBase Backup/Restore Phase 1: hbase:backup - backup system table > > > Key: HBASE-14035 > URL: https://issues.apache.org/jira/browse/HBASE-14035 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > > *hbase:backup* - move all backup meta info from Zk (coordination manager) to > hbase system table. Do not use Zk (coordination manager) as a persistent > storage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-14034) HBase Backup/Restore Phase 1: Abstract Coordination manager (Zk) operations
[ https://issues.apache.org/jira/browse/HBASE-14034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov reopened HBASE-14034: --- I am reopening this JIRA because there is Zk-related code in distributed log roll procedure which needs to be abstracted. > HBase Backup/Restore Phase 1: Abstract Coordination manager (Zk) operations > --- > > Key: HBASE-14034 > URL: https://issues.apache.org/jira/browse/HBASE-14034 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > > Abstract Coordination manager (Zk) operations. See > org.apache.hadoop.hbase.coordination package for references. Provide > Zookeeper implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-14034) HBase Backup/Restore Phase 1: Abstract Coordination manager (Zk) operations
[ https://issues.apache.org/jira/browse/HBASE-14034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov resolved HBASE-14034. --- Resolution: Fixed Patch is attached to master JIRA (HBASE-14030) as v2. > HBase Backup/Restore Phase 1: Abstract Coordination manager (Zk) operations > --- > > Key: HBASE-14034 > URL: https://issues.apache.org/jira/browse/HBASE-14034 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > > Abstract Coordination manager (Zk) operations. See > org.apache.hadoop.hbase.coordination package for references. Provide > Zookeeper implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-14031) HBase Backup/Restore Phase 1: Abstract DistCp in incremental backup
[ https://issues.apache.org/jira/browse/HBASE-14031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov resolved HBASE-14031. --- Resolution: Implemented HBASE-14030 v3 contains the patch for the feature. > HBase Backup/Restore Phase 1: Abstract DistCp in incremental backup > --- > > Key: HBASE-14031 > URL: https://issues.apache.org/jira/browse/HBASE-14031 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > > Abstract DistCp (incremental backup) to support non-M/R based > implementations. Provide M/R implementation. DistCp is used to copy WAL files > during incremental backup. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-14032) HBase Backup/Restore Phase 1: Abstract SnapshotCopy (full backup)
[ https://issues.apache.org/jira/browse/HBASE-14032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov resolved HBASE-14032. --- Resolution: Implemented HBASE-14030 v3 contains the patch for the feature. > HBase Backup/Restore Phase 1: Abstract SnapshotCopy (full backup) > - > > Key: HBASE-14032 > URL: https://issues.apache.org/jira/browse/HBASE-14032 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > Abstract SnapshotCopy (full backup) to support non-M/R based implementations. > Provide M/R implementation. SnapshotCopy is used to copy snapshot’s data > during full backup operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-14033) HBase Backup/Restore Phase1: Abstract WALPlayer (incremental restore)
[ https://issues.apache.org/jira/browse/HBASE-14033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov resolved HBASE-14033. --- Resolution: Implemented HBASE-14030 v3 contains the patch for the feature. > HBase Backup/Restore Phase1: Abstract WALPlayer (incremental restore) > - > > Key: HBASE-14033 > URL: https://issues.apache.org/jira/browse/HBASE-14033 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > > Abstract WALPlayer (incremental restore) to support non-M/R based > implementations. Provide M/R implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-14039) BackupHandler.deleteSnapshot MUST use HBase Snapshot API
[ https://issues.apache.org/jira/browse/HBASE-14039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov resolved HBASE-14039. --- Resolution: Invalid Closing as invalid, will reopen later once we find any issues with the current implementation (deleting snapshot directory from FS directly). > BackupHandler.deleteSnapshot MUST use HBase Snapshot API > - > > Key: HBASE-14039 > URL: https://issues.apache.org/jira/browse/HBASE-14039 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > > BackupHandler.deleteSnapshot MUST use HBase API for that (HBaseAdmin) - not > direct FS access (deleting snapshot folder may be not enough?). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-14038) Incremental backup list set is ignored during backup
[ https://issues.apache.org/jira/browse/HBASE-14038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov resolved HBASE-14038. --- Resolution: Later Will work on this later in Phase 2. > Incremental backup list set is ignored during backup > > > Key: HBASE-14038 > URL: https://issues.apache.org/jira/browse/HBASE-14038 > Project: HBase > Issue Type: Bug >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > > BUG: during incremental backup, provided table list is ignored and replaced > with the set of tables which have been already backuped before. Test case: > backup T1, T2, T3, then request incremental backup for T1, T2 => T3 will be > included as well. See: BackupClient.requestBackup. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-14040) Small refactoring in BackupHandler
[ https://issues.apache.org/jira/browse/HBASE-14040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov resolved HBASE-14040. --- Resolution: Won't Fix Not sure if it is worth doing. Too much code change for the sake of small improvement. > Small refactoring in BackupHandler > -- > > Key: HBASE-14040 > URL: https://issues.apache.org/jira/browse/HBASE-14040 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > > Move distributed log roll procedure call to BackupHandler.call from > IncrementalBackupManager.getLogFilesForNewBackup. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-14036) HBase Backup/Restore Phase 1: Custom WAL archive cleaner
[ https://issues.apache.org/jira/browse/HBASE-14036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov resolved HBASE-14036. --- Resolution: Implemented This feature is part of patch v4. See parent JIRA. > HBase Backup/Restore Phase 1: Custom WAL archive cleaner > > > Key: HBASE-14036 > URL: https://issues.apache.org/jira/browse/HBASE-14036 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > > Custom WAL archive cleaner (BackupLogCleaner). We need to keep WAL files in > archive until they either get copied over to backup destination during an > incremental backup or full backup (for ALL tables) happens. This is tricky, > but is doable. Backup-aware WAL archiver cleaner should consult hbase:backup > to determine if WAL file is safe to purge. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-14037) Deletion of a table from backup set results int RTE during next backup
[ https://issues.apache.org/jira/browse/HBASE-14037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov resolved HBASE-14037. --- Resolution: Fixed The fix is part of patch v4. See parent JIRA. > Deletion of a table from backup set results int RTE during next backup > --- > > Key: HBASE-14037 > URL: https://issues.apache.org/jira/browse/HBASE-14037 > Project: HBase > Issue Type: Bug >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > > Deletion of a table with backup history (has Zk node) results in > RuntimeException on all subsequent backup requests. See: > BackupClient.requestBackup. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14123) HBase Backup/Restore Phase 2
Vladimir Rodionov created HBASE-14123: - Summary: HBase Backup/Restore Phase 2 Key: HBASE-14123 URL: https://issues.apache.org/jira/browse/HBASE-14123 Project: HBase Issue Type: Umbrella Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14124) Failed backup is not handled properly in incremental mode
Vladimir Rodionov created HBASE-14124: - Summary: Failed backup is not handled properly in incremental mode Key: HBASE-14124 URL: https://issues.apache.org/jira/browse/HBASE-14124 Project: HBase Issue Type: Bug Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov BackupHandler failedBackup method does not clean failed incremental backup artefacts on HDFS (and in HBase). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14125) HBAse Backup/Restore Phase 2: Cancel backup
Vladimir Rodionov created HBASE-14125: - Summary: HBAse Backup/Restore Phase 2: Cancel backup Key: HBASE-14125 URL: https://issues.apache.org/jira/browse/HBASE-14125 Project: HBase Issue Type: New Feature Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Cancel backup operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-14039) BackupHandler.deleteSnapshot MUST use HBase Snapshot API
[ https://issues.apache.org/jira/browse/HBASE-14039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov reopened HBASE-14039: --- OK, reopening this JIRA. The point of using HBAse API and not internal hacks is important. > BackupHandler.deleteSnapshot MUST use HBase Snapshot API > - > > Key: HBASE-14039 > URL: https://issues.apache.org/jira/browse/HBASE-14039 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > > BackupHandler.deleteSnapshot MUST use HBase API for that (HBaseAdmin) - not > direct FS access (deleting snapshot folder may be not enough?). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-14039) BackupHandler.deleteSnapshot MUST use HBase Snapshot API
[ https://issues.apache.org/jira/browse/HBASE-14039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov resolved HBASE-14039. --- Resolution: Fixed Part of HBASE-14030 patch v6. > BackupHandler.deleteSnapshot MUST use HBase Snapshot API > - > > Key: HBASE-14039 > URL: https://issues.apache.org/jira/browse/HBASE-14039 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > > BackupHandler.deleteSnapshot MUST use HBase API for that (HBaseAdmin) - not > direct FS access (deleting snapshot folder may be not enough?). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14130) HBase Backup/Restore Phase 2: Delete backup image
Vladimir Rodionov created HBASE-14130: - Summary: HBase Backup/Restore Phase 2: Delete backup image Key: HBASE-14130 URL: https://issues.apache.org/jira/browse/HBASE-14130 Project: HBase Issue Type: New Feature Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14131) HBase Backup/Restore Phase 2: Describe backup image
Vladimir Rodionov created HBASE-14131: - Summary: HBase Backup/Restore Phase 2: Describe backup image Key: HBASE-14131 URL: https://issues.apache.org/jira/browse/HBASE-14131 Project: HBase Issue Type: New Feature Reporter: Vladimir Rodionov -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14132) HBase Backup/Restore Phase 2: History of backups
Vladimir Rodionov created HBASE-14132: - Summary: HBase Backup/Restore Phase 2: History of backups Key: HBASE-14132 URL: https://issues.apache.org/jira/browse/HBASE-14132 Project: HBase Issue Type: New Feature Reporter: Vladimir Rodionov -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14133) HBase Backup/Restore Phase 2: Status (and progress) of backup request
Vladimir Rodionov created HBASE-14133: - Summary: HBase Backup/Restore Phase 2: Status (and progress) of backup request Key: HBASE-14133 URL: https://issues.apache.org/jira/browse/HBASE-14133 Project: HBase Issue Type: New Feature Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14134) HBase Backup/Restore Phase 2: Backup sets management
Vladimir Rodionov created HBASE-14134: - Summary: HBase Backup/Restore Phase 2: Backup sets management Key: HBASE-14134 URL: https://issues.apache.org/jira/browse/HBASE-14134 Project: HBase Issue Type: New Feature Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14135) HBase Backup/Restore Phase 2: Merge backup images
Vladimir Rodionov created HBASE-14135: - Summary: HBase Backup/Restore Phase 2: Merge backup images Key: HBASE-14135 URL: https://issues.apache.org/jira/browse/HBASE-14135 Project: HBase Issue Type: New Feature Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14136) HBase Backup/Restore Phase 2: Convert backup image(s)
Vladimir Rodionov created HBASE-14136: - Summary: HBase Backup/Restore Phase 2: Convert backup image(s) Key: HBASE-14136 URL: https://issues.apache.org/jira/browse/HBASE-14136 Project: HBase Issue Type: New Feature Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14137) HBase Backup/Restore Phase 2: Backup throttling
Vladimir Rodionov created HBASE-14137: - Summary: HBase Backup/Restore Phase 2: Backup throttling Key: HBASE-14137 URL: https://issues.apache.org/jira/browse/HBASE-14137 Project: HBase Issue Type: New Feature Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov ExportSnapshot/DistCp supports IO throttling per map task - this needs to be exposed to backup utility command line tool. Backups must not interfere with regular HBase cluster operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14138) HBase Backup/Restore Phase 2: Security
Vladimir Rodionov created HBASE-14138: - Summary: HBase Backup/Restore Phase 2: Security Key: HBASE-14138 URL: https://issues.apache.org/jira/browse/HBASE-14138 Project: HBase Issue Type: New Feature Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Security is not supported. Only authorized user (GLOBAL ADMIN) must be allowed to perform backup/restore. See: HBASE-7367 for good discussion on snapshot security model. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14139) HBase Backup/Restore Phase 2: Backup from existing snapshot
Vladimir Rodionov created HBASE-14139: - Summary: HBase Backup/Restore Phase 2: Backup from existing snapshot Key: HBASE-14139 URL: https://issues.apache.org/jira/browse/HBASE-14139 Project: HBase Issue Type: New Feature Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14140) HBase Backup/Restore Phase 2: Enhance HBaseAdmin API to include backup/restore - related API
Vladimir Rodionov created HBASE-14140: - Summary: HBase Backup/Restore Phase 2: Enhance HBaseAdmin API to include backup/restore - related API Key: HBASE-14140 URL: https://issues.apache.org/jira/browse/HBASE-14140 Project: HBase Issue Type: New Feature Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14141) HBase Backup/Restore Phase 2: Filter WALs on backup to include only edits from backup set tables
Vladimir Rodionov created HBASE-14141: - Summary: HBase Backup/Restore Phase 2: Filter WALs on backup to include only edits from backup set tables Key: HBASE-14141 URL: https://issues.apache.org/jira/browse/HBASE-14141 Project: HBase Issue Type: New Feature Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14142) HBase Backup/Restore Phase 2: Cells deduplication during backup
Vladimir Rodionov created HBASE-14142: - Summary: HBase Backup/Restore Phase 2: Cells deduplication during backup Key: HBASE-14142 URL: https://issues.apache.org/jira/browse/HBASE-14142 Project: HBase Issue Type: New Feature Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov As since we do not record last backed up sequence ids (MVCC) and do not restore up to that sequence id - that is kind of tricky, there will be some duplicates of KVs in store files after first incremental restore after full backup. These duplicates are result of how we do full backup and first incremental backup after full one. During full backup we perform distributed log roll and record, for every RS, last WAL timestamp, then we do snapshot. The next WAL after recorded one will make it into a next incremental backup set, but it will contains some edits (puts, deletes) which have been recorded by a previous snapshot. During restore, we, first, restore snapshot, then we will re-play WALs and this operation can create some duplicates of KVs in different store files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-11172) Cancel a backup process
[ https://issues.apache.org/jira/browse/HBASE-11172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov resolved HBASE-11172. --- Resolution: Invalid > Cancel a backup process > > > Key: HBASE-11172 > URL: https://issues.apache.org/jira/browse/HBASE-11172 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.99.0 >Reporter: Demai Ni > > h2. Feature Description > the jira is part of > [HBASE-7912|https://issues.apache.org/jira/browse/HBASE-7912], and depend on > full backup [HBASE-10900| https://issues.apache.org/jira/browse/HBASE-10900] > and incremental backup [HBASE-11085| > https://issues.apache.org/jira/browse/HBASE-11085]. for the detail layout and > frame work, please reference to [HBASE-10900| > https://issues.apache.org/jira/browse/HBASE-10900]. > A backup operation may need to move handreds/thousands GB of data, and takes > hours. Sometimes, the operation may take longer than the original maintenance > time window planned by the administration. So it is necessary to have the > functionality to cancel the operation and reset all the history/manifest info > whenever necessary. so that we can have a clean backup in the next time > window -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-11173) Show Backup History
[ https://issues.apache.org/jira/browse/HBASE-11173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov resolved HBASE-11173. --- Resolution: Invalid > Show Backup History > > > Key: HBASE-11173 > URL: https://issues.apache.org/jira/browse/HBASE-11173 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.99.0 >Reporter: Demai Ni > > h2. Feature Description > the jira is part of > [HBASE-7912|https://issues.apache.org/jira/browse/HBASE-7912], and depend on > full backup [HBASE-10900| https://issues.apache.org/jira/browse/HBASE-10900] > and incremental backup [HBASE-11085| > https://issues.apache.org/jira/browse/HBASE-11085]. for the detail layout and > frame work, please reference to [HBASE-10900| > https://issues.apache.org/jira/browse/HBASE-10900]. > After several backup operations executed in the past, he may like to know > what tables were backuped at what time, so that a restore or future backup > operation can be performanced accordingly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-11175) improve Backup/Restore framework by abstracting out zookeeper
[ https://issues.apache.org/jira/browse/HBASE-11175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov resolved HBASE-11175. --- Resolution: Invalid > improve Backup/Restore framework by abstracting out zookeeper > - > > Key: HBASE-11175 > URL: https://issues.apache.org/jira/browse/HBASE-11175 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.99.0 >Reporter: Demai Ni >Assignee: Vladimir Rodionov > > the jira is part of > [HBASE-7912|https://issues.apache.org/jira/browse/HBASE-7912], > h2. Feature Description > current backup/restore patches are using zookeeper to keep the history and > the dependency on source cluster. This jira is to abstract out the zookeeper > usage. > The jira is kind of follow up of > [HBASE-10909|https://issues.apache.org/jira/browse/HBASE-10909] and > [HBASE-10296|https://issues.apache.org/jira/browse/HBASE-10296] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-11174) show backup/restore progress
[ https://issues.apache.org/jira/browse/HBASE-11174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov resolved HBASE-11174. --- Resolution: Invalid > show backup/restore progress > > > Key: HBASE-11174 > URL: https://issues.apache.org/jira/browse/HBASE-11174 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.99.0 >Reporter: Demai Ni > > h2. Feature Description > the jira is part of > [HBASE-7912|https://issues.apache.org/jira/browse/HBASE-7912], and depend on > full backup [HBASE-10900| https://issues.apache.org/jira/browse/HBASE-10900] > and incremental backup [HBASE-11085| > https://issues.apache.org/jira/browse/HBASE-11085]. for the detail layout and > frame work, please reference to [HBASE-10900| > https://issues.apache.org/jira/browse/HBASE-10900]. > A backup/restore operation may take a while to complete, sometimes hours. It > will be helpful to show the estimated progress as percentage to user. The > jira will provide such functionally -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-11180) Merge backups
[ https://issues.apache.org/jira/browse/HBASE-11180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov resolved HBASE-11180. --- Resolution: Invalid > Merge backups > - > > Key: HBASE-11180 > URL: https://issues.apache.org/jira/browse/HBASE-11180 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.99.0 >Reporter: Jerry He >Assignee: Enoch Hsu > > This JIRA covers merging backups. > Multiple incremental backups can be merged to reduce granularity. > Incremental backups can also be merged with full backups. > Additional processing of the data files can be done during merge, e.g. > compaction. > Merge is an offline operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-14130) HBase Backup/Restore Phase 2: Delete backup image
[ https://issues.apache.org/jira/browse/HBASE-14130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov resolved HBASE-14130. --- Resolution: Implemented See parent JIRA (HBASE-14123). > HBase Backup/Restore Phase 2: Delete backup image > - > > Key: HBASE-14130 > URL: https://issues.apache.org/jira/browse/HBASE-14130 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-14125) HBase Backup/Restore Phase 2: Cancel backup
[ https://issues.apache.org/jira/browse/HBASE-14125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov resolved HBASE-14125. --- Resolution: Implemented See parent JIRA (HBASE-14123). > HBase Backup/Restore Phase 2: Cancel backup > --- > > Key: HBASE-14125 > URL: https://issues.apache.org/jira/browse/HBASE-14125 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > Cancel backup operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-14131) HBase Backup/Restore Phase 2: Describe backup image
[ https://issues.apache.org/jira/browse/HBASE-14131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov resolved HBASE-14131. --- Resolution: Implemented See parent JIRA (HBASE-14123). > HBase Backup/Restore Phase 2: Describe backup image > --- > > Key: HBASE-14131 > URL: https://issues.apache.org/jira/browse/HBASE-14131 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-14133) HBase Backup/Restore Phase 2: Status (and progress) of backup request
[ https://issues.apache.org/jira/browse/HBASE-14133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov resolved HBASE-14133. --- Resolution: Implemented > HBase Backup/Restore Phase 2: Status (and progress) of backup request > - > > Key: HBASE-14133 > URL: https://issues.apache.org/jira/browse/HBASE-14133 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-14132) HBase Backup/Restore Phase 2: History of backups
[ https://issues.apache.org/jira/browse/HBASE-14132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov resolved HBASE-14132. --- Resolution: Implemented See parent JIRA (HBASE-14123). > HBase Backup/Restore Phase 2: History of backups > > > Key: HBASE-14132 > URL: https://issues.apache.org/jira/browse/HBASE-14132 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14195) Closed connections are not removed from ConnectionCache (Thrift, REST)
Vladimir Rodionov created HBASE-14195: - Summary: Closed connections are not removed from ConnectionCache (Thrift, REST) Key: HBASE-14195 URL: https://issues.apache.org/jira/browse/HBASE-14195 Project: HBase Issue Type: Bug Components: REST, Thrift Affects Versions: 1.1.0.1, 1.0.1.1, 1.1.1, 0.98.13 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.2, 1.3.0 There is the issue with ConnectionCache (used to cache connections in Thrift and REST servers) chore implementation when closed connection is not removed from cache. Thrift server is affected most because it has local thread cache of Table instances and it does not check if Table instance is invalid (due to closed underlying connection) and it can't do it - no API. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14196) Do not use local thread cache of table instances in Thrift/Thrift2 server
Vladimir Rodionov created HBASE-14196: - Summary: Do not use local thread cache of table instances in Thrift/Thrift2 server Key: HBASE-14196 URL: https://issues.apache.org/jira/browse/HBASE-14196 Project: HBase Issue Type: Bug Components: Thrift Affects Versions: 1.1.0.1, 1.0.1.1, 1.1.1, 0.98.13 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.2, 1.3.0 This is the antipattern. Table objects are lightweight and should not be cached, besides this, underlying connections can be closed by periodic connection cleaner chore (ConnectionCache) and cached table instances may become invalid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14198) Eclipse project generation is broken in master
Vladimir Rodionov created HBASE-14198:
-
Summary: Eclipse project generation is broken in master
Key: HBASE-14198
URL: https://issues.apache.org/jira/browse/HBASE-14198
Project: HBase
Issue Type: Bug
Reporter: Vladimir Rodionov
After running
mvn eclipse:eclipse I tried to import projects into Eclipse (Luna) and got
multiple build errors, similar to:
{code}
Cannot nest output folder 'hbase-thrift/target/test-classes/META-INF' inside
output folder 'hbase-thrift/target/test-classes' hbase-thrift
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-14195) Closed connections are not removed from ConnectionCache (Thrift, REST)
[ https://issues.apache.org/jira/browse/HBASE-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov resolved HBASE-14195. --- Resolution: Invalid They are closed actually. My bad. Resolving as invalid. > Closed connections are not removed from ConnectionCache (Thrift, REST) > -- > > Key: HBASE-14195 > URL: https://issues.apache.org/jira/browse/HBASE-14195 > Project: HBase > Issue Type: Bug > Components: REST, Thrift >Affects Versions: 0.98.13, 1.1.1, 1.0.1.1, 1.1.0.1 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.2, 1.3.0 > > > There is the issue with ConnectionCache (used to cache connections in Thrift > and REST servers) chore implementation when closed connection is not removed > from a cache. > Thrift server is affected most because it has local thread cache of Table > instances and it does not check if Table instance is invalid (due to closed > underlying connection) and it can't do it - no API. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-14196) Do not use local thread cache of table instances in Thrift/Thrift2 server
[ https://issues.apache.org/jira/browse/HBASE-14196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov resolved HBASE-14196. --- Resolution: Invalid Invalid again. > Do not use local thread cache of table instances in Thrift/Thrift2 server > - > > Key: HBASE-14196 > URL: https://issues.apache.org/jira/browse/HBASE-14196 > Project: HBase > Issue Type: Bug > Components: Thrift >Affects Versions: 0.98.13, 1.1.1, 1.0.1.1, 1.1.0.1 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.2, 1.3.0 > > > This is the antipattern. Table objects are lightweight and should not be > cached, besides this, underlying connections can be closed by periodic > connection cleaner chore (ConnectionCache) and cached table instances may > become invalid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-14196) Do not use local thread cache of table instances in Thrift/Thrift2 server
[ https://issues.apache.org/jira/browse/HBASE-14196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov reopened HBASE-14196: --- > Do not use local thread cache of table instances in Thrift/Thrift2 server > - > > Key: HBASE-14196 > URL: https://issues.apache.org/jira/browse/HBASE-14196 > Project: HBase > Issue Type: Bug > Components: Thrift >Affects Versions: 0.98.13, 1.1.1, 1.0.1.1, 1.1.0.1 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.2, 1.3.0 > > > This is the antipattern. Table objects are lightweight and should not be > cached, besides this, underlying connections can be closed by periodic > connection cleaner chore (ConnectionCache) and cached table instances may > become invalid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14235) HBase Backup/Restore Phase 2: Async backup procedure
Vladimir Rodionov created HBASE-14235: - Summary: HBase Backup/Restore Phase 2: Async backup procedure Key: HBASE-14235 URL: https://issues.apache.org/jira/browse/HBASE-14235 Project: HBase Issue Type: Task Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 2.0.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14258) Make region_mover.rb script case insensitive in regard to hostname
Vladimir Rodionov created HBASE-14258: - Summary: Make region_mover.rb script case insensitive in regard to hostname Key: HBASE-14258 URL: https://issues.apache.org/jira/browse/HBASE-14258 Project: HBase Issue Type: Bug Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov The script is case sensitive and fails when case of a host name being unloaded does not match with a case of a region server name returned by HBase API. This doc clarifies IETF rules on case insensitivities in DNS: https://www.ietf.org/rfc/rfc4343.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
