subject:"\[jira\] \[Commented\] \(HBASE\-5547\) Don't delete HFiles when in backup mode"

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420118#comment-13420118
]

Hadoop QA commented on HBASE-5547:
--

-1 overall. Here are the results of testing the latest attachment

http://issues.apache.org/jira/secure/attachment/12537478/java_HBASE-5547.addendum
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

+1 javadoc. The javadoc tool did not generate any warning messages.

-1 javac. The applied patch generated 5 javac compiler warnings (more than
the trunk's current 4 warnings).

-1 findbugs. The patch appears to introduce 14 new Findbugs (version
1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:

org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/2424//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2424//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2424//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2424//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2424//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2424//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/2424//console

This message is automatically generated.

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.2

Attachments: 5547-v12.txt, 5547-v16.txt, hbase-5447-v8.patch,
hbase-5447-v8.patch, hbase-5547-v9.patch, java_HBASE-5547.addendum,
java_HBASE-5547_v13.patch, java_HBASE-5547_v14.patch,
java_HBASE-5547_v15.patch, java_HBASE-5547_v4.patch,
java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, java_HBASE-5547_v7.patch

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

2012-07-22 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420159#comment-13420159
 ] 

Hudson commented on HBASE-5547:
---

Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #103 (See 
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/103/])
HBASE-5547 Don't delete HFiles when in backup mode (Jesse Yates) 
(Revision 1364203)

 Result = FAILURE
larsh : 
Files : 
* 
/hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/BaseConfigurable.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/Chore.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/HFileArchiveManager.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/HFileArchiveTableMonitor.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/LongTermArchivingHFileCleaner.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/TableHFileArchiveTracker.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/ZKTableArchiveClient.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/LogCleaner.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/LogCleanerDelegate.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/TimeToLiveLogCleaner.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseHFileCleanerDelegate.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseLogCleanerDelegate.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/FileCleanerDelegate.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/LogCleaner.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/TimeToLiveHFileCleaner.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/TimeToLiveLogCleaner.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/master/ReplicationLogCleaner.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
* /hbase/trunk/hbase-server/src/main/resources/hbase-default.xml
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestHFileArchiving.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example/TestZooKeeperTableArchiveClient.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestLogsCleaner.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileCleaner.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestLogsCleaner.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/CheckedArchivingHFileCleaner.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java
*

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420187#comment-13420187
 ] 

Zhihong Ted Yu commented on HBASE-5547:
---

For the NPE, I think the null reference should have come from below code 
snippet:
{code}
(sf == null ? none : sf.getPath().getName()) +
, size= + (sf == null ? none :
  StringUtils.humanReadableInt(sf.getReader().length()))
{code}
Meaning either sf.getPath() or sf.getReader() was null.

For the addendum, if TimeToLiveHFileCleaner is removed from 
TestZooKeeperTableArchiveClient, we should add test for it since it is a new 
class introduced by this feature.

 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.2

 Attachments: 5547-v12.txt, 5547-v16.txt, hbase-5447-v8.patch, 
 hbase-5447-v8.patch, hbase-5547-v9.patch, java_HBASE-5547.addendum, 
 java_HBASE-5547_v13.patch, java_HBASE-5547_v14.patch, 
 java_HBASE-5547_v15.patch, java_HBASE-5547_v4.patch, 
 java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420226#comment-13420226
]

Jesse Yates commented on HBASE-5547:

{quote}
Meaning either sf.getPath() or sf.getReader() was null.
{quote}

Looked through the code and can't say I see that was possible (though it seems
the most likely given the setting/checking done on the store file). Looking
through the test of the code, it doesn't look like an issue if the reader is
null since it either gets closed immediately or is re-created when needed.

The getPath() call could be bad, but its not doing anything funky and we would
have seen a warning/error earlier in the run if that had actually been a
problem.

{quote}
For the addendum, if TimeToLiveHFileCleaner is removed from
TestZooKeeperTableArchiveClient, we should add test for it since it is a new
class introduced by this feature.
{quote}

It is tested in o.a.h.h.master.cleaner.TestHFileCleaner. This change just
cleans up the test so we don't have to worry about interaction with the
TTLCleaner.

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.2

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420233#comment-13420233
]

Zhihong Ted Yu commented on HBASE-5547:
---

bq. we don't have to worry about interaction with the TTLCleaner.
Users may choose to enable the combination of cleaners shown in the current
TestZooKeeperTableArchiveClient.
We should verify that this combination works.

From
https://builds.apache.org/view/G-L/view/HBase/job/HBase-TRUNK-on-Hadoop-2.0.0/lastCompletedBuild/testReport/org.apache.hadoop.hbase.backup.example/TestZooKeeperTableArchiveClient/testMultipleTables/,
there is no NPE.

I used the following command to run the test:

mvn clean -Dhadoop.profile=2.0 test
-Dtest=TestZooKeeperTableArchiveClient#testMultipleTables
{code}
Failed tests:
testMultipleTables(org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient):
Archived HFiles should have gotten deleted, but didn't
{code}
The failure was due to IllegalArgumentException shown above.

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.2

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420235#comment-13420235
]

Jesse Yates commented on HBASE-5547:

bq. The failure was due to IllegalArgumentException shown above.

Yeah, it seems to be breaking all the tests. Its really odd that HDFS goes to
the DistributedFileSystem to do the listStatus when its a local filesystem -
seems wrong.

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.2

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420237#comment-13420237
 ] 

Zhihong Ted Yu commented on HBASE-5547:
---

Could this be related to HBASE-6403 ?

 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.2

 Attachments: 5547-v12.txt, 5547-v16.txt, hbase-5447-v8.patch, 
 hbase-5447-v8.patch, hbase-5547-v9.patch, java_HBASE-5547.addendum, 
 java_HBASE-5547_v13.patch, java_HBASE-5547_v14.patch, 
 java_HBASE-5547_v15.patch, java_HBASE-5547_v4.patch, 
 java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420250#comment-13420250
 ] 

Jesse Yates commented on HBASE-5547:


bq. Could this be related to HBASE-6403 ?

Similar, but not exactly the same. In short, the FS has a configuration, but 
that configuration isn't correctly specifying the filesystem (from the 
MasterFileSystem). Working on a fix now.

 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.2

 Attachments: 5547-v12.txt, 5547-v16.txt, hbase-5447-v8.patch, 
 hbase-5447-v8.patch, hbase-5547-v9.patch, java_HBASE-5547.addendum, 
 java_HBASE-5547_v13.patch, java_HBASE-5547_v14.patch, 
 java_HBASE-5547_v15.patch, java_HBASE-5547_v4.patch, 
 java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420266#comment-13420266
]

Jesse Yates commented on HBASE-5547:

{quote}
Users may choose to enable the combination of cleaners shown in the current
TestZooKeeperTableArchiveClient.
We should verify that this combination works.
{quote}

I'd argue that we have verified by inspection. Using the time-based waiting on
the cluster tends to lead to flapping tests (e.g. the original error), which is
bad. Because the cleaner chain works, the TTL cleaner works (both of which are
tested) and the zk archiver cleaner works, we can then say that they work
together.

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.2

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420269#comment-13420269
]

Hadoop QA commented on HBASE-5547:
--

-1 overall. Here are the results of testing the latest attachment

http://issues.apache.org/jira/secure/attachment/12537517/java_HBASE-5547.addendum-v1
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 6 new or modified tests.

+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

+1 javadoc. The javadoc tool did not generate any warning messages.

-1 javac. The applied patch generated 5 javac compiler warnings (more than
the trunk's current 4 warnings).

-1 findbugs. The patch appears to introduce 14 new Findbugs (version
1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestCatalogJanitor

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/2425//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2425//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2425//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2425//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2425//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2425//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/2425//console

This message is automatically generated.

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.2

Attachments: 5547-v12.txt, 5547-v16.txt, hbase-5447-v8.patch,
hbase-5447-v8.patch, hbase-5547-v9.patch, java_HBASE-5547.addendum,
java_HBASE-5547.addendum-v1, java_HBASE-5547_v13.patch,
java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch,
java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch,
java_HBASE-5547_v7.patch

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420271#comment-13420271
]

Lars Hofhansl commented on HBASE-5547:
--

Addendum v1 looks good to me. I also agree that we should remove the TTL
cleaner when the ZK cleaner is tested.
Might be prudent to add an integration test that uses both in some manner.

One question looking at the addendum: When would the passed configuration to
archiveRegion be different from fs.getConfiguration()? It seems that should not
happen; if it did other confusion might arise.

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.2

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420272#comment-13420272
]

Jesse Yates commented on HBASE-5547:

{quote}
When would the passed configuration to archiveRegion be different from
fs.getConfiguration()? It seems that should not happen; if it did other
confusion might arise.
{quote}

The reason the error was happening was the conf in fs.getConf() didn't have the
fs set correctly, so it was going to the local filesystem, but the FS we were
deleting from was a DistributedFileSystem. The passed in conf is now from an
'authorative' source (the HMaster).

Didn't run the catalogjanitor test...working on fix.

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.2

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420275#comment-13420275
 ] 

Lars Hofhansl commented on HBASE-5547:
--

I see... Seems something else is wrong then, I think. Don't have to fix it 
here, just something to keep in mind.


 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.2

 Attachments: 5547-v12.txt, 5547-v16.txt, hbase-5447-v8.patch, 
 hbase-5447-v8.patch, hbase-5547-v9.patch, java_HBASE-5547.addendum, 
 java_HBASE-5547.addendum-v1, java_HBASE-5547_v13.patch, 
 java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch, 
 java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, 
 java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420279#comment-13420279
]

Jesse Yates commented on HBASE-5547:

I'd rather have the fix in, then revert as necessary when we find the root
cause (just in case this is 'expected' behavior. Particularly given the
occasional oddness between hdfs-1.X and 2.x., hack it to fix the build and then
fix it 'right' later seems ok.

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.2

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420280#comment-13420280
 ] 

Jesse Yates commented on HBASE-5547:


oh, wait, spent the two minutes to RTFS and realized that, when calling 
path#getFileSystem(conf), if there is no scheme for the path (an unqualified 
path), it tries to make a filesystem object from the configuration using the 
default URI, which if it isn't set, uses file:/// (see FileSystem.java:131, 
hadoop-1.0.3).

Guess we need to make sure that the fileSystem sets the default URI for its 
configuration. 

Should I file another ticket or just roll the fix in here?



 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.2

 Attachments: 5547-v12.txt, 5547-v16.txt, hbase-5447-v8.patch, 
 hbase-5447-v8.patch, hbase-5547-v9.patch, java_HBASE-5547.addendum, 
 java_HBASE-5547.addendum-v1, java_HBASE-5547_v13.patch, 
 java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch, 
 java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, 
 java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420284#comment-13420284
]

Lars Hofhansl commented on HBASE-5547:
--

Yeah, separate ticket.
Should we commit v1 as is now to get most of the tests working again and then
one last addendum to fix the catalog janitor test?

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.2

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420285#comment-13420285
 ] 

Jesse Yates commented on HBASE-5547:


The catalog janitor is just one more line. I'll post a new addendum momentarily.

 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.2

 Attachments: 5547-v12.txt, 5547-v16.txt, hbase-5447-v8.patch, 
 hbase-5447-v8.patch, hbase-5547-v9.patch, java_HBASE-5547.addendum, 
 java_HBASE-5547.addendum-v1, java_HBASE-5547_v13.patch, 
 java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch, 
 java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, 
 java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420293#comment-13420293
]

Zhihong Ted Yu commented on HBASE-5547:
---

bq. Because the cleaner chain works, the TTL cleaner works (both of which are
tested) and the zk archiver cleaner works, we can then say that they work
together.
If some user encounters the scenario shown in existing
TestZooKeeperTableArchiveClient#testMultipleTables and asked us on mailing
list, can we go with the above answer ?

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.2

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420296#comment-13420296
]

Jesse Yates commented on HBASE-5547:

@Ted: yeah. In the worst case, the data is kept around for an extra ttl
(because that's how the cleaner works), so at worst they have the reference
file for another 5-10 mins (depending on defaults).

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.2

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420305#comment-13420305
 ] 

Zhihong Ted Yu commented on HBASE-5547:
---

If I keep the cleaner in addendum v1, testMultipleTables failed at the 
following assertion:
{code}
assertFalse(fs.exists(otherStoreArchive));
{code}

 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.2

 Attachments: 5547-v12.txt, 5547-v16.txt, hbase-5447-v8.patch, 
 hbase-5447-v8.patch, hbase-5547-v9.patch, java_HBASE-5547.addendum, 
 java_HBASE-5547.addendum-v1, java_HBASE-5547_v13.patch, 
 java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch, 
 java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, 
 java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420310#comment-13420310
 ] 

Jesse Yates commented on HBASE-5547:


@Ted yes, the sleeps for the ttl cleaner were also removed, which probably 
caused the error. Give me a little bit and I'll post an addendum that:

* removes the TLL cleaner from the primary ZKTableArchiver test
* adds a test that just tests the interaction between the ZKTableArchiver and 
the TTLCleaner
* fixes the fs scheme for all tests (general things like avro server and the 
catalogjanitor)

And I'll file another ticket to fix the fs configuration/rootdir stuff from the 
master.

Did I miss anything?

 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.2

 Attachments: 5547-v12.txt, 5547-v16.txt, hbase-5447-v8.patch, 
 hbase-5447-v8.patch, hbase-5547-v9.patch, java_HBASE-5547.addendum, 
 java_HBASE-5547.addendum-v1, java_HBASE-5547_v13.patch, 
 java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch, 
 java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, 
 java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420312#comment-13420312
]

Lars Hofhansl commented on HBASE-5547:
--

@Ted: The two cleaners should be tested in isolation. That's what unit tests
are all about.

We can have an additional test that verifies that the cleaners work together
(which just means that if one cleaner decides the file needs to be kept it will
be kept around). Let's do that in a separate jira.

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.2

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420329#comment-13420329
]

Jesse Yates commented on HBASE-5547:

Attached fix for the fs scheme to HBASE-6441. Seems to fix the hadoop-2.0 test
problem. Working on simpler version of addendum.v1 that just handles the
multipleTests issue and formatting.

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.2

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420349#comment-13420349
]

Hadoop QA commented on HBASE-5547:
--

-1 overall. Here are the results of testing the latest attachment

http://issues.apache.org/jira/secure/attachment/12537526/java_HBASE-5547.addendum-v2
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

+1 javadoc. The javadoc tool did not generate any warning messages.

-1 javac. The applied patch generated 5 javac compiler warnings (more than
the trunk's current 4 warnings).

-1 findbugs. The patch appears to introduce 14 new Findbugs (version
1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in .

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/2429//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2429//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2429//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2429//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2429//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2429//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/2429//console

This message is automatically generated.

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.2

Attachments: 5547-v12.txt, 5547-v16.txt, hbase-5447-v8.patch,
hbase-5447-v8.patch, hbase-5547-v9.patch, java_HBASE-5547.addendum,
java_HBASE-5547.addendum-v1, java_HBASE-5547.addendum-v2,
java_HBASE-5547_v13.patch, java_HBASE-5547_v14.patch,
java_HBASE-5547_v15.patch, java_HBASE-5547_v4.patch,
java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, java_HBASE-5547_v7.patch

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420350#comment-13420350
 ] 

Zhihong Ted Yu commented on HBASE-5547:
---

From https://builds.apache.org/job/PreCommit-HBASE-Build/2429//testReport/:
{code}
Running org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 238.611 sec
{code}
I think TestZooKeeperTableArchiveClient should be categorized as large test.

 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.2

 Attachments: 5547-v12.txt, 5547-v16.txt, hbase-5447-v8.patch, 
 hbase-5447-v8.patch, hbase-5547-v9.patch, java_HBASE-5547.addendum, 
 java_HBASE-5547.addendum-v1, java_HBASE-5547.addendum-v2, 
 java_HBASE-5547_v13.patch, java_HBASE-5547_v14.patch, 
 java_HBASE-5547_v15.patch, java_HBASE-5547_v4.patch, 
 java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420419#comment-13420419
 ] 

Jesse Yates commented on HBASE-5547:


Sounds good to me.

 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.2

 Attachments: 5547-v12.txt, 5547-v16.txt, 5547.addendum-v3, 
 hbase-5447-v8.patch, hbase-5447-v8.patch, hbase-5547-v9.patch, 
 java_HBASE-5547.addendum, java_HBASE-5547.addendum-v1, 
 java_HBASE-5547.addendum-v2, java_HBASE-5547_v13.patch, 
 java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch, 
 java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, 
 java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420423#comment-13420423
 ] 

Lars Hofhansl commented on HBASE-5547:
--

Addendum v3 looks good to me. I'm happy to commit, unless you would like to, 
Ted.

 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.2

 Attachments: 5547-v12.txt, 5547-v16.txt, 5547.addendum-v3, 
 hbase-5447-v8.patch, hbase-5447-v8.patch, hbase-5547-v9.patch, 
 java_HBASE-5547.addendum, java_HBASE-5547.addendum-v1, 
 java_HBASE-5547.addendum-v2, java_HBASE-5547_v13.patch, 
 java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch, 
 java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, 
 java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420425#comment-13420425
 ] 

Zhihong Ted Yu commented on HBASE-5547:
---

@Lars:
Go ahead.

 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.2

 Attachments: 5547-v12.txt, 5547-v16.txt, 5547.addendum-v3, 
 hbase-5447-v8.patch, hbase-5447-v8.patch, hbase-5547-v9.patch, 
 java_HBASE-5547.addendum, java_HBASE-5547.addendum-v1, 
 java_HBASE-5547.addendum-v2, java_HBASE-5547_v13.patch, 
 java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch, 
 java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, 
 java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420427#comment-13420427
 ] 

Lars Hofhansl commented on HBASE-5547:
--

Done. Should we file an extra jira and a cleaner chain test?

 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.2

 Attachments: 5547-v12.txt, 5547-v16.txt, 5547.addendum-v3, 
 hbase-5447-v8.patch, hbase-5447-v8.patch, hbase-5547-v9.patch, 
 java_HBASE-5547.addendum, java_HBASE-5547.addendum-v1, 
 java_HBASE-5547.addendum-v2, java_HBASE-5547_v13.patch, 
 java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch, 
 java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, 
 java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420429#comment-13420429
 ] 

Zhihong Ted Yu commented on HBASE-5547:
---

HBASE-6442 has been filed as a sub-task of this JIRA.
Feel free to either put detail there or add new task(s).

 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.2

 Attachments: 5547-v12.txt, 5547-v16.txt, 5547.addendum-v3, 
 hbase-5447-v8.patch, hbase-5447-v8.patch, hbase-5547-v9.patch, 
 java_HBASE-5547.addendum, java_HBASE-5547.addendum-v1, 
 java_HBASE-5547.addendum-v2, java_HBASE-5547_v13.patch, 
 java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch, 
 java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, 
 java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420434#comment-13420434
]

Hadoop QA commented on HBASE-5547:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12537542/5547.addendum-v3
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

+1 javadoc. The javadoc tool did not generate any warning messages.

-1 javac. The applied patch generated 5 javac compiler warnings (more than
the trunk's current 4 warnings).

-1 findbugs. The patch appears to introduce 14 new Findbugs (version
1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in .

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/2431//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2431//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2431//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2431//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2431//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/2431//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/2431//console

This message is automatically generated.

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.96.0, 0.94.2

Attachments: 5547-v12.txt, 5547-v16.txt, 5547.addendum-v3,
hbase-5447-v8.patch, hbase-5447-v8.patch, hbase-5547-v9.patch,
java_HBASE-5547.addendum, java_HBASE-5547.addendum-v1,
java_HBASE-5547.addendum-v2, java_HBASE-5547_v13.patch,
java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch,
java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch,
java_HBASE-5547_v7.patch

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

2012-07-22 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420449#comment-13420449
 ] 

Hudson commented on HBASE-5547:
---

Integrated in HBase-TRUNK #3162 (See 
[https://builds.apache.org/job/HBase-TRUNK/3162/])
HBASE-5547 Addendum (Jesse and Ted) (Revision 1364493)

 Result = FAILURE
larsh : 
Files : 
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/TableHFileArchiveTracker.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example/TestZooKeeperTableArchiveClient.java


 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.2

 Attachments: 5547-v12.txt, 5547-v16.txt, 5547.addendum-v3, 
 hbase-5447-v8.patch, hbase-5447-v8.patch, hbase-5547-v9.patch, 
 java_HBASE-5547.addendum, java_HBASE-5547.addendum-v1, 
 java_HBASE-5547.addendum-v2, java_HBASE-5547_v13.patch, 
 java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch, 
 java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, 
 java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

2012-07-21 Thread Jesse Yates (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13419756#comment-13419756
]

Jesse Yates commented on HBASE-5547:

bq. I find that deleteRegion() was addressed, but not deleteTable().

In the deletetablehandler, it first goes and deletes each region (which gets
replaced with archiveRegion) and then once all the regions have been archived,
we call deleteTable, which basically just rm -rf's the entire directory.

TL;DR - hfiles on table delete are archived, but not any more information about
the state of the table. Wrote a unit test to check that too, so should be good.

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.94.2

Attachments: 5547-v12.txt, 5547-v16.txt, hbase-5447-v8.patch,
hbase-5447-v8.patch, hbase-5547-v9.patch, java_HBASE-5547_v13.patch,
java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch,
java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch,
java_HBASE-5547_v7.patch

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

2012-07-21 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420095#comment-13420095
 ] 

Hudson commented on HBASE-5547:
---

Integrated in HBase-TRUNK #3158 (See 
[https://builds.apache.org/job/HBase-TRUNK/3158/])
HBASE-5547 Don't delete HFiles when in backup mode (Jesse Yates) 
(Revision 1364203)

 Result = FAILURE
larsh : 
Files : 
* 
/hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/BaseConfigurable.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/Chore.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/HFileArchiver.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/HFileArchiveManager.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/HFileArchiveTableMonitor.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/LongTermArchivingHFileCleaner.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/TableHFileArchiveTracker.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/ZKTableArchiveClient.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/LogCleaner.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/LogCleanerDelegate.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/TimeToLiveLogCleaner.java
* /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseHFileCleanerDelegate.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/BaseLogCleanerDelegate.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/FileCleanerDelegate.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/LogCleaner.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/TimeToLiveHFileCleaner.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/TimeToLiveLogCleaner.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/master/ReplicationLogCleaner.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
* 
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
* /hbase/trunk/hbase-server/src/main/resources/hbase-default.xml
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestHFileArchiving.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example/TestZooKeeperTableArchiveClient.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestLogsCleaner.java
* /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestHFileCleaner.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestLogsCleaner.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/CheckedArchivingHFileCleaner.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java
* 
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestHFileArchiveUtil.java

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

2012-07-21 Thread Lars Hofhansl (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13420097#comment-13420097
 ] 

Lars Hofhansl commented on HBASE-5547:
--

Hmm... From the failed run:
{quote}
Results :

Failed tests:  
testMultipleTables(org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient):
 Archived HFiles should have gotten deleted, but didn't
  
testRowMutationMultiThreads(org.apache.hadoop.hbase.regionserver.TestAtomicOperation):
 expected:0 but was:5

Tests in error:
  
testMultiRowMutationMultiThreads(org.apache.hadoop.hbase.regionserver.TestAtomicOperation):
 java.io.FileNotFoundException: 
https://builds.apache.org/job/HBase-TRUNK/ws/trunk/hbase-server/target/classes/hbase-default.xml
 (Too many open files)
{quote}

The first one is suspicious. Could you have a look at it Jesse when you get a 
chance.

 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.96.0, 0.94.2

 Attachments: 5547-v12.txt, 5547-v16.txt, hbase-5447-v8.patch, 
 hbase-5447-v8.patch, hbase-5547-v9.patch, java_HBASE-5547_v13.patch, 
 java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch, 
 java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, 
 java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

2012-07-20 Thread Zhihong Ted Yu (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13419207#comment-13419207
 ] 

Zhihong Ted Yu commented on HBASE-5547:
---

Will integrate in 3 hours if there is no objection.

 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.94.2

 Attachments: 5547-v12.txt, 5547-v16.txt, hbase-5447-v8.patch, 
 hbase-5447-v8.patch, hbase-5547-v9.patch, java_HBASE-5547_v13.patch, 
 java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch, 
 java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, 
 java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13419696#comment-13419696
 ] 

Lars Hofhansl commented on HBASE-5547:
--

+1 on patch. Ted pinged me, that he is out already.
Since this is a Salesforce patch, I should commit it anyway.

Will do so as soon as I get to it.

Jesse, do you have a feeling about how different a 0.94 patch would be?

 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.94.2

 Attachments: 5547-v12.txt, 5547-v16.txt, hbase-5447-v8.patch, 
 hbase-5447-v8.patch, hbase-5547-v9.patch, java_HBASE-5547_v13.patch, 
 java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch, 
 java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, 
 java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

2012-07-20 Thread Jesse Yates (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13419697#comment-13419697
 ] 

Jesse Yates commented on HBASE-5547:


@Lars I don't think it would be all that different. I'll take a crack next week 
(after dealing with the next round of HBASE-6055 stuff).

 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.94.2

 Attachments: 5547-v12.txt, 5547-v16.txt, hbase-5447-v8.patch, 
 hbase-5447-v8.patch, hbase-5547-v9.patch, java_HBASE-5547_v13.patch, 
 java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch, 
 java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, 
 java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13419723#comment-13419723
]

Lars Hofhansl commented on HBASE-5547:
--

I also verified in a real setup, that an HFile is indeed archived and (by
default) removed after 5 mins. Was thrown off first, because the
table/region/cf directory is not removed when empty.
Also made sure I can create/drop tables and then .META. is backed up correctly.

So still +1 :)

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.94.2

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13419724#comment-13419724
]

Lars Hofhansl commented on HBASE-5547:
--

Ok. On more question I asked just now on RB:

From Matteo:
{quote}
MasterFileSystem contains deleteRegion() and deleteTable() that calls
fs.delete() with the recursive flag on.
This two methods get called by DeleteTableHandler (drop table).
In a backup/snapshot situation we want to keep the regions/hfiles.
{quote}

My follow up question:
{quote}
I find that deleteRegion() was addressed, but not deleteTable().

That means if a table is dropped the HFiles would be deleted and not
archived.
So it seems we should either:
- also delete the table's archive directory (since it would be
incomplete anyway).
- archive all the HFile before deleting them.

What do you think Jesse?
{quote}

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.94.2

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode


[ 
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13419725#comment-13419725
 ] 

Lars Hofhansl commented on HBASE-5547:
--

Or is it that all region are first deleted anyway, and only then the 
deleteTable is called (in DeleteTableHandler.handleTableOperation)

 Don't delete HFiles when in backup mode
 -

 Key: HBASE-5547
 URL: https://issues.apache.org/jira/browse/HBASE-5547
 Project: HBase
  Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
 Fix For: 0.94.2

 Attachments: 5547-v12.txt, 5547-v16.txt, hbase-5447-v8.patch, 
 hbase-5447-v8.patch, hbase-5547-v9.patch, java_HBASE-5547_v13.patch, 
 java_HBASE-5547_v14.patch, java_HBASE-5547_v15.patch, 
 java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, 
 java_HBASE-5547_v7.patch


 This came up in a discussion I had with Stack.
 It would be nice if HBase could be notified that a backup is in progress (via 
 a znode for example) and in that case either:
 1. rename HFiles to be delete to file.bck
 2. rename the HFiles into a special directory
 3. rename them to a general trash directory (which would not need to be tied 
 to backup mode).
 That way it should be able to get a consistent backup based on HFiles (HDFS 
 snapshots or hard links would be better options here, but we do not have 
 those).
 #1 makes cleanup a bit harder.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode

2012-07-20 Thread Zhihong Ted Yu (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13419746#comment-13419746
]

Zhihong Ted Yu commented on HBASE-5547:
---

I think the latest patch has addressed HFile archival when deleteRegion() is
called.

Backing up / restoring table can be addressed in HBASE-6055.

Don't delete HFiles when in backup mode
-

Key: HBASE-5547
URL: https://issues.apache.org/jira/browse/HBASE-5547
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Assignee: Jesse Yates
Fix For: 0.94.2

[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode