[jira] [Commented] (HBASE-12271) Add counters for files skipped during snapshot export
[ https://issues.apache.org/jira/browse/HBASE-12271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172600#comment-14172600 ] Elliott Clark commented on HBASE-12271: --- +1 Running in production with it. Add counters for files skipped during snapshot export - Key: HBASE-12271 URL: https://issues.apache.org/jira/browse/HBASE-12271 Project: HBase Issue Type: Improvement Components: snapshots Affects Versions: 0.98.7 Reporter: Patrick White Assignee: Patrick White Priority: Minor Fix For: 2.0.0, 0.98.8, 0.99.2 Attachments: 0001-Add-counters-for-skipped-files.patch It's incredibly handy to see the number of files skipped to know/verify delta backups are doing what they should, when they should. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12271) Add counters for files skipped during snapshot export
[ https://issues.apache.org/jira/browse/HBASE-12271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172598#comment-14172598 ] Matteo Bertozzi commented on HBASE-12271: - +1 Add counters for files skipped during snapshot export - Key: HBASE-12271 URL: https://issues.apache.org/jira/browse/HBASE-12271 Project: HBase Issue Type: Improvement Components: snapshots Affects Versions: 0.98.7 Reporter: Patrick White Assignee: Patrick White Priority: Minor Fix For: 2.0.0, 0.98.8, 0.99.2 Attachments: 0001-Add-counters-for-skipped-files.patch It's incredibly handy to see the number of files skipped to know/verify delta backups are doing what they should, when they should. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12271) Add counters for files skipped during snapshot export
[ https://issues.apache.org/jira/browse/HBASE-12271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172671#comment-14172671 ] Lars Hofhansl commented on HBASE-12271: --- Can you provide a paragraph of background? I.e. how do you use this for delta backup? (Just curious) Add counters for files skipped during snapshot export - Key: HBASE-12271 URL: https://issues.apache.org/jira/browse/HBASE-12271 Project: HBase Issue Type: Improvement Components: snapshots Affects Versions: 0.98.7 Reporter: Patrick White Assignee: Patrick White Priority: Minor Fix For: 2.0.0, 0.98.8, 0.99.2 Attachments: 0001-Add-counters-for-skipped-files.patch It's incredibly handy to see the number of files skipped to know/verify delta backups are doing what they should, when they should. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12271) Add counters for files skipped during snapshot export
[ https://issues.apache.org/jira/browse/HBASE-12271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172690#comment-14172690 ] Matteo Bertozzi commented on HBASE-12271: - [~larsh] export snapshot send only the difference between src and dst, so if two snapshots share some of the same files (e.g. no compaction between the two take snapshots) you export just the delta. this is just a counter to verify if the tool does this for real :) Add counters for files skipped during snapshot export - Key: HBASE-12271 URL: https://issues.apache.org/jira/browse/HBASE-12271 Project: HBase Issue Type: Improvement Components: snapshots Affects Versions: 0.98.7 Reporter: Patrick White Assignee: Patrick White Priority: Minor Fix For: 2.0.0, 0.98.8, 0.99.2 Attachments: 0001-Add-counters-for-skipped-files.patch It's incredibly handy to see the number of files skipped to know/verify delta backups are doing what they should, when they should. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12271) Add counters for files skipped during snapshot export
[
https://issues.apache.org/jira/browse/HBASE-12271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172694#comment-14172694
]
Hadoop QA commented on HBASE-12271:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12675042/0001-Add-counters-for-skipped-files.patch
against trunk revision .
ATTACHMENT ID: 12675042
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 2 zombie test(s):
at
org.apache.hadoop.hbase.util.TestBytes.testToStringBytesBinaryReversible(TestBytes.java:296)
at
org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles.testSimpleHFileSplit(TestLoadIncrementalHFiles.java:151)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/11358//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11358//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11358//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11358//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11358//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11358//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11358//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11358//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11358//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11358//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11358//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/11358//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/11358//console
This message is automatically generated.
Add counters for files skipped during snapshot export
-
Key: HBASE-12271
URL: https://issues.apache.org/jira/browse/HBASE-12271
Project: HBase
Issue Type: Improvement
Components: snapshots
Affects Versions: 0.98.7
Reporter: Patrick White
Assignee: Patrick White
Priority: Minor
Fix For: 2.0.0, 0.98.8, 0.99.2
Attachments: 0001-Add-counters-for-skipped-files.patch
It's incredibly handy to see the number of files skipped to know/verify delta
backups are doing what they should, when they should.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12271) Add counters for files skipped during snapshot export
[ https://issues.apache.org/jira/browse/HBASE-12271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172711#comment-14172711 ] Patrick White commented on HBASE-12271: --- ExportSnapshot will skip files that are the same (either via checksum or via name+length comparison). We use this to cut down on transfer time. First we snapshot the table, then export that snapshot to the remote DFS. Next run, we instead start by copying the previously exported snapshot into the current snapshot's destination (all on the remote DFS). We then do the snapshot and export dance again, and it dutifully copies only the changed files. Our (fb) HDFS has hard links (last I heard upstream does not) so we can copy backups around pretty cheaply. In the future we may just throw everything into one directory and let the manifest be the decider of what files are involved, but in the mean time it's copy and differential. Add counters for files skipped during snapshot export - Key: HBASE-12271 URL: https://issues.apache.org/jira/browse/HBASE-12271 Project: HBase Issue Type: Improvement Components: snapshots Affects Versions: 0.98.7 Reporter: Patrick White Assignee: Patrick White Priority: Minor Fix For: 2.0.0, 0.98.8, 0.99.2 Attachments: 0001-Add-counters-for-skipped-files.patch It's incredibly handy to see the number of files skipped to know/verify delta backups are doing what they should, when they should. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12271) Add counters for files skipped during snapshot export
[ https://issues.apache.org/jira/browse/HBASE-12271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172712#comment-14172712 ] Patrick White commented on HBASE-12271: --- Didn't see that [~mbertozzi] beat me to it :p Add counters for files skipped during snapshot export - Key: HBASE-12271 URL: https://issues.apache.org/jira/browse/HBASE-12271 Project: HBase Issue Type: Improvement Components: snapshots Affects Versions: 0.98.7 Reporter: Patrick White Assignee: Patrick White Priority: Minor Fix For: 2.0.0, 0.98.8, 0.99.2 Attachments: 0001-Add-counters-for-skipped-files.patch It's incredibly handy to see the number of files skipped to know/verify delta backups are doing what they should, when they should. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12271) Add counters for files skipped during snapshot export
[ https://issues.apache.org/jira/browse/HBASE-12271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172815#comment-14172815 ] Hudson commented on HBASE-12271: SUCCESS: Integrated in HBase-TRUNK #5666 (See [https://builds.apache.org/job/HBase-TRUNK/5666/]) HBASE-12271 Add counters for files skipped during snapshot export (eclark: rev ba20d4df8ced17e175d6a1d57982559d4dce79cd) * hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java Add counters for files skipped during snapshot export - Key: HBASE-12271 URL: https://issues.apache.org/jira/browse/HBASE-12271 Project: HBase Issue Type: Improvement Components: snapshots Affects Versions: 0.98.7 Reporter: Patrick White Assignee: Patrick White Priority: Minor Fix For: 2.0.0, 0.98.8, 0.99.2 Attachments: 0001-Add-counters-for-skipped-files.patch It's incredibly handy to see the number of files skipped to know/verify delta backups are doing what they should, when they should. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12271) Add counters for files skipped during snapshot export
[ https://issues.apache.org/jira/browse/HBASE-12271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172837#comment-14172837 ] Hudson commented on HBASE-12271: FAILURE: Integrated in HBase-1.0 #320 (See [https://builds.apache.org/job/HBase-1.0/320/]) HBASE-12271 Add counters for files skipped during snapshot export (eclark: rev c68c17ffec9fd83ff25928c0c0292f43f78acf02) * hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java Add counters for files skipped during snapshot export - Key: HBASE-12271 URL: https://issues.apache.org/jira/browse/HBASE-12271 Project: HBase Issue Type: Improvement Components: snapshots Affects Versions: 0.98.7 Reporter: Patrick White Assignee: Patrick White Priority: Minor Fix For: 2.0.0, 0.98.8, 0.99.2 Attachments: 0001-Add-counters-for-skipped-files.patch It's incredibly handy to see the number of files skipped to know/verify delta backups are doing what they should, when they should. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12271) Add counters for files skipped during snapshot export
[ https://issues.apache.org/jira/browse/HBASE-12271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172855#comment-14172855 ] Hudson commented on HBASE-12271: FAILURE: Integrated in HBase-0.98 #604 (See [https://builds.apache.org/job/HBase-0.98/604/]) HBASE-12271 Add counters for files skipped during snapshot export (eclark: rev 271d11a8592fdba9a8fd5ea5974cd934744235d3) * hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java Add counters for files skipped during snapshot export - Key: HBASE-12271 URL: https://issues.apache.org/jira/browse/HBASE-12271 Project: HBase Issue Type: Improvement Components: snapshots Affects Versions: 0.98.7 Reporter: Patrick White Assignee: Patrick White Priority: Minor Fix For: 2.0.0, 0.98.8, 0.99.2 Attachments: 0001-Add-counters-for-skipped-files.patch It's incredibly handy to see the number of files skipped to know/verify delta backups are doing what they should, when they should. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12271) Add counters for files skipped during snapshot export
[ https://issues.apache.org/jira/browse/HBASE-12271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172891#comment-14172891 ] Hudson commented on HBASE-12271: FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #574 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/574/]) HBASE-12271 Add counters for files skipped during snapshot export (eclark: rev 271d11a8592fdba9a8fd5ea5974cd934744235d3) * hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java Add counters for files skipped during snapshot export - Key: HBASE-12271 URL: https://issues.apache.org/jira/browse/HBASE-12271 Project: HBase Issue Type: Improvement Components: snapshots Affects Versions: 0.98.7 Reporter: Patrick White Assignee: Patrick White Priority: Minor Fix For: 2.0.0, 0.98.8, 0.99.2 Attachments: 0001-Add-counters-for-skipped-files.patch It's incredibly handy to see the number of files skipped to know/verify delta backups are doing what they should, when they should. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12271) Add counters for files skipped during snapshot export
[ https://issues.apache.org/jira/browse/HBASE-12271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173307#comment-14173307 ] Lars Hofhansl commented on HBASE-12271: --- Thanks for explaining [~mbertozzi], and [~patrickwhite]. Add counters for files skipped during snapshot export - Key: HBASE-12271 URL: https://issues.apache.org/jira/browse/HBASE-12271 Project: HBase Issue Type: Improvement Components: snapshots Affects Versions: 0.98.7 Reporter: Patrick White Assignee: Patrick White Priority: Minor Fix For: 2.0.0, 0.98.8, 0.99.2 Attachments: 0001-Add-counters-for-skipped-files.patch It's incredibly handy to see the number of files skipped to know/verify delta backups are doing what they should, when they should. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
