[
https://issues.apache.org/jira/browse/HBASE-28702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17864549#comment-17864549
]
Duo Zhang commented on HBASE-28702:
-----------------------------------
{noformat}
2024-07-10T15:34:52,265 ERROR [LocalJobRunner Map Task Executor #0 {}]
snapshot.ExportSnapshot$ExportMapper(349): Error copying
hdfs://localhost:39945/user/zhangduo/test-data/a21e8cf5-9af5-126a-855b-71affd337fcc/data/default/test-1720596876827/56758af424851cf1fc306e0093a67f53/f/09e33b3554344debbb5b11b056e35668
to
file:/home/zhangduo/hbase/code/hbase-backup/target/test-data/18db087b-d1f1-4bb8-82c3-61742253c6ae/backupTarget/backup_1720596883706/default/test-1720596876827/archive/data/default/test-1720596876827/56758af424851cf1fc306e0093a67f53/f/09e33b3554344debbb5b11b056e35668
java.io.IOException: Checksum mismatch between
hdfs://localhost:39945/user/zhangduo/test-data/a21e8cf5-9af5-126a-855b-71affd337fcc/data/default/test-1720596876827/56758af424851cf1fc306e0093a67f53/f/09e33b3554344debbb5b11b056e35668
and
file:/home/zhangduo/hbase/code/hbase-backup/target/test-data/18db087b-d1f1-4bb8-82c3-61742253c6ae/backupTarget/backup_1720596883706/default/test-1720596876827/archive/data/default/test-1720596876827/56758af424851cf1fc306e0093a67f53/f/09e33b3554344debbb5b11b056e35668.
Input and output filesystems are of different types.
Their checksum algorithms may be incompatible. You can choose file-level
checksum validation via -Ddfs.checksum.combine.mode=COMPOSITE_CRC when
block-sizes or filesystems are different. Or you can skip checksum-checks
altogether with --no-checksum-verify.
(NOTE: By skipping checksums, one runs the risk of masking data-corruption
during file-transfer.)
at
org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.verifyCopyResult(ExportSnapshot.java:596)
~[hbase-mapreduce-4.0.0-alpha-1-SNAPSHOT.jar:4.0.0-alpha-1-SNAPSHOT]
at
org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.copyFile(ExportSnapshot.java:334)
~[hbase-mapreduce-4.0.0-alpha-1-SNAPSHOT.jar:4.0.0-alpha-1-SNAPSHOT]
at
org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.map(ExportSnapshot.java:256)
~[hbase-mapreduce-4.0.0-alpha-1-SNAPSHOT.jar:4.0.0-alpha-1-SNAPSHOT]
at
org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportMapper.map(ExportSnapshot.java:180)
~[hbase-mapreduce-4.0.0-alpha-1-SNAPSHOT.jar:4.0.0-alpha-1-SNAPSHOT]
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
~[hadoop-mapreduce-client-core-3.3.5.jar:?]
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:800)
~[hadoop-mapreduce-client-core-3.3.5.jar:?]
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:348)
~[hadoop-mapreduce-client-core-3.3.5.jar:?]
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
~[hadoop-mapreduce-client-common-3.3.5.jar:?]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
~[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
~[?:?]
at java.lang.Thread.run(Thread.java:840) ~[?:?]
{noformat}
This is the error, I think t is introduced by HBASE-28625, where we add
checksum verification when copying snapshot files. For different file system
types, we use different checksum algorithm so the checksum mismatches and fail
the ExportSnapshot job.
I think this is a real problem in production, not only for test.
[~bbeaudreault] This should be a blocker for 2.6.1?
Thanks.
> TestBackupMerge fails 100% of times on flaky dashboard
> ------------------------------------------------------
>
> Key: HBASE-28702
> URL: https://issues.apache.org/jira/browse/HBASE-28702
> Project: HBase
> Issue Type: Bug
> Components: backup&restore
> Reporter: Duo Zhang
> Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
