[
https://issues.apache.org/jira/browse/HBASE-22607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16995257#comment-16995257
]
Mingliang Liu edited comment on HBASE-22607 at 12/13/19 1:16 AM:
-----------------------------------------------------------------
[~AK2019] That is interesting.
Can you reproduce this consistently? If so, the problem might be easier to
debug. I can not debug here because I never see this with multiple runs.
{code}
git checkout rel/2.2.0
commit=$(git log master | grep -B 5 HBASE-22607 | grep commit | awk '{print
$2}')
git cherry-pick $commit
mvn clean package
mvn test -Dtest=TestExportSnapshotNoCluster
{code}
So I check the line number and it is not very clear which line error out in
{{testSnapshotWithRefsExportFileSystemState}}. I guess it's in LoC 216 of
{{TestExportSnapshot}}.
{code:title=TestExportSnapshot.java:216}
copyDir = copyDir.makeQualified(fs);
{code}
If so, the {{fs}} is created using a new Configuration which is NOT patched as
in {{TestExportSnapshotNoCluster}}. Could you try the addendum diff
[^HBASE-22607.addendum.000.patch] ? Hopefully it will fix this. Otherwise we
may have to debug further, which perhaps is orthogonal to this patch.
was (Author: liuml07):
[~AK2019] That is interesting.
Can you reproduce this consistently? If so, the problem might be easier to
debug. I can not debug here because I never see this with multiple runs.
{code}
git checkout rel/2.2.0
commit=$(git log master | grep -B 5 HBASE-22607 | grep commit | awk '{print
$2}')
git cherry-pick $commit
mvn clean package
mvn test -Dtest=TestExportSnapshotNoCluster
{code}
So I check the line number and it is not very clear which line error out in
{{testSnapshotWithRefsExportFileSystemState(}}. I guess it's in LoC 216 of
{{TestExportSnapshot}}. If so, the fs is created using new Configuration which
is patched as in {{TestExportSnapshotNoCluster}}.
{code:title=TestExportSnapshot.java:216}
copyDir = copyDir.makeQualified(fs);
{code}
Could you try the addendum diff [^HBASE-22607.addendum.000.patch] ? Hopefully
it will fix this. Otherwise we may have to debug further, which perhaps is
orthogonal to this patch.
> TestExportSnapshotNoCluster::testSnapshotWithRefsExportFileSystemState()
> fails intermittently
> ---------------------------------------------------------------------------------------------
>
> Key: HBASE-22607
> URL: https://issues.apache.org/jira/browse/HBASE-22607
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 3.0.0, 2.2.0, 2.0.6
> Reporter: Mingliang Liu
> Assignee: Mingliang Liu
> Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.2.3, 2.1.9
>
> Attachments: HBASE-22607.000.patch, HBASE-22607.001.patch,
> HBASE-22607.002.patch, HBASE-22607.addendum.000.patch
>
>
> In previous runs, test
> {{TestExportSnapshotNoCluster.testSnapshotWithRefsExportFileSystemState}}
> fails intermittently with {{java.net.ConnectException: Connection refused}}
> exception, see build
> [510|https://builds.apache.org/job/PreCommit-HBASE-Build/510/testReport/org.apache.hadoop.hbase.snapshot/TestExportSnapshotNoCluster/testSnapshotWithRefsExportFileSystemState/],
>
> [545|https://builds.apache.org/job/PreCommit-HBASE-Build/545/testReport/org.apache.hadoop.hbase.snapshot/TestExportSnapshotNoCluster/testSnapshotWithRefsExportFileSystemState/],
> and
> [556|https://builds.apache.org/job/PreCommit-HBASE-Build/556/testReport/org.apache.hadoop.hbase.snapshot/TestExportSnapshotNoCluster/testSnapshotWithRefsExportFileSystemState/].
> So one sample exception is like:
> {quote}
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:346)
> at com.sun.proxy.$Proxy20.getListing(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1630)
> at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1614)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:900)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:114)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:964)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:961)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:961)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1537)
> at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1580)
> at
> org.apache.hadoop.hbase.util.CommonFSUtils.listStatus(CommonFSUtils.java:693)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.getCurrentTableInfoStatus(FSTableDescriptors.java:448)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.getTableInfoPath(FSTableDescriptors.java:429)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.getTableInfoPath(FSTableDescriptors.java:410)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.createTableDescriptorForTableDirectory(FSTableDescriptors.java:763)
> at
> org.apache.hadoop.hbase.snapshot.SnapshotTestingUtils$SnapshotMock.createTable(SnapshotTestingUtils.java:675)
> at
> org.apache.hadoop.hbase.snapshot.SnapshotTestingUtils$SnapshotMock.createSnapshot(SnapshotTestingUtils.java:653)
> at
> org.apache.hadoop.hbase.snapshot.SnapshotTestingUtils$SnapshotMock.createSnapshot(SnapshotTestingUtils.java:647)
> at
> org.apache.hadoop.hbase.snapshot.SnapshotTestingUtils$SnapshotMock.createSnapshotV2(SnapshotTestingUtils.java:637)
> at
> org.apache.hadoop.hbase.snapshot.TestExportSnapshotNoCluster.testSnapshotWithRefsExportFileSystemState(TestExportSnapshotNoCluster.java:80)
> {quote}
> This seems that, somehow the rootdir filesystem is not LocalFileSystem, but
> on HDFS. I have not dig deeper why this happens since it's failing
> intermittently and I can not reproduce it locally. Since this is testing
> export snapshot tool without cluster, we can enforce it using
> LocalFileSystem; no breaking change.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
