[jira] [Created] (HDFS-13696) Distcp between 2 secure clusters and non encrypted zones fails with connection timeout
Rohit Pegallapati created HDFS-13696:
Summary: Distcp between 2 secure clusters and non encrypted zones
fails with connection timeout
Key: HDFS-13696
URL: https://issues.apache.org/jira/browse/HDFS-13696
Project: Hadoop HDFS
Issue Type: Bug
Components: distcp
Reporter: Rohit Pegallapati
We are trying to do a distcp between 2 secure clusters. But distcp is between 2
unencrypted zones.
eg : hadoop distcp sourceHdfs://tmp/text1.txt destHdfs://tmp
throws
{code:java}
org.apache.oozie.action.ActionExecutorException: JA009: connect timed out
at
org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:463)
at
org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:437)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1247)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1425)
at
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:232)
at
org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:63)
at org.apache.oozie.command.XCommand.call(XCommand.java:286)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
at sun.net.www.protocol.https.HttpsClient.(HttpsClient.java:264)
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
at
sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
at
sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138)
at
sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032)
at
sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
at
sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153)
at
org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:184)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:139)
at
org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:348)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:308)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:180)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:382)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider$4.run(KMSClientProvider.java:1014)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider$4.run(KMSClientProvider.java:1008)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at
org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:1008)
at
org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:110)
at
org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2333)
at
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/821/ [Jun 23, 2018 10:49:44 PM] (aengineer) HDDS-184. Upgrade common-langs version to 3.7 in [Jun 24, 2018 8:05:04 AM] (aengineer) HDDS-177. Create a releasable ozonefs artifact Contributed by Marton, -1 overall The following subsystems voted -1: asflicense compile findbugs pathlen unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.security.TestRaceWhenRelogin hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageEntities hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageSchema hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageDomain hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage hadoop.mapred.TestMRTimelineEventHandling compile: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/821/artifact/out/patch-compile-root.txt [560K] cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/821/artifact/out/patch-compile-root.txt [560K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/821/artifact/out/patch-compile-root.txt [560K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/821/artifact/out/diff-checkstyle-root.txt [4.0K] pathlen: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/821/artifact/out/pathlen.txt [12K] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/821/artifact/out/diff-patch-pylint.txt [24K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/821/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/821/artifact/out/diff-patch-shelldocs.txt [16K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/821/artifact/out/whitespace-eol.txt [9.4M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/821/artifact/out/whitespace-tabs.txt [1.1M] xml: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/821/artifact/out/xml.txt [4.0K] findbugs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/821/artifact/out/branch-findbugs-hadoop-hdds_client.txt [56K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/821/artifact/out/branch-findbugs-hadoop-hdds_container-service.txt [48K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/821/artifact/out/branch-findbugs-hadoop-hdds_server-scm.txt [60K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/821/artifact/out/branch-findbugs-hadoop-hdds_tools.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/821/artifact/out/branch-findbugs-hadoop-ozone_client.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/821/artifact/out/branch-findbugs-hadoop-ozone_common.txt [24K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/821/artifact/out/branch-findbugs-hadoop-ozone_objectstore-service.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/821/artifact/out/branch-findbugs-hadoop-ozone_ozone-manager.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/821/artifact/out/branch-findbugs-hadoop-ozone_ozonefs.txt [8.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/821/artifact/out/branch-findbugs-hadoop-ozone_tools.txt [4.0K] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/821/artifact/out/diff-javadoc-javadoc-root.txt [760K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/821/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [168K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/821/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [340K]
Apache Hadoop qbt Report: trunk+JDK8 on Windows/x64
For more details, see https://builds.apache.org/job/hadoop-trunk-win/507/ No changes -1 overall The following subsystems voted -1: compile mvninstall pathlen unit The following subsystems voted -1 but were configured to be filtered/ignored: cc javac The following subsystems are considered long running: (runtime bigger than 1h 00m 00s) unit Specific tests: Failed junit tests : hadoop.crypto.TestCryptoStreamsWithOpensslAesCtrCryptoCodec hadoop.fs.contract.rawlocal.TestRawlocalContractAppend hadoop.fs.TestFileUtil hadoop.fs.TestFsShellCopy hadoop.fs.TestFsShellList hadoop.fs.TestLocalFileSystem hadoop.http.TestHttpServer hadoop.http.TestHttpServerLogs hadoop.io.nativeio.TestNativeIO hadoop.ipc.TestIPC hadoop.ipc.TestSocketFactory hadoop.metrics2.impl.TestStatsDMetrics hadoop.security.TestSecurityUtil hadoop.security.TestShellBasedUnixGroupsMapping hadoop.security.token.TestDtUtilShell hadoop.util.TestDiskCheckerWithDiskIo hadoop.util.TestNativeCodeLoader hadoop.hdfs.client.impl.TestBlockReaderLocal hadoop.hdfs.qjournal.server.TestJournalNode hadoop.hdfs.qjournal.server.TestJournalNodeSync hadoop.hdfs.server.blockmanagement.TestNameNodePrunesMissingStorages hadoop.hdfs.server.datanode.fsdataset.impl.TestProvidedImpl hadoop.hdfs.server.datanode.TestBlockPoolSliceStorage hadoop.hdfs.server.datanode.TestBlockScanner hadoop.hdfs.server.datanode.TestDataNodeFaultInjector hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure hadoop.hdfs.server.datanode.TestDirectoryScanner hadoop.hdfs.server.diskbalancer.TestDiskBalancerRPC hadoop.hdfs.server.mover.TestMover hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA hadoop.hdfs.server.namenode.ha.TestHASafeMode hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics hadoop.hdfs.server.namenode.TestFileTruncate hadoop.hdfs.TestDatanodeStartupFixesLegacyStorageIDs hadoop.hdfs.TestDFSShell hadoop.hdfs.TestDFSStripedOutputStreamWithFailure hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy hadoop.hdfs.TestDFSUpgradeFromImage hadoop.hdfs.TestFetchImage hadoop.hdfs.TestFileConcurrentReader hadoop.hdfs.TestHDFSFileSystemContract hadoop.hdfs.TestLeaseRecovery hadoop.hdfs.TestPread hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy hadoop.hdfs.TestSecureEncryptionZoneWithKMS hadoop.hdfs.TestTrashWithSecureEncryptionZones hadoop.hdfs.tools.TestDFSAdmin hadoop.hdfs.web.TestWebHDFS hadoop.hdfs.web.TestWebHdfsUrl hadoop.fs.http.server.TestHttpFSServerWebServer hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch hadoop.yarn.server.nodemanager.containermanager.TestAuxServices hadoop.yarn.server.nodemanager.containermanager.TestContainerManager hadoop.yarn.server.nodemanager.recovery.TestNMLeveldbStateStoreService hadoop.yarn.server.nodemanager.TestContainerExecutor hadoop.yarn.server.nodemanager.TestNodeManagerResync hadoop.yarn.server.webproxy.amfilter.TestAmFilter hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer hadoop.yarn.server.timeline.security.TestTimelineAuthenticationFilterForV1 hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.TestFSSchedulerConfigurationStore hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.TestLeveldbConfigurationStore hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueManagementDynamicEditPolicy hadoop.yarn.server.resourcemanager.scheduler.constraint.TestPlacementProcessor hadoop.yarn.server.resourcemanager.scheduler.fair.TestAllocationFileLoaderService hadoop.yarn.server.resourcemanager.scheduler.TestSchedulingWithAllocationRequestId hadoop.yarn.server.resourcemanager.TestCapacitySchedulerMetrics hadoop.yarn.server.resourcemanager.TestResourceTrackerService hadoop.yarn.client.api.impl.TestNMClient hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps
Apache Hadoop qbt Report: trunk+JDK9 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java9-linux-x86/29/ [Jun 17, 2018 6:54:26 PM] (cdouglas) HADOOP-13186. Multipart Uploader API. Contributed by Ewan Higgs [Jun 18, 2018 6:12:18 AM] (cdouglas) HADOOP-15533. Make WASB listStatus messages consistent. Contributed by [Jun 18, 2018 6:48:49 AM] (msingh) HDDS-141. Remove PipeLine Class from SCM and move the data field in the [Jun 18, 2018 5:17:07 PM] (aajisaka) HDFS-13621. Upgrade commons-lang version to 3.7 in hadoop-hdfs-project. [Jun 18, 2018 6:28:22 PM] (eyang) HADOOP-15527. Improve delay check for stopping processes. [Jun 18, 2018 9:25:11 PM] (aajisaka) MAPREDUCE-7063. Fix log level inconsistency in [Jun 18, 2018 11:45:50 PM] (aajisaka) YARN-7668. Remove unused variables from ContainerLocalizer [Jun 19, 2018 5:38:13 PM] (eyang) HADOOP-15527. Improve delay check for stopping processes. [Jun 20, 2018 5:42:13 PM] (haibochen) YARN-8437. Build oom-listener fails on older versions. (Miklos Szegedi [Jun 20, 2018 5:59:33 PM] (miklos.szegedi) YARN-8391. Investigate AllocationFileLoaderService.reloadListener [Jun 20, 2018 6:36:12 PM] (miklos.szegedi) YARN-8440. Typo in YarnConfiguration javadoc: "Miniumum request [Jun 20, 2018 6:40:56 PM] (miklos.szegedi) YARN-7449. Split up class TestYarnClient to TestYarnClient and [Jun 20, 2018 6:55:43 PM] (miklos.szegedi) YARN-8442. Strange characters and missing spaces in FairScheduler [Jun 20, 2018 6:58:18 PM] (miklos.szegedi) YARN-8441. Typo in CSQueueUtils local variable names: [Jun 20, 2018 7:04:44 PM] (miklos.szegedi) MAPREDUCE-7113. Typos in test names in TestTaskAttempt: [Jun 20, 2018 10:45:08 PM] (mackrorysd) HADOOP-14918. Remove the Local Dynamo DB test option. Contributed by [Jun 20, 2018 10:58:26 PM] (xiao) HDFS-13682. Cannot create encryption zone after KMS auth token expires. [Jun 20, 2018 11:43:10 PM] (todd) HADOOP-15551. Avoid use of Arrays.stream in Configuration.addTags [Jun 21, 2018 3:20:24 PM] (eyang) YARN-8445. Improved error message for duplicated service and component [Jun 21, 2018 5:32:52 PM] (todd) HADOOP-15549. Upgrade to commons-configuration 2.1 regresses task CPU [Jun 22, 2018 1:24:10 AM] (inigoiri) YARN-8412. Move ResourceRequest.clone logic everywhere into a proper [Jun 22, 2018 2:50:54 AM] (yqlin) HDFS-13692. StorageInfoDefragmenter floods log when compacting [Jun 22, 2018 5:15:29 PM] (ericp) YARN-8444: NodeResourceMonitor crashes on bad swapFree value. [Jun 22, 2018 5:36:56 PM] (mackrorysd) HADOOP-15416. Clear error message in S3Guard diff if source not found. [Jun 22, 2018 8:05:41 PM] (miklos.szegedi) MAPREDUCE-7114. Make FrameworkUploader symlink ignore improvement. [Jun 22, 2018 9:03:55 PM] (yufei_gu) YARN-8184. Too many metrics if [Jun 22, 2018 11:12:48 PM] (eyang) YARN-8326. Removed exit code file check for launched container. [Jun 23, 2018 10:49:44 PM] (aengineer) HDDS-184. Upgrade common-langs version to 3.7 in [Jun 24, 2018 8:05:04 AM] (aengineer) HDDS-177. Create a releasable ozonefs artifact Contributed by Marton, -1 overall The following subsystems voted -1: compile findbugs mvninstall mvnsite pathlen shadedclient unit xml The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace Specific tests: Failed junit tests : hadoop.security.authentication.util.TestKerberosName hadoop.security.authentication.server.TestAltKerberosAuthenticationHandler hadoop.security.authentication.client.TestKerberosAuthenticator mvninstall: https://builds.apache.org/job/hadoop-qbt-trunk-java9-linux-x86/29/artifact/out/patch-mvninstall-root.txt [120K] compile: https://builds.apache.org/job/hadoop-qbt-trunk-java9-linux-x86/29/artifact/out/patch-compile-root.txt [52K] cc: https://builds.apache.org/job/hadoop-qbt-trunk-java9-linux-x86/29/artifact/out/patch-compile-root.txt [52K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java9-linux-x86/29/artifact/out/patch-compile-root.txt [52K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java9-linux-x86/29/artifact/out/diff-checkstyle-root.txt [4.0K] mvnsite: https://builds.apache.org/job/hadoop-qbt-trunk-java9-linux-x86/29/artifact/out/patch-mvnsite-root.txt [36K] pathlen: https://builds.apache.org/job/hadoop-qbt-trunk-java9-linux-x86/29/artifact/out/pathlen.txt [12K] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java9-linux-x86/29/artifact/out/diff-patch-pylint.txt [24K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java9-linux-x86/29/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java9-linux-x86/29/artifact/out/diff-patch-shelldocs.txt [16K] whitespace:
