[
https://issues.apache.org/jira/browse/HDFS-15596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17201340#comment-17201340
]
Uma Maheswara Rao G commented on HDFS-15596:
--------------------------------------------
After the fix distCp ran successfully:
{code:java}
[root@uma-1 /]# sudo -u hdfs hadoop distcp /test /OzoneTest
20/09/24 06:56:09 INFO tools.DistCp: Input Options:
DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false,
ignoreFailures=false, overwrite=false, append=false, useDiff=false,
useRdiff=false, fromSnapshot=null, toSnapshot=null, skipCRC=false,
blocking=true, numListstatusThreads=0, maxMaps=20, mapBandwidth=0.0,
copyStrategy='uniformsize', preserveStatus=[], atomicWorkPath=null,
logPath=null, sourceFileListing=null, sourcePaths=[/test],
targetPath=/OzoneTest, filtersFile='null', blocksPerChunk=0,
copyBufferSize=8192, verboseLog=false, directWrite=false}, sourcePaths=[/test],
targetPathExists=true, preserveRawXattrsfalse
20/09/24 06:56:10 INFO tools.SimpleCopyListing: Paths (files+dirs) cnt = 2;
dirCnt = 1
20/09/24 06:56:10 INFO tools.SimpleCopyListing: Build file listing completed.
20/09/24 06:56:10 INFO Configuration.deprecation: io.sort.mb is deprecated.
Instead, use mapreduce.task.io.sort.mb
20/09/24 06:56:10 INFO Configuration.deprecation: io.sort.factor is deprecated.
Instead, use mapreduce.task.io.sort.factor
20/09/24 06:56:10 INFO tools.DistCp: Number of paths in the copy list: 2
20/09/24 06:56:10 INFO tools.DistCp: Number of paths in the copy list: 2
20/09/24 06:56:10 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
to rm23
20/09/24 06:56:10 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding
for path: /user/hdfs/.staging/job_1600930164279_0009
20/09/24 06:56:10 INFO mapreduce.JobSubmitter: number of splits:2
20/09/24 06:56:11 INFO mapreduce.JobSubmitter: Submitting tokens for job:
job_1600930164279_0009
20/09/24 06:56:11 INFO mapreduce.JobSubmitter: Executing with tokens: []
20/09/24 06:56:11 INFO conf.Configuration: resource-types.xml not found
20/09/24 06:56:11 INFO resource.ResourceUtils: Unable to find
'resource-types.xml'.
20/09/24 06:56:11 INFO impl.YarnClientImpl: Submitted application
application_1600930164279_0009
20/09/24 06:56:11 INFO mapreduce.Job: The url to track the job:
http://uma-2.uma.xxx.xxx.xxx:8088/proxy/application_1600930164279_0009/
20/09/24 06:56:11 INFO tools.DistCp: DistCp job-id: job_1600930164279_0009
20/09/24 06:56:11 INFO mapreduce.Job: Running job: job_1600930164279_0009
20/09/24 06:56:22 INFO mapreduce.Job: Job job_1600930164279_0009 running in
uber mode : false
20/09/24 06:56:22 INFO mapreduce.Job: map 0% reduce 0%
20/09/24 06:56:28 INFO mapreduce.Job: map 50% reduce 0%
20/09/24 06:56:34 INFO mapreduce.Job: map 100% reduce 0%
20/09/24 06:56:34 INFO mapreduce.Job: Job job_1600930164279_0009 completed
successfully
20/09/24 06:56:35 INFO mapreduce.Job: Counters: 43
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=583482
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=865
HDFS: Number of bytes written=0
HDFS: Number of read operations=20
HDFS: Number of large read operations=0
HDFS: Number of write operations=4
HDFS: Number of bytes read erasure-coded=0
O3FS: Number of bytes read=0
O3FS: Number of bytes written=17
O3FS: Number of read operations=12
O3FS: Number of large read operations=0
O3FS: Number of write operations=3
Job Counters
Launched map tasks=2
Other local map tasks=2
Total time spent by all maps in occupied slots (ms)=13070
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=13070
Total vcore-milliseconds taken by all map tasks=13070
Total megabyte-milliseconds taken by all map tasks=13383680
Map-Reduce Framework
Map input records=2
Map output records=0
Input split bytes=228
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=285
CPU time spent (ms)=3030
Physical memory (bytes) snapshot=804298752
Virtual memory (bytes) snapshot=5346963456
Total committed heap usage (bytes)=729284608
Peak Map Physical memory (bytes)=428199936
Peak Map Virtual memory (bytes)=2674827264
File Input Format Counters
Bytes Read=620
File Output Format Counters
Bytes Written=0
DistCp Counters
Bandwidth in Btyes=17
Bytes Copied=17
Bytes Expected=17
Files Copied=1
DIR_COPY=1
{code}
> ViewHDFS#create(f, permission, cflags, bufferSize, replication, blockSize,
> progress, checksumOpt) should not be restricted to DFS only.
> ---------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-15596
> URL: https://issues.apache.org/jira/browse/HDFS-15596
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Uma Maheswara Rao G
> Assignee: Uma Maheswara Rao G
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The ViewHDFS#create(f, permission, cflags, bufferSize, replication,
> blockSize, progress, checksumOpt) API already available in FileSystem. It
> will use other overloaded API and finally can go to ViewFileSystem. This case
> works in regular ViewFileSystem also. With ViewHDFS, we restricted this to
> DFS only which cause discp to fail when target is non hdfs as it's using this
> API.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
