[
https://issues.apache.org/jira/browse/HADOOP-18723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17719108#comment-17719108
]
ASF GitHub Bot commented on HADOOP-18723:
-----------------------------------------
symious commented on PR #5603:
URL: https://github.com/apache/hadoop/pull/5603#issuecomment-1533942745
> Well, I'm afraid your specific problem does not match Dee why do use cases
of uploading to stores without checksums. Now, I would I've been happier if
distcp's -skipCrc option was required to copy data from an FS with checksums to
one without, but it is not and to add it now would break so many people's
workflows.
>
> So what do we do here?
>
> maybe: create counters of why files were copied, specifically
>
> * not found at destination
> * file length different
> * modtime
> * checksum
>
> Then after a job you can see why files were copied from the host where the
job was launched. Then if you want to know why there were issues such as
checksums and modtimes, you can log out to debug. Obviously, this will be
something to add to the distcp documentation.
>
> Now: big warning. I am personally scared of distCp. It is a critical
workflow tool and even use programmatically, yet it is surprisingly brittle. It
is a running joke that's the last person two add any code to the module gets to
field or support calls until someone else comes along. Thank you for
volunteering! This also explains why we will be very reluctant/strict about
taking on changes. Don't take it personally is as hey everyone gets that same
grilling here.
Totally understand.
I have removed the error log if checksum is null for sourcePath and
targetPath, so a FS without checksum won't be affected.
I have also changed the error log to INFO level if both checksum exists but
not equal, or a DEBUG level will be more preferred? Personally I think INFO
level can save us more time since we don't need to reconfig and restart the
job.
> Add detail logs if distcp checksum mismatch
> -------------------------------------------
>
> Key: HADOOP-18723
> URL: https://issues.apache.org/jira/browse/HADOOP-18723
> Project: Hadoop Common
> Issue Type: Improvement
> Reporter: Janus Chow
> Assignee: Janus Chow
> Priority: Major
> Labels: pull-request-available
>
> We encountered some errors of mismatch checksum during Distcp jobs. It took
> us some time to figure out that checksum type is different.
> Adding error logs shall help us to figure out such problems faster.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
